add types for iterator helpers proposal #58222

bakkot · 2024-04-17T02:59:15Z

I have only minimal tests because almost all the types are copied directly from Array with very minimal changes (mostly dropping the third "array" parameter to callbacks and the "thisArg" parameter to callback-taking methods, and returning NativeIterator<T> instead of T[]). But I'm happy to make the tests more extensive if you'd like.

For reviewing:

ignore bd51b00 through 30d9bc2; the last one reverts the earlier ones so the next commits can be clearer
f5393db and ebc4e71 add the new BuiltinIterator and AsyncBuiltinIterator types and adopt them everywhere except the .generated files. AsyncBuiltinIterator isn't necessary yet but I figured we might as well do this plumbing work now.
c4a8a08 adds the new types following the strategy @rbuckton suggested below

The "rebaseline" commits are just running npm run test -- --no-lint; npx hereby baseline-accept and committing the result.

Current status (as of 2024-04-25):

waiting on Add TReturn/TNext to Iterable et al #58243
needs CloseableBuiltinIterator or something to mean "built-in iterator with a return method"
needs a rebase
needs reviews from the team

I'll address the outstanding todos one #58243 lands.

fatcerberus · 2024-04-17T03:25:54Z

I will need someone to tell me how to handle the generated DOM typings

The DOM typings are maintained at https://github.com/microsoft/TypeScript-DOM-lib-generator

bakkot · 2024-04-17T03:39:32Z

More concretely, I'll need someone to tell me how to handle the change: changing the DOM typings to use NativeIterator would break anyone not importing esnext.iterator, but there's not (currently) an equivalent of esnext for DOM typings, so there's not anywhere else to do the update.

Renegade334 · 2024-04-17T10:21:06Z

More concretely, I'll need someone to tell me how to handle the change: changing the DOM typings to use NativeIterator would break anyone not importing esnext.iterator, but there's not (currently) an equivalent of esnext for DOM typings, so there's not anywhere else to do the update.

Would the following work?

Introduce a hollow NativeIterator interface in es2015.iterable.d.ts
Re-type all intrinsic appearances of IterableIterator in es2015.iterable.d.ts etc. to NativeIterator
Extend the NativeIterator interface in esnext.iterator.d.ts, add the constructor definition, etc.

This would allow what you're looking for: keeps compatibility for pre-esnext targets, while allowing the helper methods to be merged into NativeIterator in esnext. The DOM library could then be retyped without causing BC issues.

Sparse illustration:

// @target: esnext
// @lib: esnext

// @filename: lib.es2015.iterable.d.ts

interface Iterator<T, TReturn = any, TNext = undefined> {
    next(...args: [] | [TNext]): IteratorResult<T, TReturn>;
    return?(value?: TReturn): IteratorResult<T, TReturn>;
    throw?(e?: any): IteratorResult<T, TReturn>;
}

interface Iterable<T> {
    [Symbol.iterator](): Iterator<T>;
}

interface IterableIterator<T> extends Iterator<T> {
    [Symbol.iterator](): IterableIterator<T>;
}

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {
    [Symbol.iterator](): NativeIterator<T, TReturn, TNext>;
}

interface Generator<T = unknown, TReturn = any, TNext = unknown> extends NativeIterator<T, TReturn, TNext> {
    // etc.
}

interface Array<T> /* etc. */ {
    // was: [Symbol.iterator](): IterableIterator<T>;
    [Symbol.iterator](): NativeIterator<T>;

    // was: entries(): IterableIterator<[number, T]>;
    entries(): NativeIterator<[number, T]>;

    // etc.
}


// @filename: lib.esnext.iterator.d.ts

interface NativeIterator<T, TReturn, TNext> {
    map<U>(callbackfn: (value: T, index: number) => U): NativeIterator<U>;

    // etc.
}

// @filename: test.ts

const mappedArrayIterator = ['a', 'b', 'c'].entries().map(([k, s]) => k * s.charCodeAt(0)); // NativeIterator<number>

const castToIterableIterator: IterableIterator<number> = mappedArrayIterator; // NativeIterator<T> remains assignable to IterableIterator<T>, for what it's worth

As for the question of extending IterableIterator versus creating a new interface for native Iterator instances, my own $0.02 is in favour of the latter. IterableIterator is a mature interface, and it isn't just used internally: a cursory glance through GH shows quite a few examples of manually-implemented iterable iterator objects in the ecosystem, and these remain valid iterables/iterators in esnext even though they are not descended from Iterator.prototype and don't implement helper methods. Maintaining separation between the "implements iterator protocols" interface and the "descends from Iterator.prototype" interface makes sense logically, and has the added benefit of not breaking third-party implementations of the existing interface.

bakkot · 2024-04-17T15:01:31Z

Would the following not work?

Nice, that sounds like a good approach to me.

rbuckton · 2024-04-17T16:44:54Z

@bakkot the approach I am experimenting with to support abstract on NativeIterator looks something like this:

// lib.esnext.iterator.d.ts
/// <reference lib="es2015.iterable" />
export {};

// Abstract type that allows us to mark `next` as `abstract`
declare abstract class Iterator<T> {
  abstract next(value?: undefined): IteratorResult<T, void>;
}

// Merge all members of `NativeIterator<T>` into `Iterator<T>`
interface Iterator<T> extends globalThis.NativeIterator<T, void, undefined> {}

// Capture the `Iterator` constructor in a type we can use in the `extends` clause of `IteratorConstructor`.
type NativeIteratorConstructor = typeof Iterator;

declare global {
  // Global `NativeIterator<T>` interface that can be augmented by polyfills
  interface NativeIterator<T, TReturn, TNext> {
    // prototype elements
  }

  // Global `IteratorConstructor` interface that can be augmented by polyfills
  interface IteratorConstructor extends NativeIteratorConstructor {
    // static elements
  }

  var Iterator: IteratorConstructor;
}

// lib.es2015.iterable.d.ts

...

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {
  [Symbol.iterator](): NativeIterator<T>;
}

...

And in use:

new Iterator<number>(); // ts(2511): Cannot create an instance of an abstract class.

class C extends Iterator<number> {} // ts(2515): Non-abstract class 'C' does not implement inherited
                                    //           abstract member next from class 'Iterator<number>'.

Lib references can't really be referenced as modules, and even if you could this provides no exports. However, we will still augment the global scope with the types defined in declare global.

rbuckton · 2024-04-17T16:48:19Z

There need to be a few other changes so that it can also be used with generators, though.

rbuckton · 2024-04-17T16:59:11Z

src/lib/esnext.iterator.d.ts

+    find<S extends T>(predicate: (value: T, index: number) => value is S): S | undefined;
+    find(predicate: (value: T, index: number) => unknown): T | undefined;
+
+    readonly [Symbol.toStringTag]: "Iterator";


I would suggest defining this as

Suggested change

readonly [Symbol.toStringTag]: "Iterator";

readonly [Symbol.toStringTag]: string;

otherwise subclasses of Iterator won't be able to redefine it.

rbuckton · 2024-04-17T17:14:17Z

src/lib/esnext.iterator.d.ts

+
+declare var Iterator: (abstract new <T>() => NativeIterator<T>) & IteratorConstructor;
+
+// TODO BEFORE MERGING: update all existing IterableIterator-return methods to return NativeIterator


As suggested elsewhere, I would suggest you define NativeIterator as follows in the es2015 libs and then just update all of the IterableIterator references to NativeIterator:

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> { [Symbol.iterator](): NativeIterator<T, TReturn, TNext>; }

Should I split that into a separate PR, or do it here?

rbuckton · 2024-04-17T17:21:11Z

src/lib/esnext.iterator.d.ts

@@ -0,0 +1,133 @@
+interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {


The export {} trick mentioned in my earlier comment can address the abstract next() method definition.

Jack-Works · 2024-04-17T17:44:24Z

src/lib/esnext.iterator.d.ts

@@ -0,0 +1,133 @@
+interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {


The word choose "Native" is not so good. Do we have other options?

@rbuckton had previously suggested Builtin, which I'm also fine with.

Renegade334 · 2024-04-17T17:44:38Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

Edit: I overlooked the module hacking. Very nice.

Jack-Works · 2024-04-17T17:45:44Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

It does not merging the current Iterator interface, but create a new one called NativeIterator

rbuckton · 2024-04-17T20:28:49Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

@bakkot is correct. This should not affect anyone implementing a custom Iterator, or even IterableIterator.

After this is in we may want to add a shortcuts for it to getIterationTypesOfIterableFast and getIterationTypesOfIteratorFast in the checker.

This reverts commit bd51b00. (and subsequent commits)

…es to that

bakkot · 2024-04-18T02:04:00Z

Updated almost exactly following this comment, except that I had BuiltinIterator's Symbol.iterator method pass all the type arguments to the return type; this is necessary so Generator can extend BuiltinIterator (otherwise the Symbol.iterator methods' types conflict).

A consequence of this change is that the type of yield* [] is now void instead of any, because the default for TReturn on BuiltinIterator is void instead of any (as it is for `Iterator).

Also, passing void for the second argument in the return type in next in

declare abstract class Iterator<T> {
    abstract next(value?: undefined): IteratorResult<T, void>;
}

means that classes attempted to implement Iterator have some pain: you can't just return { done: false, value: 0 } from your next because the done value there will be inferred as boolean, which means that the value field has to conform to void so that it can extend the { done: true, value: TReturn } half of IteratorResult<T, void>, which it does not. Also, you can't make an iterator which actually does have a non-void value for the done: true case, which seems like an unnecessary constraint.

I have test cases illustrating the difficulties:

    class BadIterator1 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator1' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { readonly done: false; readonly value: 0; } | { readonly done: true; readonly value: "a string"; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ readonly done: false; readonly value: 0; } | { readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:         Type '{ readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:           Types of property 'value' are incompatible.
!!! error TS2416:             Type 'string' is not assignable to type 'void'.
        if (Math.random() < .5) {
          return { done: false, value: 0 } as const;
        } else {
          return { done: true, value: "a string" } as const;
        }
      }
    }
    
    class BadIterator2 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator2' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { done: boolean; value: number; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:         Types of property 'done' are incompatible.
!!! error TS2416:           Type 'boolean' is not assignable to type 'true'.
        return { done: false, value: 0 };
      }
    }
    
    class BadIterator3 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator3' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { done: boolean; value: number; } | { done: boolean; value: string; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ done: boolean; value: number; } | { done: boolean; value: string; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:         Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:           Types of property 'done' are incompatible.
!!! error TS2416:             Type 'boolean' is not assignable to type 'true'.
        if (Math.random() < .5) {
          return { done: false, value: 0 };
        } else {
          return { done: true, value: "a string" };
        }
      }
    }

I see a few approaches to improving this last issue:

pass any as the second parameter for IteratorResult in the abstract class Iterator (this fixes the BadIterator1 and BadIterator2 cases)
pass unknown as the second parameter for IteratorResult in the above (this fixes the BadIterator1 and BadIterator2 cases)
make abstract class Iterator take a second parameter for TReturn, defaulting to void (this fixes the BadIterator1 and BadIterator2 cases if the user explicitly specifies the second parameter in the classes)

bakkot · 2024-04-18T02:10:08Z

One detail not captured by the types here is that the iterator returned by helpers like map unconditionally has a return method. I guess that's probably worth expressing?

rbuckton · 2024-04-18T16:56:38Z

means that classes attempted to implement Iterator have some pain: you can't just return { done: false, value: 0 } from your next because the done value there will be inferred as boolean, which means that the value field has to conform to void so that it can extend the { done: true, value: TReturn } half of IteratorResult<T, void>, which it does not.

We could probably add TReturn = void, TNext = undefined type parameters to the built in Iterator class without issue, since it's up to the user to implement next anyways.

rbuckton · 2024-04-19T14:04:31Z

One detail not captured by the types here is that the iterator returned by helpers like map unconditionally has a return method. I guess that's probably worth expressing?

Do helpers like map also have a throw method? If so, you could just use Generator for those. If not, we'd have to add another NativeIterator subtype for those.

rbuckton · 2024-04-19T15:29:29Z

Regarding the typing issues, I'm hoping #58243 might help with that.

bakkot · 2024-04-19T15:41:31Z

Do helpers like map also have a throw method? If so, you could just use Generator for those. If not, we'd have to add another NativeIterator subtype for those.

Nope, just return. throw is really a generator-specific thing.

Suggestions for the name? CloseableBuiltinIterator, maybe, since presence of return indicates that you can close the iterator?

rbuckton · 2024-04-19T18:28:00Z

Suggestions for the name? CloseableBuiltinIterator, maybe, since presence of return indicates that you can close the iterator?

I suppose that's fine. @DanielRosenwasser, do you have any thoughts on the name for this type?

rbuckton · 2024-07-19T17:15:17Z

#58243 has merged. I'm happy to align this PR to get it merged today.

src/lib/es2015.iterable.d.ts

src/lib/esnext.iterator.d.ts

rbuckton · 2024-07-19T20:25:27Z

I've made updates to align with the changes in #58243, though it required a few small compromises. If necessary, we can refine this somewhat after its in the beta.

typescript-bot · 2024-07-19T20:29:57Z

The TypeScript team hasn't accepted the linked issue #54481. If you can get it accepted, this PR will have a better chance of being reviewed.

rbuckton · 2024-07-19T21:27:14Z

I realize I neglected to run the user test suite before merging, though I expect it will only call out projects that were already called out by #58243. Running that suite now in case that assumption turns out to be false.

@typescript-bot run dt
@typescript-bot test top400
@typescript-bot test tsserver top100
@typescript-bot user test this
@typescript-bot user test tsserver

typescript-bot · 2024-07-19T21:27:17Z

Starting jobs; this comment will be updated as builds start and complete.

Command	Status	Results
`run dt`	✅ Started
`test top400`	✅ Started
`test tsserver top100`	✅ Started
`user test this`	✅ Started
`user test tsserver`	✅ Started

rbuckton · 2024-07-19T21:31:46Z

Also running benchmarks as I expect I need to put up a follow-up PR that adds a BuiltinIterator optimization to the checker like we do for Iterator

@typescript-bot perf test

typescript-bot · 2024-07-19T21:31:50Z

Starting jobs; this comment will be updated as builds start and complete.

Command	Status	Results
`perf test`	✅ Started

petamoriken · 2024-07-21T22:27:51Z

Why AsyncBuiltinIterator? I think BuiltinAsyncIterator is a more consistent name.

petamoriken · 2024-07-22T04:12:25Z

ref: #59388

bakkot added 2 commits April 16, 2024 19:46

add types for iterator helpers proposal

bd51b00

run baselines

075ce43

typescript-bot added the For Uncommitted Bug PR for untriaged, rejected, closed or missing bug label Apr 17, 2024

format

4449c82

bakkot mentioned this pull request Apr 17, 2024

PSA: potential lib breaking change after iterator-helpers proposal #54481

Closed

rbuckton requested changes Apr 17, 2024

View reviewed changes

Jack-Works reviewed Apr 17, 2024

View reviewed changes

bakkot added 6 commits April 17, 2024 15:55

Revert "add types for iterator helpers proposal"

30d9bc2

This reverts commit bd51b00. (and subsequent commits)

add BuiltinIterator and switch existing IterableIterator uses to that

f5393db

add AsyncBuiltinIterator and switch existing AsyncIterableIterator us…

ebc4e71

…es to that

rebaseline

e748158

add types for iterator helpers

c4a8a08

rebaseline

51124ae

rbuckton mentioned this pull request Apr 18, 2024

Add TReturn/TNext to Iterable et al #58243

Merged

rbuckton reviewed Jul 19, 2024

View reviewed changes

src/lib/es2015.iterable.d.ts Outdated Show resolved Hide resolved

rbuckton reviewed Jul 19, 2024

View reviewed changes

src/lib/esnext.iterator.d.ts Outdated Show resolved Hide resolved

rbuckton reviewed Jul 19, 2024

View reviewed changes

src/lib/esnext.iterator.d.ts Outdated Show resolved Hide resolved

rbuckton reviewed Jul 19, 2024

View reviewed changes

src/lib/esnext.iterator.d.ts Outdated Show resolved Hide resolved

Merge branch 'main' into iterator-helpers

d8282a4

rbuckton approved these changes Jul 19, 2024

View reviewed changes

rbuckton marked this pull request as ready for review July 19, 2024 20:29

rbuckton merged commit 307ff6c into microsoft:main Jul 19, 2024
32 checks passed

bakkot deleted the iterator-helpers branch July 19, 2024 20:52

rbuckton mentioned this pull request Jul 19, 2024

Add BuiltInIterableIterator for built-in iterables with undefined return values #56517

Closed

rbuckton added a commit to DefinitelyTyped/DefinitelyTyped that referenced this pull request Jul 19, 2024

Additional updates following microsoft/TypeScript#58222

3c16c23

petamoriken mentioned this pull request Jul 21, 2024

Add BuiltinIteratorReturn to IterableIterator and AsyncIterableIterator return types microsoft/TypeScript-DOM-lib-generator#1713

Merged

2 tasks

This was referenced Jul 27, 2024

Lib generator needs work to support TS 5.6 uhyo/better-typescript-lib#47

Closed

TS 5.6 Beta BuiltinIteratorReturn: can this be the default type argument of BuiltinIterator's TReturn? #59444

Closed

petamoriken mentioned this pull request Jul 28, 2024

Remove some compat quirk microsoft/TypeScript-DOM-lib-generator#1761

Draft

This was referenced Aug 1, 2024

Disambiguate BuiltinIterator/BuiltinIteratorReturn #59506

Merged

Generators compiled with --target es5 --lib esnext do not support iterator helpers #59513

Closed

xiBread mentioned this pull request Aug 9, 2024

Add support for Iterator Helpers (now stage 3) #59568

Closed

nikolaybotev mentioned this pull request Sep 10, 2024

Iterator Helpers Typing #59908

Closed

bakkot mentioned this pull request Sep 10, 2024

Improve Iterator Helper Type Signatures #59926

Open

8 tasks

chances mentioned this pull request Nov 8, 2024

Type Declarations for Iterator.from denoland/deno#26783

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add types for iterator helpers proposal #58222

add types for iterator helpers proposal #58222

bakkot commented Apr 17, 2024 •

edited

Loading

fatcerberus commented Apr 17, 2024

bakkot commented Apr 17, 2024

Renegade334 commented Apr 17, 2024 •

edited

Loading

bakkot commented Apr 17, 2024

rbuckton commented Apr 17, 2024 •

edited

Loading

rbuckton commented Apr 17, 2024

rbuckton Apr 17, 2024

rbuckton Apr 17, 2024

bakkot Apr 17, 2024

rbuckton Apr 17, 2024

Jack-Works Apr 17, 2024

bakkot Apr 17, 2024

Renegade334 commented Apr 17, 2024 •

edited

Loading

Jack-Works commented Apr 17, 2024

rbuckton commented Apr 17, 2024

bakkot commented Apr 18, 2024

bakkot commented Apr 18, 2024

rbuckton commented Apr 18, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Apr 19, 2024

bakkot commented Apr 19, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Jul 19, 2024

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024 •

edited

Loading

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024 •

edited

Loading

petamoriken commented Jul 21, 2024

petamoriken commented Jul 22, 2024

	readonly [Symbol.toStringTag]: "Iterator";
	readonly [Symbol.toStringTag]: string;


		declare var Iterator: (abstract new <T>() => NativeIterator<T>) & IteratorConstructor;

		// TODO BEFORE MERGING: update all existing IterableIterator-return methods to return NativeIterator

		@@ -0,0 +1,133 @@
		interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {

add types for iterator helpers proposal #58222

add types for iterator helpers proposal #58222

Conversation

bakkot commented Apr 17, 2024 • edited Loading

fatcerberus commented Apr 17, 2024

bakkot commented Apr 17, 2024

Renegade334 commented Apr 17, 2024 • edited Loading

bakkot commented Apr 17, 2024

rbuckton commented Apr 17, 2024 • edited Loading

rbuckton commented Apr 17, 2024

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

bakkot Apr 17, 2024

Choose a reason for hiding this comment

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

Jack-Works Apr 17, 2024

Choose a reason for hiding this comment

bakkot Apr 17, 2024

Choose a reason for hiding this comment

Renegade334 commented Apr 17, 2024 • edited Loading

Jack-Works commented Apr 17, 2024

rbuckton commented Apr 17, 2024

bakkot commented Apr 18, 2024

bakkot commented Apr 18, 2024

rbuckton commented Apr 18, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Apr 19, 2024

bakkot commented Apr 19, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Jul 19, 2024

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024 • edited Loading

rbuckton commented Jul 19, 2024

typescript-bot commented Jul 19, 2024 • edited Loading

petamoriken commented Jul 21, 2024

petamoriken commented Jul 22, 2024

bakkot commented Apr 17, 2024 •

edited

Loading

Renegade334 commented Apr 17, 2024 •

edited

Loading

rbuckton commented Apr 17, 2024 •

edited

Loading

Renegade334 commented Apr 17, 2024 •

edited

Loading

typescript-bot commented Jul 19, 2024 •

edited

Loading

typescript-bot commented Jul 19, 2024 •

edited

Loading