Update return rules for null-safety?

[eernstg]

Cf. dart-lang/sdk#41324.

We may wish to revisit the rules about return e; because the rules in the language specification mention Null specifically in one case, and may need to treat Never specifically in other cases).

For instance, Future<Never> f() async => null as dynamic; is a compile-time error with the current rules, with null-safety. But this is because Future<dynamic> is no longer assignable to Future<Never> (so the wrapping with Future gives the new assignability a distinction that the old assignability did not have). In contrast, Never f1() => null as dynamic; is OK (with null-safety as well as without) because dynamic is assignable to Never.

In both cases we could argue that an expression of type dynamic can be trusted to throw or loop (that is, to not complete, with or without a value) just as well as it could be trusted to return an object of an arbitrary type which is required by the context, or to some type that has a specific member with a specific signature. And we do trust the dynamic expression for all those purposes.

So it seems consistent to allow return e; when e has type dynamic, both in a synchronous non-generator function and in an asynchronous non-generator function.

If we do this then we'd need to adjust the language specification to avoid the wrapping in Future<...>. This change should be a no-op semantically for now (same meaning), but it should ensure that the rules are still specifying the intended behavior when we introduce null-safety into the language specification.

@munificent, @lrhn, @leafpetersen, WDYT?

[leafpetersen]

I'm not sure I see any reason to change any of these. I do think we should clean up the errors/warnings around return statements as part of this release, since they're in an odd state. We specified them as warnings to avoid breakage, but then some or all of them got implemented as errors in the analyzer, so currently the analyzer and CFE differ. We should take this opportunity to make all of them consistent.

[eernstg]

The point is that null-safety introduces an anomaly:

// In a legacy library.
int f1() => 42 as dynamic; // OK.
Future<int> f2() async => 42 as dynamic; // OK.

// In a library where null-safety is enabled.
int f3() => 42 as dynamic; // OK.
Future<int> f4() async => 42 as dynamic; // Error.

The intuition is that we always "return" 42 as dynamic, and there is no difference between the situation where we check (statically) that it is OK to cast from dynamic to int, both in the case where we are actually returning it (f1) and in the case where we are using the "returned" value to complete the Future<int> which was returned previously (f2).

However, we have specified the async case in terms of a wrapped type: We require that in the case where flatten(dynamic) is not void (which is true) that Future<flatten(dynamic)> (that is, Future<dynamic>) must be assignable to the return type Future<int>. The latter is not true with null-safety, so we get an error in f4.

But note that we are never performing an assignment from an expression of type Future<dynamic> to a variable of type Future<int>!

So we're performing a cast from dynamic to int in the process where the "returned" value completes the future, which is OK also with null-safety. But this is accidentally specified in terms of a cast from Future<dynamic> to Future<int>, which is not allowed with null-safety.

We might just as well have used the following (arguably more natural) specification:

It is a compile-time error if $s$ is \code{\RETURN{} $e$;},
\flatten{S} is not \VOID,
and \flatten{S} is not assignable to the element type of $f$
(\ref{functions}).

The only reason why we did not do this was that we did not have a definition of the relevant concept when \section{Return} was last updated.

We have the notion of the 'element type of a generator function', and we'd need a corresponding concept of something like the 'element type of an asynchronous non-generator function'. With return type Future<T> or FutureOr<T> that's T, and with return type void or dynamic it is dynamic.

So I'd recommend that we define this new concept, and make that change in \section{Return}. We then avoid making f4 an error, maintain the conceptual consistency (f3 as well as f4 is actually about a cast from dynamic to int, so they should both be an error, or none of them should be an error). We then also avoid breaking code just because we used an approach in the specification which is needlessly convoluted, which produces this error by accident, and which does not reflect the actual semantics.

I closed #913 (a PR with two changes), because the first change was subsumed by existing rules about return statements in the language specification, and the second change was intended to specify that certain errors (here) are indeed errors — but at this point I'd prefer that we change the specification such that they are not errors after all.

[lrhn]

I agree in principle that a return in an async function should work like any other FutureOr<flatten(returnType)> context.

The context for f4 is FutureOr<int>, and that's what we should be downcasting dynamic to (then check whether it's a future and await it if so).

[leafpetersen]

[Edit: Never mind - user error. The below does fail at runtime.]

The context for f4 is FutureOr<int>, and that's what we should be downcasting dynamic to (then check whether it's a future and await it if so).

I don't think you really mean that do you? If so then this program (which currently works) becomes an error:

Future<int> foo() async => (Future<num>.value(3) as dynamic);

void main() async {
  print(await foo());
}

[leafpetersen]

As I said at the high level - I'm not averse to cleaning up the errors and warnings around returns. The spec was added after the fact to try to unify the de facto rules implemented by the two front ends into something consistent and minimally breaking - hence all of the odd exceptions around Null.

If someone wants to take this on (@eernstg?) then what needs to happen is:

• A proposal for the new rules
• Matching updates to the appropriate tests (see tests/language_2/invalid_returns.*), plus possibly new tests covering any new behavior not otherwise covered.
• Once we are agreed, implementation issues for the front ends.

[lrhn]

It's never simple with async functions!

Assume we have an async function with declared return type R, and let T be flatten(R). (We can call this the "element type" of the async function, or something).

First of all: The simplest solution to specify, is to disallow returning Future<T>. Require the user to always write the await if they want to return the result of a future, not return the future directly. That would reduce the problem to ... zero. Context type of the return statement expression is the element type of the function. That is all. If you want to await, you use an explicit await with the same rules as all other awaits. That's both prettier and simpler and quite a lot harder to migrate.
(Not impossible, though, and I hope we'll do it some day).

Assuming we want to keep returning futures, the typing is tricky.

Let's have a return statement of the form return e; where e has static type S.
The run-time behavior is:

• Evaluate e to a value o.
• If o implements Future<T>, await o and let v be the resulting value.
• Otherwise let v be o.
• If o does not implement T, throw a TypeError (this handles implicit downcast from dynamic).
• Complete the returned future with v.

That can be written as something like:

return let tmp = e in tmp is Future<T> ? await tmp : (tmp implicit-downcast-to T);`.

Statically typing that to disallow cases where tmp as T would definitely throw is not easy.
I guess we can simply ignore the tmp is Future<T> case because that one will definitely generate something of type T.
So, we need to detect the cases where the tmp is not implicitly downcastable to T, and that branch is still potentially reachable.

1. We could say that S must be assignable to FutureOr<T>.

That will disallow return Future<dynamic>.value(2); in an async function returning Future<int>, even though return await Future<dynamic>.value(2); is allowed.
Do we want to allow this case?
As currently specified, we will not await that future because it does not implement Future<int>. We don't await all futures, only those where awaiting them will give a valid return value.
Maybe allowing it is not actually desirable.

The static return expression types allowed by this rule, for the element type T, are:

• X, X <: T
• Future<X>, X <: T
• FutureOr<T>, X <: T
• dynamic – Assignable to anything.

That's the same as saying: return e; is valid iff S is assignable to T or flatten(S) is a subtype of T.

2. Or we could say that a return e; where e has type S is allowed iff S is assignable to T or flatten(S) is assignable to T.

That's almost the same as S being assignable to FutureOr<T>, but allows the case above because flatten(Future<dynamic>) is dynamic which is assignable to int.
Which is not necessarily an improvement, since it doesn't match run-time behavior.

The static return expression types allowed by the rule, for the element type T, are:

• X, X <: T
• Future<X>, X <: T
• FutureOr<T>, X <: T
• dynamic – Assignable to anything.
• Future<dynamic>
• FutureOr<dynamic>

These are the types which are either subtypes of T or are dynamic, or which flattens to something which is a subtype of T or is dynamic.

So, let's look at some cases:

import "dart:async";

var f1 = Future<int>.value(0);
var f2 = Future<Future<int>>.value(f1);
var f3 = Future<Future<Future<int>>>.value(f2);
var fo1 = Future<dynamic>.value(2);

Future<int> foo(int x) async {
  if (x == 0) return x;
  if (x == 1) return x as dynamic;
  if (x == 2) return x as FutureOr<int>;
  if (x == 3) return x as FutureOr<dynamic>;
  if (x == 10) return f1;
  if (x == 11) return f1 as dynamic;
  if (x == 12) return f1 as Future<dynamic>; // Disallowed by 1.
  if (x == 13) return f1 as FutureOr<int>; 
  if (x == 14) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (x == 15) return f1 as FutureOr<FutureOr<int>>; // Disallowed by 1 and 2.
  if (x == 16) return f1 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (x == 20) return fo1;  // Disallowed by 1. Throws at runtime.
  if (x == 21) return fo1 as dynamic;
  if (x == 22) return fo1 as FutureOr<dynamic>;  // Disallowed by 1.
  return throw "not";
}

Future<Future<int>> bar(int y) async {
  if (y == 0) return f1; // Valid. Disallowed by analyzer.
  if (y == 1) return f1 as Future<dynamic>; // Disallowed by 1.
  if (y == 2) return f1 as FutureOr<int>; // Disallowed by 1 and 2. Allowed by CFE.
  if (y == 3) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (y == 10) return f2;
  if (y == 11) return f2 as dynamic;
  if (y == 12) return f2 as Future<dynamic>; // Disallowed by 1.
  if (y == 13) return f2 as FutureOr<dynamic>; // Disallowed by 1.
  if (y == 14) return f2 as Future<Future<dynamic>>;  // Disallowed by 1 and 2.
  if (y == 15) return f2 as FutureOr<Future<dynamic>>; // Disallowed by 1 and 2
  if (y == 16) return f2 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (y == 17) return f2 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  return throw "not";
}

Future<FutureOr<int>> baz(int z) async {
  if (z == 0) return z;
  if (z == 1) return z as dynamic;
  if (z == 2) return z as FutureOr<int>;
  if (z == 3) return z as FutureOr<FutureOr<int>>;
  if (z == 10) return f1;
  if (z == 11) return f1 as dynamic;
  if (z == 12) return f1 as Future<dynamic>;  // Disallowed by 1.
  if (z == 13) return f1 as FutureOr<int>;
  if (z == 14) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (z == 15) return f1 as Future<FutureOr<int>>;
  if (z == 16) return f1 as FutureOr<Future<int>>;
  if (z == 17) return f1 as FutureOr<FutureOr<int>>;
  if (z == 18) return f1 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 19) return f1 as FutureOr<Future<dynamic>>; // Disallowed by 1, 2 and Analyzer.
  if (z == 20) return f1 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 30) return f2; // Disallowed by CFE
  if (z == 31) return f2 as dynamic;
  if (z == 32) return f2 as Future<dynamic>; // Disallowed by 1.
  if (z == 33) return f2 as Future<Future<int>>; // Disallowed by CFE
  if (z == 34) return f2 as Future<Future<dynamic>>; // Disallowed by 1 and Analyzer
  if (z == 35) return f2 as FutureOr<dynamic>; // Disallowed by 1.
  if (z == 36) return f2 as FutureOr<Future<int>>;
  if (z == 37) return f2 as Future<FutureOr<int>>;
  if (z == 38) return f2 as FutureOr<FutureOr<int>>;
  if (z == 39) return f2 as FutureOr<Future<dynamic>>; // Disallowed by 1, 2, and Analyzer
  if (z == 40) return f2 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 41) return f2 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 50) return f3; // Disallowed by Analyzer
  return throw "not";
}

Currently the cases without specific comments are accepted by both CFE and analyzer (checked in DartPad). The disallowed by 1/2 are referring to the rules above in a setting with no implicit downcasts except from dynamic (assignable-to means either subtype-of or dynamic)

[eernstg]

Nice analysis, @lrhn!

It matches my perspective on the situation in PR #930, where I've chosen the approach that you call model 2, and it could obviously be changed to model 1 by editing a couple of words.

I introduced the notion of "the future value type" of an async function, which is the same thing as the "element type" that you mention. However, with null-safety it is not exactly the same thing as flatten(R) where R is the declared return type (in particular Future<int>? f() async {...} is allowed unless we introduce some extra errors for certain return types, and that function should have a future value type of int, not int?).

Model 2 does allow for an extra case compared to model 1: we can return an expression of type Future<dynamic> in an async function with future value type T which is not dynamic, and it will not be an error, we await it and check dynamically whether the completion has the required type T. This seems conceptually consistent to me: It is a separate feature that return may await a future, so awaiting those futures that implement Future<U> such that U is assignable to the future value type is the obvious criterion, and that's model 2. We will need to adjust the tools such that they allow this statically, and such that they generate code to await a Future<dynamic>.

We can also return an expression e of type dynamic, so how do we treat the object obtained from evaluation of e if it has type Future<dynamic> or Future<Object?>? In this case there is no static expectation that we might obtain an instance of any particular type, so any reference to assignability (being a static concept) is questionable. It seems clearly wrong to await the Future<Object?> (we do not even have assignability), and it seems inconsistent to treat the Future<Object?> differently from the Future<dynamic> at run time.

So I'd argue that with model 2 we should only await a Future<dynamic> based on the static type, because the run-time treatment of an object o whose type contains dynamic does not otherwise differ from the run-time treatment where those occurrences of dynamic in the run-time type of o have been replaced by another top type.

So the dynamic semantics should continue to check for a Future<T> (where T is the future value type of the function) and await that, and otherwise not await any futures. If it turns out to be a Future<dynamic> completed with a T we still fail at run time (because we did not await that T), and the motivation is that the special powers of dynamic only apply during static analysis.

Conversely, model 1 introduces the conceptual oddity that the object obtained directly by evaluation of the returned expression is treated differently from an object which is awaited: For the former we allow static type dynamic and check the actual type at run-time, for the latter we insist that the static type is safe.

The whole point of this issue is that this is new, and it is a breaking change. Also, the above oddity indicates to me that this breaking change is poorly justified conceptually.

So that's the reason why I chose model 2 when I wrote PR #930. Note that PR #930 fixes an issue with the existing spec text, and is otherwise a no-op (no changes), as long as we consider pre-null-safety Dart; but it prepares the spec for null-safety because it avoids introducing the above mentioned breaking change "by accident" when null-safety is introduced.

[lrhn]

Good point that flatten(R) is not sufficient to find the future value type. The flatten function matches what await does, and await on a Future<T>? is T?, not T. The future value type instead has to find the actual type parameter of the Future which is going to be returned, no matter which supertype of Future is specified as return type.

Model 2 does allow for an extra case compared to model 1: we can return an expression of type Future<dynamic> in an async function with future value type T which is not dynamic, and it will not be an error, we await it and check dynamically whether the completion has the required type T.

We actually do not.

With the currently specified dynamic behavior, we evaluate the expression with static type Future<dynamic> to a value o.
Then, if o is not a Future<T>, we fail unless o is a T.
So, if T is int, and e has static type Future<dynamic>, model 2 allows that, but throws unless the value is actually a Future<int>. If it's a Future<dynamic> containing an int, we do not await it, and it's not an int. That's why model 1 requires the awaited type (the flatten(S)) to be a subtype, because those are the only ones we actually await.

We act like that because we:

• Want to await a future when the value of the future is a T.
• Do not want to await a future when the value of the future might not be a T, but the future itself might is a T.
• Don't want to special case too much to also handle the case where the the future value of the future might not be a T, and the future itself also might not.

So, for the case:

Future<Future<int>> foo(Future<dynamic> f) async {
  return f;
}

we are in a situation where f might or might not be a Future<int> at run-time, and it might or might not contain a Future<int>. Model 2 accepts this. Model 1 does not.
I'm leaning heavily towards model 1 here because it is just inherently unsafe.

If f ends up being a Future<int>, we must not await it.
If f ends up being a Future<Future<int>>, we must await it to succeed
If f is a Future<Object> (Future<Object?> if NNBD), maybe we can await it optimistically, but we currently don't.
I wouldn't. And I wouldn't treat a Future<dynamic> differently from a Future<Object> at run-time. Being dynamic is (or should be) a static concept.
And if f is a Future<Future<Object>>, we also won't await it.

The oddity that the return behaves differently from an await is inherent in the way a return may or may not do an await. For an explicit await, we can determine the resulting expression's static type. For the return, the resulting objects type depends on the run-time type of the expression, and that's really too late to take dynamic into account.

We could change the run-time behavior to be more optimistic about futures.

• Evaluate e to a value o.
• If o is Future<T>, await o and let v be the result.
• Otherwise, if o is T, let v be o.
• Otherwise, if o is Future<X> for some type T <: X, await o and let v be the result.
• Otherwise let v be o.
• If v is not T, a type error occurs.
• The return statement returns the v value.

This is the behavior which matches Model 2. We statically accept dynamic or a subtype of T, or a Future or FutureOr of those. At run-time we need to handle those cases, and only those where dynamic occurs in the static type should be able to generate type errors.
It's more complicated than the specified behavior matching model 1.

I don't think we should ever generate a type error for a future which matches the static type, and which is not awaited. We should just reject those statically.

So, with the current specified run-time behavior, I'd prefer model 1.

[eernstg]

.. we can return an expression of type Future<dynamic> .. we await it ..

We actually do not.

That is, if you're assuming that we make the static changes and do not follow up with any changes in the semantics. They would have to be non-breaking, of course, but allowing an extra case to succeed is non-breaking.

However, there is an underlying conflict: As long as the semantics contains non-trivial behavioral choices based on the dynamic types of objects, it will be difficult to align the dynamic behavior with the static analysis. I believe that's the core issue here.

As a clarifying detour, let's consider a design where the choice to await a future is made at compile-time. We would then have two kinds of async returns, looking the same but statically determined to be one or the other. Let's call the underlying return that doesn't await _return to show the difference below (of course, _return doesn't "return" its operand, it completes the returned future with the value of its operand, and then suspends).

Static await rules

With an async function with a future value type T and a returned expression e of static type S, evaluated to an object r:

• If S is assignable to T: Desugar to _return r when S <: T, and _return r as T when S is dynamic.
• If S implements Future<U>, U assignable to T: Desugar to _return await r, resp. _return (await r) as T.
• If S is FutureOr<U>, U assignable to T: Desugar to if (r is Future<T>) _return await r; else _return r;, resp. the same thing with as T in the else branch.

With these rules, the presence of an await operation is determined statically, which is a useful and clarifying position to consider. Also, they are obviously sound.

We preserve the property that the dynamic choice to await does not rely on any special powers granted to dynamic (at run time that's just another top type), it actually has to be a Future<T> such that we know the wait is worthwhile.

However, the static choice to await can take dynamic into account, similarly to other parts of the static analysis when the type dynamic is encountered.

Extended static await rules

To avoid the breakage implied by the static await rules, we'd need to ensure that we can also await a future which is the result of evaluating a returned expression of type dynamic. Again, T is the future value type of the function, e has static type S, and evaluates to the object r:

• If S <: T: Desugar to _return r.
• If S is dynamic: Desugar to _return (r is Future<T>) ? await r : (r as T).
• If S implements Future<U>, U assignable to T: Desugar to _return await r, resp. _return (await r) as T.
• If S is FutureOr<U>, U assignable to T: Desugar to if (r is Future<T>) _return await r; else _return r;, resp. the same thing with as T in the else branch.

This is obviously still sound, it just allows some additional executions to succeed when S is dynamic. And we still preserve the property that dynamic gets special treatment statically, but not dynamically.

Comparison with today's semantics

Here's the current dynamic semantics again, where T is the future value type of the function:

• Evaluate e to an object r.
• If the dynamic type of r is a subtype of Future<T>, await r to obtain an object v.
• Otherwise let v be r.
• If the dynamic type of v is not a subtype of T, a dynamic type error occurs.
• Complete the returned future with r and suspend.

Let 'xsa' stand for the extended static await rules.

• If S <: T: Same behavior as xsa, except when r has type Future<T> & T (rare).
• If S is dynamic: Same behavior as xsa.
• If S implements Future<U>, U <: T: Same behavior as xsa.
• If S implements Future<dynamic>, T not a top type: Compile-time error if we keep current rules. Running it anyway: current semantics same as xsa when r is Future<T>, otherwise current semantics is dynamic type error and xsa semantics is _return (await r) as T: So some dynamic errors go away, no other changes.
• If S is FutureOr<U>, U <: T: Same behavior as xsa.
• If S is FutureOr<dynamic>, T not a top type: Compile-time error if we keep current rules. Running it anyway: current semantics same as xsa.

So the extended static await rules are a candidate for the dynamic semantics that we could consider: It is non-breaking (with the extremely marginal exception that a Future<T> & T is awaited with the current semantics, but would be used for completion with the new semantics).

• This clarifies the situation in that the presence of an await is determined statically wherever possible.
• The inherently dynamic choices of whether to await are based on Future<T>.
• So the type dynamic consistently gets special treatment statically, and doesn't get special treatment at run time.

The novelty here is simply that we support returning expressions of type Future<dynamic> and FutureOr<dynamic>. This makes sense when considering that the object used to complete the returned future is in focus here, and it removes a couple of anomalies.

So, with the current specified run-time behavior, I'd prefer model 1.

Makes sense. But I believe that the extended static await rules are a useful design point to consider, which would fit well with model 2 ("S is assignable to T or flatten(S) is assignable to T"), and it's less weird than the current approach. ;-)

[leafpetersen]

I read over this. I don't have any strong feelings here. As I understand it, the motivation for re-examining this was the asymmetry from sync to async, in which given a return type of Future<int> return e as dynamic is allowed in sync, but statically rejected in async. As I understand the above, option 2 from @lrhn replaces this with the opposite asymmetry: return e as Future<dynamic> is allowed in async, but statically rejected in sync. I'm also inclined towards option 1.

[eernstg]

given a return type of Future<int> return e as dynamic is allowed in sync,
but statically rejected in async

I wasn't looking at that specific comparison, and return e means something so different in a sync function and in an async function that it is unobvious to me how important it is to make that comparison. For instance, in an async function with return type Future<U> it is allowed to have return u where the static type of u is U, and that's obviously not allowed in a sync function. Returns in sync and async functions are just different.

The point I'm making is that the (arbitrary) choice to specify a rule about return e in an async function f with future value type T in terms of Future<T> rather than T causes a conceptually unjustified breaking change when we change the definition of assignability. I'm just proposing that we avoid introducing that breaking change by accident.

[lrhn]

I agree that the accidental breaking change should be fixed. The question is where the accidental undesired breaking change ends and the deliberate "no implicit downcast" breaking change begins.

Previously we allowed you to return an expression with static type Future<dynamic> in an async function with return type Future<int>. That was because Future<dynamic> was implicitly down-castable to Future<int>, not because dynamic was down-castable to int.

Do we consider that part of the accidental breaking change, or part of the intentional "no implicit downcast" breaking change? If accidental, it should be fixed, and if intentional, it should not.

I consider it intentional. You only get dynamic behavior for expressions with actual static type dynamic.
For await of a Future<dynamic> or FutureOr<dynamic> we always await a future and then give the result a static type of dynamic. Await always flattens.

For return, it's not that simple. We actually have to look at the run-time type of a future before deciding whether to await it or not. Assume that the return statement expression has static type dynamic, and then consider the cases:

• Return type Future<int>, return value type Future<int>. This must work, and we need to fix that.
• Return type Future<dynamic>, return value type Future<dynamic>. This value should be awaited. It's safe to await and it's probably what the author wants.
• Return type Future<dynamic>, return value type Future<Future<dynamic>>. This value should be awaited. It's safe to await and it's probably what the author wants.
• Return type Future<Future<dynamic>>, return value type Future<dynamic>. This should not be awaited. It does not need to be awaited, and the result is not guaranteed to be valid.
• Return type Future<Future<dynamic>>, return value type Future<Future<dynamic>>. This should be awaited.

Based on the simple rule that you should await a future when the result is guaranteed to be useful, and you must not await a future when the result is not guaranteed to be useful (and assuming that the static rules will prevent cases where awaiting might be useful, and not awaiting will not).

The static type of the expression and the static return type are not enough to decide whether to await. The run-time type of the value is also not enough, it is the interaction between the declared return type and the run-time value type that decides whether awaiting is correct.

So while we are here, we should also make sure that whatever we change the rules to is precisely what we want.

I do think the run-time semantics are fine: For a return type of Future<R>, evaluate e to a value o. Then first await that if it it implements Future<S>, S <: R. Then return the result if it implements a subtype of R.

The "Model 1" matches this behavior, so I think that's what we should use.

(Although I would highly prefer to just never await implicitly, and always require an explicit await, but that might be too breaking, and too late in the NNBD-flow to add it now).

[eernstg]

@lrhn wrote:

For await of a Future<dynamic> or FutureOr<dynamic> we always await
a future and then give the result a static type of dynamic. Await always flattens.

We may await a future—an expression of type FutureOr<dynamic> may evaluate to 42. But I agree that await e always awaits when e evaluates to a future, and the static type of await e is flatten(S) when e has type S, no exceptions.

That was because Future<dynamic> was implicitly down-castable to
Future<int>, not because dynamic was down-castable to int.

Well, that is the accident! ;-)

Even with return e, where the static type of e is Future<dynamic>, in an async function with future value type int, we will never actually return a Future<dynamic>. What we will do is to obtain an object and use that to complete the Future<int> which was already returned. Which means that the situation is similar to int i = dynamicExpression;.

For the dynamic semantics we do have two reasonable approaches:

• (Current semantics) Evaluate e to an object v, throw unless v is Future<int>, otherwise await v and complete with the obtained int.

• (Extended static await rules) Evaluate e to an object v, await v to an object w, throw unless w is int, and complete with w.

I think the second makes more sense: We use the static type Future<dynamic> to determine that an await is definitely relevant (making the presence/absence of await statically known whenever possible is the main point of the static await rules), and we use the statically known type of the completion object, namely dynamic, to determine that a dynamically checked downcast is justified.

The only difference is that the current semantics will throw in some cases where the extended static await rules will allow the computation to succeed. Of course, it is no surprise that there are some cases where both dynamic semantics will end in a dynamic error.

So it is true that there will be some cases where a future is awaited, and the obtained object is rejected by a dynamic type check. But the alternative is not that we could have had a successful execution, allowing us to conclude that it was better to avoid awaiting anything; the alternative is that the program throws immediately when the future dynamically fails to give a static guarantee that the awaited object will work. I just don't think that's going to make anybody more happy.

• Return type Future<int>, return value type Future<int>. This must work,
  and we need to fix that.

Why would we need to fix anything? The previous specification wording as well as all proposals will ensure that the Future<int> is awaited, and the given int (or thrown object, but let's ignore them because they don't matter for the types) is used to complete the returned Future<int>.

But maybe it's supposed to be the future value type which is Future<int>, rather than the return type (and then the return type could be, say, Future<Future<int>>)? That's the example that I characterize as a spec bug in #929.

• Return type Future<dynamic>, return value type Future<dynamic>.
  This value should be awaited. It's safe to await and it's probably what the author wants.

It would be awaited with all the proposals, because the future value type is dynamic, and the result of evaluating the returned expression is guaranteed to be a Future<dynamic>. So: no problem.

• Return type Future<dynamic>, return value type Future<Future<dynamic>>.
  This value should be awaited. It's safe to await and it's probably what the author wants.

Same situation, awaited in all proposals, no problem.

• Return type Future<Future<dynamic>>, return value type Future<dynamic>.
  This should not be awaited. It does not need to be awaited, and the result is not
  guaranteed to be valid.

Agreed: In an async function with return type Future<U> and a return e where the static type of e is U, we should never await. This is again true for all proposals, no problem.

• Return type Future<Future<dynamic>>, return value type
  Future<Future<dynamic>>. This should be awaited.

Again ensured by all proposals, no problem.

So there are no problems relative to these expected properties, no matter which dynamic semantics we'd adopt.

Also, in every case where we await, the static type of the returned expression is always a subtype of Future<T> where T is the future value type of the function, so the behavior is identical for both choices of dynamic semantics (because the dynamic check for .. is Future<T> in the semantics has a statically known outcome).

To summarize:

I believe that we can land #930 (which specifies model 2, because that's the one that isn't a breaking change, pre-null-safety). We may or may not wish to change it to model 1 when we introduce null-safety. This will be a breaking change, but null-safety breaks lots of things.

However, I still think that model 2 is more consistent, and pragmatically more useful, and if we do keep model 2 for code with null-safety then we may wish to use the 'extended static await' semantics, in order to make the presence/absence of await more predictable, and in order to allow a few more program executions to succeed where they would otherwise incur a dynamic error.

[natebosch]

For a return statement of the form return e;, e must have a static type S such that flatten(S) is assignable to flatten(R).

That's not the same as Future<S> being assignable to R because Future<dynamic> is not downcastable to Future<int> any more, but dynamic is downcastable to int. Any new errors here are new and introduced by removing implicit downcasts.

Would this be less confusing for users and our implementations if dynamic isn't allowed to implicitly downcast?

[eernstg]

I'm pretty sure that's a yes. ;-)

[lrhn]

I wrote a comment on the spec CL, so I'll just restate it here for completeness:

I like rules that are effectively statically deciding whether to await (as far as possible).

Consider "generalized static await" rules (to give them a name), where the rules are in priority order, so read an "otherwise" before all but the first.

1. S <: T ↦ _return r;
2. S is dynamic ↦ _return r is Future<T> ? await r : r as T;
3. S implements Future<T> ↦ _return await r;
4. S implements Future<dynamic> ↦ _return (await r) as T;
5. S is FutureOr<U> and U <: T ↦ _return r is Future<U> ? await r : r;
6. S is FutureOr<dynamic> _return (r is Future<dynamic> ? await r : r) as T;
7. It's a static type error.

The type we are checking the FutureOr<U>/FutureOr<dynamic> against is one of the parts of the union type (Future<U> or Future<dynamic>). This guarantees that we are actually deconstructing the union type, and not accidentally cutting along some other division line and getting accidental matches.

The "S is dynamic" is special, as usual. That's really a heuristic that handle cases like T being Object and r being a Future<Object> meaningfully. It doesn't optimistically try to await a Future<dynamic> again, that dynamic was not there at compile-time.

I think the rules can be written shorter as:

1. S <: T ↦ _return r as T;
2. Otherwise, S is dynamic ↦ _return r is Future<T> ? await r : (r as T);
3. Otherwise, R = flatten(S) and R assignable to T ↦ _return (r is Future<R> ? await r : r) as T;
4. Otherwise it's a static type error.

with some of the tests and casts being redundant in some cases.

So I propose those rules for the Null Safe semantics of async return.

They're static, and feel more orthogonal to me than the "extended static await" rules.

They only do implicit down-casts when there is a dynamic involved, and only on the value that is actually statically typed as dynamic.

The breaking case is, as usual:

Future<Future<int>> foo() async {
  Future<dynamic> x = Future<int>.value(42);
  return x;
}

because the Future<dynamic> case awaits any future. I think it's ... reasonable. We treat all members of the type consistently. The static type is Future<dynamic> is not assignable to Future<int>, so to treat them consistently, we need to await all of them. If the code really meant FutureOr<dynamic>, it should have said so.

[eernstg]

I like it! ;-)

[eernstg]

Closing: New rules landed as #941, #948, tests landed as https://dart-review.googlesource.com/c/sdk/+/145587.