GracefulTermination does not take response entity streams into account

[jrudolph]

As observed in #3139 (comment), graceful termination does not take response entity streams into account. I.e. those streams could still be running when the deadline triggers. What currently happens is that due to

akka-http/akka-http-core/src/main/scala/akka/http/impl/engine/server/ServerTerminator.scala

Line 264 in 27d987f

if (!pendingUserHandlerResponse) completeStage()

, the top of the server stack is immediately completed on graceful termination (i.e. even before the deadline). However, the controller prevents that this completion is propagated to the connection. Instead, after some time (related to server.stream-cancellation-delay) also cancellation arrives at the controller stage in which case the rest of the stack is completed. There are a few issues with that:

• The stack should fail (where it just regularly completes currently), so that the connection is abandoned and the peer knows that something bad happened. Especially, for CloseDelimited entities, the stream will otherwise be silently truncated.
• The stack should be completed only after the deadline and regardless of the stream-cancellation-delay or lingerTimeout.
• (incidentally) the controller should be reviewed, why it closes the connection regularly after it has received cancellation from the user side

[johanandren]

Guessing there is something left here given that it was pushed across previous milestones

[zvuki]

Hi all, I am trying to take advantage of graceful shutdown using the code from the docs https://doc.akka.io/docs/akka-http/current/server-side/graceful-termination.html

val bindingFuture = Http().newServerAt("localhost", 8080).bind(routes)
  .map(_.addToCoordinatedShutdown(hardTerminationDeadline = 10.seconds))

I can't make it work though, it seems the connections are being dropped right away, no matter what deadline I set:

Oct 15, 2021 @ 16:18:34.106 -05:00    Aborting tcp connection to /127.0.0.1:41870 because of upstream failure: akka.http.impl.engine.server.ServerTerminationDeadlineReached: 
Server termination deadline reached, shutting down all connections and terminating server...
akka.http.impl.engine.server.GracefulTerminatorStage$$anon$1.installTerminationHandlers(ServerTerminator.scala:265)
akka.http.impl.engine.server.GracefulTerminatorStage$$anon$1.$anonfun$preStart$1(ServerTerminator.scala:211)
akka.http.impl.engine.server.GracefulTerminatorStage$$anon$1.$anonfun$preStart$1$adapted(ServerTerminator.scala:209)

Oct 15, 2021 @ 16:18:34.105 -05:00    [terminator] Initializing termination of server, deadline: 10.00 s

I can confirm that I have ~5 in-flight requests when it happens.