Transport:Waiting for go routines to exit before closing transport

[eshitachandwani]

Fixes: #2869

Previously, clientconn.Close() did not guarantee termination of some go routines upon return.

Now, clientconn.Close() ensures that:

• It waits for the readerDone channel to signal completion of reader go routine.
• It waits for the keepAliveDone channel to signal completion of keepAlive go routine.
• Transports are closed simultaneously, and clientconn.Close() waits for all transports to be closed before returning.

RELEASE NOTES:

• transport: fixed a bug where the transport was closed before all goroutines had exited.
• grpc: transports are closed simultaneously, and clientconn.Close() waits for all transports to be closed before returning.

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 81.75%. Comparing base (8ea3460) to head (59ae8aa).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/transport/http2_client.go	91.30%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7666      +/-   ##
==========================================
- Coverage   81.86%   81.75%   -0.11%     
==========================================
  Files         361      361              
  Lines       27821    27838      +17     
==========================================
- Hits        22775    22760      -15     
- Misses       3847     3871      +24     
- Partials     1199     1207       +8     

Files with missing lines	Coverage Δ
clientconn.go	92.45% <100.00%> (-0.67%)	⬇️
internal/transport/http2_client.go	92.20% <91.30%> (+0.18%)	⬆️

... and 16 files with indirect coverage changes

[aranjans]

Is there any motivation behind replacing the tear down process with goroutines with sync.WaitGroup? Is this tearDown going to be heavy ops, if not i feel earlier approach might be better? WDYT?

[eshitachandwani]

This change was suggested by Doug, because now we wait for all go routines inside transport to exit, so it will take time if we serially wait for all transports to close. So we changed it to do the following:
-we will instruct all transports to close simultaneously

• All the transports will wait for all go routines inside it to exit
• we wait for all transports to close before exiting
  Instead for asking each transport to close serially , as other transport might not even be signaled to close until all resources in previous transport are cleaned up. Since the transports are not dependent on each other, this approach saves time.

[aranjans]

Won't we end up calling t.Close() multiple times in this case? One another instance where we are calling t.Close is this. Thoughts?

[eshitachandwani]

Yes I think t.Close() might be called multiple times but it will execute only once always becuase we check the state of the transport beofre actually executing anything, look here ,

grpc-go/internal/transport/http2_client.go

Line 991 in bcf9171

if t.state == closing {

And also, t.Close was already being called inside the reader go routine , I just changed it to specifically be called only after the t,readerDone channel is closed, or it was going into infinite wait.