xds/xdsclient: Fix flaky test TestLRSClient

[arjan-bal]

Fixes: #7241

Background

The test is sometimes getting a second load report before the client closes the stream. The test is setting the load report interval to 50 ms here:

grpc-go/xds/internal/xdsclient/loadreport_test.go

Lines 106 to 109 in cfd14ba

	Resp: &v3lrspb.LoadStatsResponse{
	SendAllClusters: true,
	LoadReportingInterval: &durationpb.Duration{Nanos: 50000000},
	},

In the final assertion of the test, the test is checking that the very next request present in the fake server's request channel is the client cancellation:

grpc-go/xds/internal/xdsclient/loadreport_test.go

Lines 136 to 138 in cfd14ba

	if u, err := fs2.LRSRequestChan.Receive(ctx); err != nil || status.Code(u.(*fakeserver.Request).Err) != codes.Canceled {
	t.Errorf("unexpected LRS request: %v, %v, want error canceled", u, err)
	}

When the the test takes more than 50 ms b/w sending the first LRS response with the load report interval and this final assertion, another load report can be sent by the client here:

grpc-go/xds/internal/xdsclient/transport/loadreport.go

Lines 136 to 150 in cfd14ba

	func (t *Transport) sendLoads(ctx context.Context, stream lrsStream, clusterNames []string, interval time.Duration) {
	tick := time.NewTicker(interval)
	defer tick.Stop()
	for {
	select {
	case <-tick.C:
	case <-ctx.Done():
	return
	}
	if err := t.sendLoadStatsRequest(stream, t.lrsStore.Stats(clusterNames)); err != nil {
	t.logger.Warningf("Writing to LRS stream failed: %v", err)
	return
	}
	}
	}

Repro

To make the test fail with a similar error message (loadreport_test.go:139: unexpected LRS request: &{ <nil>}, <nil>, want error canceled) add a time.Sleep(100 * time.Millisecond) just after receiving the first LRS request on the server here:

grpc-go/xds/internal/xdsclient/loadreport_test.go

Lines 113 to 116 in cfd14ba

	u, err := fs2.LRSRequestChan.Receive(ctx)
	if err != nil {
	t.Fatalf("unexpected LRS request: %v, %v, want error canceled", u, err)
	}

Fix

This PR keeps reading from fs2.LRSRequestChan.Receive until we get an error due to context expiration or see the expected client context cancellation response.

RELEASE NOTES: None

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.83%. Comparing base (b45fc41) to head (7c0505d).
Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7559      +/-   ##
==========================================
- Coverage   82.07%   81.83%   -0.25%     
==========================================
  Files         360      361       +1     
  Lines       27533    27816     +283     
==========================================
+ Hits        22599    22764     +165     
- Misses       3759     3855      +96     
- Partials     1175     1197      +22     

see 47 files with indirect coverage changes