Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Test failure: TestLoopbackAsync #56087

Closed
karelz opened this issue Jul 21, 2021 · 9 comments
Closed

Test failure: TestLoopbackAsync #56087

karelz opened this issue Jul 21, 2021 · 9 comments
Assignees
@MihaZupan
Labels
area-System.Net.Http os-linux Linux OS (any supported distro) test-bug Problem in test source code (most likely) test-run-core Test failures in .NET Core test runs
Milestone
7.0.0

Comments

@karelz
Copy link
Member

karelz commented Jul 21, 2021 •
edited
Loading

Test classes affected:

  • System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Sync
  • System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async
  • System.Net.Http.Functional.Tests.SocksProxyTest_Http2

Failed tests 5/21-9/3 (incl. PRs):

Day Run OS Test type Failure
6/24 PR #54676 Debian.10.Amd64.Open SocksProxyTest_Http1_Sync
6/26 PR #54636 Debian.10.Amd64.Open SocksProxyTest_Http1_Sync
6/29 2x PR #52237 Alpine.312.Amd64.Open both TimeoutException : The operation has timed out.
7/1 20x misc PRs 16x Alpine.312.Amd64.Open + 4x SLES.15.Amd64.Open misc TimeoutException : The operation has timed out.
7/8 PR #55292 Alpine.312.Amd64.Open SocksProxyTest_Http2 TimeoutException : The operation has timed out.
7/16 Official run SLES.12.Amd64.Open SocksProxyTest_Http1_Async SocketException : Connection reset by peer
7/20 PR #53949 Alpine.312.Amd64.Open SocksProxyTest_Http2 TimeoutException : The operation has timed out.
7/21 Official run Centos.8.Amd64.Open SocksProxyTest_Http1_Async SocketException : Connection reset by peer
7/21 Official run Centos.8.Amd64.Open SocksProxyTest_Http1_Async SocketException : Connection reset by peer
7/23 PR #56013 SLES.15.Amd64.Open SocksProxyTest_Http1_Async SocketException : Connection reset by peer
7/28 PR #56316 RedHat.7.Amd64.Open SocksProxyTest_Http1_Sync TimeoutException : The operation has timed out.
7/28 PR #56316 RedHat.7.Amd64.Open SocksProxyTest_Http2 TimeoutException : The operation has timed out.
8/6 PR #57002 SLES.15.Amd64.Open SocksProxyTest_Http1_Async HttpRequestException : Received an invalid status line: 'HTTP/2.0 400 Bad Request'.
8/6 PR #57002 SLES.15.Amd64.Open SocksProxyTest_Http2 Exception : HTTP/1.1 request sent to HTTP/2 connection
8/10 PR #56787 Debian.10.Amd64.Open SocksProxyTest_Http2 SocketException : Connection reset by peer
8/14 PR #57421 Windows.10.Amd64.Server19H1.Open SocksProxyTest_Http1_Async TimeoutException : The operation has timed out.
8/27 PR #57088 SLES.15.Amd64.Open SocksProxyTest_Http1_Async HttpRequestException : Received an invalid status line: 'HTTP/2.0 400 Bad Request'.
8/27 PR #57088 SLES.15.Amd64.Open SocksProxyTest_Http2 Exception : HTTP/1.1 request sent to HTTP/2 connection
9/2 Official run - release/6.0 Debian.9.Amd64.Open SocksProxyTest_Http2 Http2ConnectionException : The HTTP/2 server sent invalid data on the connection.
9/2 Official run - release/6.0 Debian.9.Amd64.Open SocksProxyTest_Http1_Sync TimeoutException : The operation has timed out.
9/3 PR #58199 434x failures - unrelated PR-only failures

Failure SocketException : Connection reset by peer:

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.
-------- System.Net.Sockets.SocketException : Connection reset by peer

   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 817
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 981
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1015
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1028
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1038
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 553
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 60
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 82
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 39
--- End of stack trace from previous location ---
   at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 103
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 67
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1267
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 1593
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 580
----- Inner Stack Trace -----

Alternative failure SocketException : Connection reset by peer (SocksProxyTest_Http2):

System.Net.Http.HttpRequestException : The SSL connection could not be established, see inner exception.
---- System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.
-------- System.Net.Sockets.SocketException : Connection reset by peer

   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:line 92
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1389
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 624
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskCompletionSourceWithCancellation.cs:line 21
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 778
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 943
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 991
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1008
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1027
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 527
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 56
--- End of stack trace from previous location ---
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 58
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 91
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 229
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 37
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 258
   at System.Net.Security.SyncReadWriteAdapter.ReadAsync(Memory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/ReadWriteAdapter.cs:line 53
   at System.Net.Security.SslStream.FillHandshakeBufferAsync[TIOAdapter](TIOAdapter adapter, Int32 minSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1079
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 462
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 381
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 296
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:line 74
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.Receive(Span`1 buffer, SocketFlags socketFlags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 1500
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 254

Alternative failure Exception : HTTP/1.1 request sent to HTTP/2 connection.:

System.Exception : HTTP/1.1 request sent to HTTP/2 connection.

   at System.Net.Test.Common.Http2LoopbackConnection.ReadPrefixAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 107
   at System.Net.Test.Common.Http2LoopbackConnection.CreateAsync(SocketWrapper socket, Stream stream, Http2Options httpOptions, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 82
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Nullable`1 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 95
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 126
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 115
   at System.Net.Test.Common.Http2LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 147
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c.<<TestLoopbackAsync>b__3_1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 60
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 91
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 229
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 37
--- End of stack trace from previous location ---

Alternative failure HttpRequestException : Received an invalid status line: 'HTTP/2.0 400 Bad Request'.:

System.Net.Http.HttpRequestException : Received an invalid status line: 'HTTP/2.0 400 Bad Request'.

   at System.Net.Http.HttpConnection.ParseStatusLine(ReadOnlySpan`1 line, HttpResponseMessage response) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 986
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 599
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 820
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 964
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1003
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1008
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1027
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 527
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 56
--- End of stack trace from previous location ---
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 58
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 91
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 101
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 37
--- End of stack trace from previous location ---

Alternative failure TimeoutException : The operation has timed out.:

System.TimeoutException : The operation has timed out.

   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 37
--- End of stack trace from previous location ---

Alternative failure Http2ConnectionException : The HTTP/2 server sent invalid data on the connection.

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : The request was aborted.
-------- System.Net.Http.Http2ConnectionException : The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
@karelz karelz added area-System.Net.Http test-run-core Test failures in .NET Core test runs labels Jul 21, 2021
@karelz karelz added this to the 6.0.0 milestone Jul 21, 2021
@ghost
Copy link

ghost commented Jul 21, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Tests affected:

  • System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Sync
  • System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async

Failed tests 5/21-7/21 (incl. PRs):

Day Run OS Test type
6/24 PR #54676 Debian.10.Amd64.Open SocksProxyTest_Http1_Sync
6/26 PR #54636 Debian.10.Amd64.Open SocksProxyTest_Http1_Sync
7/16 Official run SLES.12.Amd64.Open SocksProxyTest_Http1_Async
7/21 Official run Centos.8.Amd64.Open SocksProxyTest_Http1_Async
7/21 Official run Centos.8.Amd64.Open SocksProxyTest_Http1_Async

Failure:

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.
-------- System.Net.Sockets.SocketException : Connection reset by peer

   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 817
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 981
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1015
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1028
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1038
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 553
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 60
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 82
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 39
--- End of stack trace from previous location ---
   at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 103
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 67
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1267
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 1593
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 580
----- Inner Stack Trace -----
Author: karelz
Assignees: -
Labels:

area-System.Net.Http, test-run-core
Milestone: 6.0.0

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jul 21, 2021
@karelz
Copy link
Member Author

karelz commented Jul 21, 2021 •
edited
Loading

@geoffkizer @MihaZupan could it be related to some of the last-minute Preview 7 features? E.g. ConnectionPool changes?
Looks like it mostly started on 7/16 and the previous failures might be unrelated ... but I may be wrong.

@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Jul 21, 2021
@geoffkizer
Copy link
Contributor

Seems most likely that there's some race in the teardown logic in the test.

It could be the connection pooling changes, but it doesn't seem super likely to me.

@karelz karelz added the os-linux Linux OS (any supported distro) label Jul 22, 2021
@MihaZupan
Copy link
Member

MihaZupan commented Jul 26, 2021 •
edited
Loading

Failures share the fact that the client connection times out in HttpConnection's InitialFillAsync and that the remote host sent to the proxy was localhost (instead of the IP).
This indicates that the socks handshake succeeded and that something happened on the server side - likely in LoopbackSocksServer when starting to relay to the backend. The loopback is eating all exceptions, we should surface them to get more info into what actually happened.

@MihaZupan MihaZupan mentioned this issue Jul 27, 2021
Merged
@VincentBu

This comment has been minimized.

# to view
@MihaZupan
Copy link
Member

@VincentBu that looks like a completely unrelated failure.

@ManickaP
Copy link
Member

Triage: we're only started collecting more data on this problem, punting.
cc @karelz

@ManickaP ManickaP modified the milestones: 6.0.0, Future Aug 10, 2021
@karelz karelz modified the milestones: Future, 7.0.0 Aug 13, 2021
@karelz karelz added the test-bug Problem in test source code (most likely) label Aug 17, 2021
@ericstj
Copy link
Member

ericstj commented Mar 10, 2022

Hit in release/6.0 https://dev.azure.com/dnceng/public/_build/results?buildId=1653373&view=ms.vss-test-web.build-test-results-tab&runId=45585928&resultId=163163&paneView=debug

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : The request was aborted.
-------- System.IO.IOException : An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.
------------ System.IO.IOException : Unable to read data from the transport connection: Connection reset by peer.
---------------- System.Net.Sockets.SocketException : Connection reset by peer


Stack trace
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1974
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 975
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 527
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 56
--- End of stack trace from previous location ---
   at System.Net.Http.Functional.Tests.SocksProxyTest.<>c__DisplayClass3_0.<<TestLoopbackAsync>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 58
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 91
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 229
   at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs:line 37
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2081
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 945
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 960
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 997
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1963
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 485
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1370
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1267
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 409
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 472
----- Inner Stack Trace -----
Showing filters 1 through 4

@ericstj ericstj mentioned this issue Mar 10, 2022
Merged
@MihaZupan
Copy link
Member

As far as I can tell there has only been 1 failure in the last month, closing for now.

@ghost ghost locked as resolved and limited conversation to collaborators Aug 18, 2022
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees

@MihaZupan MihaZupan

Labels
area-System.Net.Http os-linux Linux OS (any supported distro) test-bug Problem in test source code (most likely) test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
6 participants
@karelz @ericstj @ManickaP @geoffkizer @MihaZupan @VincentBu