Runner stuck in " Not configured" state

[valeramaniukIHM]

Describe the bug
After a restart self-hosted Runner randomly gets stuck in "Not configured'" state. Possibly because of a failure to register with GH caused by a network interruption(?). The runner won't retry the "configure" step, but instead would retry to start even though it's not configured.
Might be similar to #62

To Reproduce
Error affects Runners seemingly at random. After the Runner tries to autoupdate and restarts it may or may not get stuck.
We are running ~30 runners for the org.

Expected behavior
Runner restarts and registers with GH. If it fails to register it retries or fails completely.

Runner Version and Platform

controller: 0.17.0
summerwind/actions-runner:v2.277.1

What's not working?

Starting Runner listener with startup type: service Started listener process An error occurred: Not configured Runner listener exited with error code 2 Runner listener exit with retryable error, re-launch runner in 5 seconds
indefinitely

Runner and Worker's Diagnostic Logs

[2021-04-23 17:20:03Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO HostContext] Well known config file 'Credentials': '/runner/.credentials'
[2021-04-23 17:20:03Z INFO Listener] Runner is built for Linux (X64) - linux-x64.
[2021-04-23 17:20:03Z INFO Listener] RuntimeInformation: Linux 4.14.198-152.320.amzn2.x86_64 #1 SMP Wed Sep 23 23:57:28 UTC 2020.
[2021-04-23 17:20:03Z INFO Listener] Version: 2.277.1
[2021-04-23 17:20:03Z INFO Listener] Commit:
[2021-04-23 17:20:03Z INFO Listener] Culture:
[2021-04-23 17:20:03Z INFO Listener] UI Culture:
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO Listener] Validating directory permissions for: '/runner'
[2021-04-23 17:20:03Z INFO CommandLineParser] Parse
[2021-04-23 17:20:03Z INFO CommandLineParser] Parsing 15 args
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: False
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding Command: configure
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: unattended
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: replace
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding flag: unattended
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: name
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding flag: replace
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'name': 'runners-hi-cpu-q6jhz-f45th'
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: url
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'url': 'https://github.com/xxx-software'
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: token
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'token': ''
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: runnergroup
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'runnergroup': ''
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: labels
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'labels': 'runners-hi-cpu'
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] arg: work
[2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument
[2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True
[2021-04-23 17:20:03Z INFO CommandLineParser] Adding option 'work': '/runner/work'
[2021-04-23 17:20:03Z INFO Listener] Arguments parsed
[2021-04-23 17:20:03Z INFO Runner] ExecuteCommand
[2021-04-23 17:20:03Z INFO ConfigurationStore] currentAssemblyLocation: /runner/bin/Runner.Listener.dll
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO ConfigurationStore] binPath: /runner/bin
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO ConfigurationStore] RootFolder: /runner
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO HostContext] Well known config file 'Runner': '/runner/.runner'
[2021-04-23 17:20:03Z INFO ConfigurationStore] ConfigFilePath: /runner/.runner
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO HostContext] Well known config file 'Credentials': '/runner/.credentials'
[2021-04-23 17:20:03Z INFO ConfigurationStore] CredFilePath: /runner/.credentials
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO HostContext] Well known config file 'MigratedCredentials': '/runner/.credentials_migrated'
[2021-04-23 17:20:03Z INFO ConfigurationStore] MigratedCredFilePath: /runner/.credentials_migrated
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Bin': '/runner/bin'
[2021-04-23 17:20:03Z INFO HostContext] Well known directory 'Root': '/runner'
[2021-04-23 17:20:03Z INFO HostContext] Well known config file 'Service': '/runner/.service'
[2021-04-23 17:20:03Z INFO ConfigurationStore] ServiceConfigFilePath: /runner/.service
[2021-04-23 17:20:03Z INFO CommandSettings] Flag 'help': 'False'
[2021-04-23 17:20:03Z INFO CommandSettings] Flag 'version': 'False'
[2021-04-23 17:20:03Z INFO CommandSettings] Flag 'commit': 'False'
[2021-04-23 17:20:03Z INFO CommandSettings] Flag 'check': 'False'
[2021-04-23 17:20:03Z INFO CommandSettings] Command 'configure': 'True'
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE:
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | ____ _ _ _ _ _ _ _ _ |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | / () || | | | | | / \ | |() ___ _ __ ___ |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | | | | __| || | | | | ' \ / _ \ / | | |/ _ | ' / | |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | || | | || _ | || | |) | / ___ \ (| || | () | | | __ \ |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | _||_|| ||_,|.__/ // __|_||_/|| ||___/ |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | |
[2021-04-23 17:20:03Z INFO Terminal] WRITE: |
[2021-04-23 17:20:03Z INFO Terminal] WRITE: Self-hosted runner registration
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | |
[2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2021-04-23 17:20:03Z INFO ConfigurationManager] ConfigureAsync
[2021-04-23 17:20:03Z INFO ConfigurationStore] IsConfigured()
[2021-04-23 17:20:03Z INFO ConfigurationStore] IsConfigured: False
[2021-04-23 17:20:03Z INFO ConfigurationManager] Is configured: False
[2021-04-23 17:20:03Z INFO CommandSettings] Arg 'url': 'https://github.com/xxx-software'
[2021-04-23 17:20:03Z INFO CommandSettings] Remove url from Arg dictionary.
[2021-04-23 17:20:03Z INFO CommandSettings] Arg 'token': ''
[2021-04-23 17:20:03Z INFO CommandSettings] Remove token from Arg dictionary.
[2021-04-23 17:20:13Z ERR Runner] System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at GitHub.Runner.Listener.Configuration.ConfigurationManager.GetTenantCredential(String githubUrl, String githubToken, String runnerEvent)
at GitHub.Runner.Listener.Configuration.ConfigurationManager.ConfigureAsync(CommandSettings command)
at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command)
[2021-04-23 17:20:13Z ERR Runner] #####################################################
[2021-04-23 17:20:13Z ERR Runner] System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
[2021-04-23 17:20:13Z ERR Terminal] WRITE ERROR: Error while copying content to a stream.
[2021-04-23 17:20:13Z INFO Listener] Runner execution has finished with return code 1

[mumoshu]

@valeramaniukIHM Hey! actions-runner-contoroller greater than 0.18.0 has a "registration timeout" feature that watches if the runner has been registered or and recreate the pod after the registration timeout. Uprading to 0.18.x, I suggest 0.18.2, would fix it.

Beware to correctly upgrade your CRDs before updraging the controller though. See #467 and #468 for more info.

[mumoshu]

Closing this as already resolved for now. Please feel free to reopen or submit another issue if you think it doesn't work even after 0.18.x