Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Spurious 30 minutes timeout on Travis CI #56112

Closed
kennytm opened this issue Nov 20, 2018 · 6 comments
Closed

Spurious 30 minutes timeout on Travis CI #56112

kennytm opened this issue Nov 20, 2018 · 6 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@kennytm
Copy link
Member

kennytm commented Nov 20, 2018

Symptom: The build just stops in the middle without any reason, and then got killed due to 30-minutes timeout.

Previous instances:

Time PR Log Last logged action
2018-11-20T07:24:15Z #55663 (comment) dist-mipsel64-linux Compile stage0-rustc
2018-11-14T07:08:11Z #55943 (comment) arm-android Download docker cache
2018-11-13T12:22:23Z #55912 (comment) dist-i686-freebsd Clone submodules
2018-11-13T11:48:10Z #55672 (comment) dist-x86_64-linux Clone submodules
2018-11-13T07:04:10Z #55912 (comment) dist-mips-linux (not even booted up)
2018-11-13T06:32:05Z #55136 (comment) i686-gnu Test liballoc
2018-11-07T22:06:07Z #55717 (comment) dist-armhf-linux Build cargo
2018-11-07T09:14:09Z #55717 (comment) x86_64-apple Test debuginfo
2018-11-07T07:14:11Z #55510 (comment) arm-android Download docker cache
@kennytm kennytm added A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC labels Nov 20, 2018
@alexcrichton
Copy link
Member

alexcrichton commented Nov 21, 2018 •
edited
Loading

Another instance:

@kennytm
Copy link
Member Author

kennytm commented Feb 22, 2019

These errors become more serious recently (probably just being masked by worse errors before).

Time PR Log Duration Last action
2019-01-03T09:38:05Z #55986 (comment) x86_64-gnu-aux 2h18m19s Test cargo
2019-01-11T06:48:10Z #57503 (comment) x86_64-gnu-full-bootstrap 37m40s Compile stage0-rustc
2019-01-12T02:00:14Z #51487 (comment) i686-apple 42m01s Compile stage0-rustc
2019-01-12T06:26:10Z #57532 (comment) x86_64-gnu-full-bootstrap 1h47m14s Compile stage2-rustc
2019-01-12T07:06:07Z #57532 (comment) x86_64-gnu-llvm-6.0 36m21s Compile stage0-rustc
2019-01-15T04:06:21Z #56044 (comment) dist-s390x-linux 30m00s (not even booted up)
2019-01-22T04:14:10Z #56221 (comment) x86_64-apple 35m27s make prepare
2019-01-29T16:58:10Z #57808 (comment) arm-android 32m26s Download Docker cache
2019-01-31T22:56:08Z #58024 (comment) dist-x86_64-apple-alt 39m12s make prepare
2019-02-05T05:14:11Z #57851 (comment) dist-x86_64-apple-alt 39m24s make prepare
2019-02-05T14:38:15Z #58162 (comment) dist-x86_64-linux-alt 31m04s Download Docker cache
2019-02-08T13:34:07Z #58294 (comment) dist-x86_64-apple 39m19s make prepare
2019-02-08T14:16:11Z #58161 (comment) dist-x86_64-apple 39m25s make prepare
2019-02-12T15:52:09Z #58341 (comment) dist-android 41m12s Download Docker cache
2019-02-15T15:46:09Z #58416 (comment) dist-x86_64-apple-alt 39m33s make prepare
2019-02-15T16:20:08Z #58470 (comment) dist-android 31m31s Download Docker cache
2019-02-15T21:48:17Z #58495 (comment) arm-android 31m00s Download Docker cache
2019-02-15T22:22:07Z #58416 (comment) arm-android 31m13s Download Docker cache
2019-02-18T02:56:08Z #58543 (comment) dist-x86_64-apple 39m23s Clone submodules
2019-02-19T06:44:08Z #58566 (comment) x86_64-gnu-llvm-6.0 30m44s Clone submodules
2019-02-19T07:20:10Z #56470 (comment) test-various 32m41s make prepare
2019-02-19T19:06:40Z #58537 (comment) dist-various-2 42m32s Download Docker cache
2019-02-20T06:34:08Z #58056 (comment) arm-android 53m57s Compile stage0-rustc
2019-02-20T07:10:10Z #58578 (comment) x86_64-gnu 30m58s Clone rust-lang/rust
2019-02-21T10:20:10Z #58597 (comment) dist-i586-gnu-i586-i686-musl 1h50m56s Test liballoc
2019-02-21T11:10:13Z #58535 (comment) dist-i686-apple 48m12s Compile stage0-rustc
2019-02-21T11:44:13Z #58597 (comment) dist-android 31m55s Download Docker cache
2019-02-21T20:52:20Z #58592 (comment) dist-android 33m19s Download Docker cache
2019-02-21T21:28:13Z #58592 (comment) dist-x86_64-linux 31m53s Download Docker cache
2019-02-22T10:08:08Z #57364 (comment) dist-x86_64-linux 32m57s Download Docker cache

@kennytm kennytm mentioned this issue Feb 22, 2019
Merged
@Mark-Simulacrum
Copy link
Member

I'm wondering if we could add the "print building - timestamp" script at the top-level. We do it for make (e.g., curl/gcc/openssl builds) but it seems like it at least would let us figure out if this is literally the builder "stopping" or if we're hanging in rustc...

bors added a commit that referenced this issue Feb 23, 2019
@bors
Auto merge of #58655 - kennytm:download-docker-cache-with-timeout, r=…
bcfb5e8 
…alexcrichton

Added a connection timeout and speed threshold when downloading the Docker cache

This is an attempt to fix one possible cause of #56112. Similar to #52846, this changed the download docker-cache command to fail if it cannot connect or falls below 10 bytes/s after 30 seconds, so it could be retried sooner.
This was referenced Feb 23, 2019
Merged
Merged
@kennytm
Copy link
Member Author

kennytm commented Feb 26, 2019

#57882 (comment)

Adding -y 30 -Y 10 --connect-timeout 30 from #58655 does not completely fix the 30 minute timeout when fetching docker cache 😕

@Centril Centril mentioned this issue Feb 27, 2019
Merged
@crlf0710
Copy link
Member

Now that we're using Azure Pipelines, i guess this can be closed now? @kennytm @Centril

@Centril
Copy link
Contributor

Centril commented Jul 28, 2019

Indeed this issue seems moot. :)

@Centril Centril closed this as completed Jul 28, 2019
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@alexcrichton @kennytm @crlf0710 @Centril @Mark-Simulacrum