Skip to content

CI: make docker cache download and docker load time out after 10 minutes #90122

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 23, 2021
Merged

CI: make docker cache download and docker load time out after 10 minutes #90122

merged 3 commits into from
Oct 23, 2021

Conversation

hkratz
Copy link
Contributor

@hkratz hkratz commented Oct 21, 2021 •
edited
Loading

@rust-highfive
Copy link
Contributor

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 21, 2021
@hkratz
Copy link
Contributor Author

hkratz commented Oct 21, 2021

On second thought it seems much more likely that docker load hangs.

@rust-log-analyzer

This comment has been minimized.

# to view
@Mark-Simulacrum
Copy link
Member

@bors r+

@bors
Copy link
Collaborator

bors commented Oct 21, 2021

📌 Commit 838e673 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2021
@hkratz
Copy link
Contributor Author

hkratz commented Oct 21, 2021 •
edited
Loading

@Mark-Simulacrum I now strongly suspect that it is docker load. Not sure if we can do anything about it, might be a problem specific to docker in the hosted GA runners.

We could try /usr/bin/timeout -s KILL 600 docker load -i /tmp/rustci_docker_cache (kill after 10 min.), then it would at least fail earlier if stuck just not use the cached docker images.

JohnTitor added a commit to JohnTitor/rust that referenced this pull request Oct 21, 2021
@JohnTitor
Rollup merge of rust-lang#90122 - rusticstuff:ci_curl_max_time, r=Mar…
6717b36 
…k-Simulacrum

CI: make docker cache download attempt time out after 10 minutes

Might help to prevent timeouts we have been seeing:
* https://github.com/rust-lang-ci/rust/runs/3946294286?check_suite_focus=true#step:25:23
* https://github.com/rust-lang-ci/rust/runs/3956799200?check_suite_focus=true#step:25:22
@JohnTitor JohnTitor mentioned this pull request Oct 21, 2021
Closed
@hkratz
Copy link
Contributor Author

hkratz commented Oct 21, 2021

Happened once more: https://github.com/rust-lang-ci/rust/runs/3962928502?check_suite_focus=true#step:25:23

JohnTitor added a commit to JohnTitor/rust that referenced this pull request Oct 21, 2021
@JohnTitor
Rollup merge of rust-lang#90122 - rusticstuff:ci_curl_max_time, r=Mar…
7a2b460 
…k-Simulacrum

CI: make docker cache download attempt time out after 10 minutes

Might help to prevent timeouts we have been seeing:
* https://github.com/rust-lang-ci/rust/runs/3946294286?check_suite_focus=true#step:25:23
* https://github.com/rust-lang-ci/rust/runs/3956799200?check_suite_focus=true#step:25:22
* https://github.com/rust-lang-ci/rust/runs/3962928502?check_suite_focus=true#step:25:23
@JohnTitor JohnTitor mentioned this pull request Oct 21, 2021
Closed
@hkratz hkratz changed the title CI: make docker cache download attempt time out after 10 minutes CI: make docker cache download and docker load time out after 10 minutes Oct 21, 2021
@hkratz
Copy link
Contributor Author

hkratz commented Oct 21, 2021

Adding timeout for docker load as the rollup will time out again anyway so the dl was not to blame: https://github.com/rust-lang-ci/rust/runs/3967892291?check_suite_focus=true

@hkratz
Copy link
Contributor Author

hkratz commented Oct 21, 2021

@rustbot label -S-waiting-on-bors +S-waiting-on-review

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 21, 2021
@Mark-Simulacrum
Copy link
Member

@bors r+ rollup

@bors
Copy link
Collaborator

bors commented Oct 23, 2021

📌 Commit 3e6a695 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 23, 2021
@matthiaskrgr matthiaskrgr mentioned this pull request Oct 23, 2021
Merged
bors added a commit to rust-lang-ci/rust that referenced this pull request Oct 23, 2021
@bors
Auto merge of rust-lang#90188 - matthiaskrgr:rollup-74cwv5c, r=matthi…
cf70855 
…askrgr

Rollup of 11 pull requests

Successful merges:

 - rust-lang#83233 (Implement split_array and split_array_mut)
 - rust-lang#88300 (Stabilise unix_process_wait_more, extra ExitStatusExt methods)
 - rust-lang#89416 (nice_region_error: Include lifetime placeholders in error output)
 - rust-lang#89468 (Report fatal lexer errors in `--cfg` command line arguments)
 - rust-lang#89730 (add feature flag for `type_changing_struct_update`)
 - rust-lang#89920 (Implement -Z location-detail flag)
 - rust-lang#90070 (Add edition configuration to compiletest)
 - rust-lang#90087 (Sync rustfmt subtree)
 - rust-lang#90117 (Make RSplit<T, P>: Clone not require T: Clone)
 - rust-lang#90122 (CI: make docker cache download and `docker load` time out after 10 minutes)
 - rust-lang#90166 (Add comment documenting why we can't use a simpler solution)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 1ea34fb into rust-lang:master Oct 23, 2021
@rustbot rustbot added this to the 1.58.0 milestone Oct 23, 2021
@hkratz hkratz deleted the ci_curl_max_time branch November 6, 2021 20:36
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees

@Mark-Simulacrum Mark-Simulacrum

Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Milestone
1.58.0
Development

Successfully merging this pull request may close these issues.
6 participants
@hkratz @rust-highfive @rust-log-analyzer @Mark-Simulacrum @bors @rustbot