Skip to content

ci: debug network failures while downloading awscli from PyPI #63007

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 26, 2019
Merged

ci: debug network failures while downloading awscli from PyPI #63007

merged 1 commit into from
Jul 26, 2019
+27 −1

Conversation

pietroalbini
Copy link
Member

This adds some random debug code to our CI script while downloading awscli, to hopefully pinpoint what's causing the network failures.

r? @Mark-Simulacrum
cc #62967

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 26, 2019
@pietroalbini pietroalbini force-pushed the debug-awscli-install branch 2 times, most recently from e951d8a to 9b0a139 Compare July 26, 2019 11:17
@Mark-Simulacrum
Copy link
Member

@bors r+ p=100

@bors
Copy link
Collaborator

bors commented Jul 26, 2019

📌 Commit 9b0a139bfdf2500d5c57c421b06a2c704b0798bf has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 26, 2019
@pietroalbini
Copy link
Member Author

@Centril feel free to rollup this btw

@pietroalbini pietroalbini mentioned this pull request Jul 26, 2019
Closed
@ethomson
Copy link

The debugging steps in this issue might also be interesting:

dig pypi.org
dig files.pythonhosted.org
openssl s_client -connect pypi.org:443
openssl s_client -connect files.pythonhosted.org:443

@pietroalbini pietroalbini force-pushed the debug-awscli-install branch from 9b0a139 to 81ac354 Compare July 26, 2019 12:30
@pietroalbini
Copy link
Member Author

@bors r=Mark-Simulacrum p=100

@ethomson good call! Added those.

@bors
Copy link
Collaborator

bors commented Jul 26, 2019

📌 Commit 81ac354898a0000e5582776c9aa4ba54110d9842 has been approved by Mark-Simulacrum

@bors
Copy link
Collaborator

bors commented Jul 26, 2019

⌛ Testing commit 81ac354898a0000e5582776c9aa4ba54110d9842 with merge daed3d5d51248a350bb2859bb76ba16ae30daa5f...

@alex
Copy link
Member

alex commented Jul 26, 2019

(👋 from the Python Infra Team, let me know if there's any way I can be helpful -- our TCP termination all happens at Fastly.)

@bors
Copy link
Collaborator

bors commented Jul 26, 2019

💥 Test timed out

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 26, 2019
@pietroalbini
Copy link
Member Author

@bors r-

Uh, it got stuck somewhere, let me investigate.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 26, 2019
@pietroalbini pietroalbini force-pushed the debug-awscli-install branch from 81ac354 to fb9f3e6 Compare July 26, 2019 16:51
@pietroalbini
Copy link
Member Author

Well, we need to close the OpenSSL connection sooner or later...

@bors r=Mark-Simulacrum p=100

@bors
Copy link
Collaborator

bors commented Jul 26, 2019

📌 Commit fb9f3e6 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 26, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 26, 2019
@Centril
Rollup merge of rust-lang#63007 - pietroalbini:debug-awscli-install, …
0222498 
…r=Mark-Simulacrum

ci: debug network failures while downloading awscli from PyPI

This adds some random debug code to our CI script while downloading awscli, to *hopefully* pinpoint what's causing the network failures.

r? @Mark-Simulacrum
cc rust-lang#62967
@Centril Centril mentioned this pull request Jul 26, 2019
Merged
bors added a commit that referenced this pull request Jul 26, 2019
@bors
Auto merge of #63015 - Centril:rollup-ydhpcas, r=Centril
c43753f 
Rollup of 22 pull requests

Successful merges:

 - #62084 (allow clippy::unreadable_literal in unicode tables)
 - #62120 (Add missing type links in documentation)
 - #62310 (Add missing doc links in boxed module)
 - #62421 (Introduce `as_deref` to Option)
 - #62583 (Implement Unpin for all raw pointers)
 - #62692 (rustc: precompute the largest Niche and store it in LayoutDetails.)
 - #62801 (Remove support for -Zlower-128bit-ops)
 - #62828 (Remove vector fadd/fmul reduction workarounds)
 - #62862 (code cleanup)
 - #62904 (Disable d32 on armv6 hf targets)
 - #62907 (Initialize the MSP430 AsmParser)
 - #62956 (Implement slow-path for FirstSets::first)
 - #62963 (Allow lexer to recover from some homoglyphs)
 - #62964 (clarify and unify some type test names)
 - #62970 (ci: gate toolstate repo pushes on the TOOLSTATE_PUBLISH envvar)
 - #62980 (std: Add more accessors for `Metadata` on Windows)
 - #62983 (Remove needless indirection through Rc)
 - #62985 (librustc_errors: Support ui-testing flag in annotate-snippet emitter)
 - #63002 (error_index_generator should output stdout/stderr when it panics.)
 - #63004 (Add test for issue-54062)
 - #63007 (ci: debug network failures while downloading awscli from PyPI)
 - #63009 (Remove redundant `mut` from variable declaration.)

Failed merges:

r? @ghost
@bors
Copy link
Collaborator

bors commented Jul 26, 2019

⌛ Testing commit fb9f3e6 with merge c43753f...

@bors bors merged commit fb9f3e6 into rust-lang:master Jul 26, 2019
@pietroalbini pietroalbini deleted the debug-awscli-install branch July 27, 2019 08:19
@pietroalbini
Copy link
Member Author

Ok, apparently every debug step I added except for pings (which don't work at all on Azure Pipelines) is fine both before and after pip failed to download awscli (build log).

@Mark-Simulacrum pointed out that we're running Ubuntu 16.04's own pip 8.1.1 while version 19.2.1 is available. We could try to land a pip self-update, but since pip is the one having trouble reaching the network I'm not sure how much it would help.

The other "solution" I'm thinking about is to vendor pip wheel awscli on our mirror bucket and skip pip's networking altogether. SInce curl to AWS still works after pip fails to connect that should work, but it would be best to get down to the cause of this.

@alex do you have any other debugging steps (other than ping, curl, dig and openssl s_client) you'd like us to add to the CI configuration?

@alex
Copy link
Member

alex commented Jul 27, 2019

Can you grab a traceroute? I think that's the only item from our usual network access issues template that you don't already have: https://github.com/pypa/warehouse/issues/new?template=access-issues.md

@pietroalbini
Copy link
Member Author

Can you grab a traceroute?

Unfortunately no, I tried a traceroute on my personal test repo and Azure Pipelines just filters all the packets.

@pietroalbini
Copy link
Member Author

Opened a PR to skip PyPI altogether and download the packages from our S3 bucket, which appears to be reachable: #63050. It's unfortunate that I can't drill down more into this, but I'm out of ideas and getting the builds working has way higher priority.

# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees

@Mark-Simulacrum Mark-Simulacrum

Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@pietroalbini @Mark-Simulacrum @bors @ethomson @alex @rust-highfive