Skip to content

Tracking issues for spurious failures related to IPv6 problems in Travis CI's new 2017-Q4 image #47002

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
kennytm opened this issue Dec 25, 2017 · 2 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@kennytm
Copy link
Member

kennytm commented Dec 25, 2017

Upstream issue: travis-ci/travis-ci#8891.

1. Network tests related to IPv6 are failing

Error first happens in #46692. Fixed in #44694 by using the older Travis image instead.

2. Docker failed to start due to "Address already in use"

The old image is reverted in #46924. It was still going well for ~5.5 hours (2 more PRs) after the PR is merged, but when bors try #46941, the jobs all immediately fail with

...
[00:01:45] Sending build context to Docker daemon  499.2kB
[00:01:45] Step 1/10 : FROM ubuntu:16.04
[00:01:45]  ---> 00fd29ccc6f1
[00:01:45] Step 2/10 : RUN apt-get update && apt-get install -y --no-install-recommends   clang   make   file   curl   ca-certificates   python2.7   git   cmake   sudo   bzip2   xz-utils   wget   libssl-dev   pkg-config
[00:01:45]  ---> Using cache
[00:01:45]  ---> f82100b9eb89
[00:01:45] Step 3/10 : COPY scripts/freebsd-toolchain.sh /tmp/
[00:01:45]  ---> Using cache
[00:01:45]  ---> 41afc339a9b0
[00:01:45] Step 4/10 : RUN /tmp/freebsd-toolchain.sh i686
[00:01:45]  ---> Running in 66ec201d700e
[00:01:45] Address already in use
[00:01:45] The command has failed after 5 attempts.

The failure is extremely quick and rapidly burnt out the entire tree. This has been fixed by #47000.

@kennytm kennytm added A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Dec 25, 2017
bors added a commit that referenced this issue Dec 25, 2017
Follow up to #46924, fix massive spurious failure when starting docker

It seems using `fe80::/64` causes `docker start` to fail with "Address already in use". Try to change to a unique local address range instead.

`fe80::/64` is a link-local address (similar to `169.254.0.0/16` in IPv4). Let's try to use a random "private network" address to see whether that fixes things.

cc #47002

r? @aidanhs
@kennytm
Copy link
Member Author

kennytm commented Dec 25, 2017

Timeline of the second bug:

  1. 2017-12-23T14:12:10Z — Revert #46694 (Temporarily use the old Travis image) #46924 rolled up into Rollup of 10 pull requests #46967
  2. 2017-12-23T15:17:33Z — It was noted that the roll up failed in Travis with "Address already in use" while building docker image. Revert #46694 (Temporarily use the old Travis image) #46924 is backed out from the rollup.
  3. 2017-12-25T10:07:15Z — Revert #46694 (Temporarily use the old Travis image) #46924 has been merged.
  4. (MIR borrowck: no "move occurs because X is not Copy` error #46949 and Update compiler_builtins #46971 are merged successfully afterwards).
  5. 2017-12-25T15:46:13Z — Re-do the FreeBSD cross-builds to use Clang and libc++. Fixes #44433 #46941 experienced the beginning of spurious failure.
  6. 2017-12-25T18:37:43Z — Error noted, suspecting to be caused by the IPv6 address. Fix attempt filed at Follow up to #46924, fix massive spurious failure when starting docker #47000.
  7. 2017-12-26T08:31:07Z — Follow up to #46924, fix massive spurious failure when starting docker #47000 is merged, and the bug no longer happens afterwards.

@kennytm
Copy link
Member Author

kennytm commented Feb 12, 2018

Since we are not seeing any more spurious errors related to this image for over a month, I consider the image is now stable and am closing this issue.

@kennytm kennytm closed this as completed Feb 12, 2018
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

1 participant