Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Bazel 8 hangs during build event upload #25232

Open
westrik opened this issue Feb 7, 2025 · 10 comments
Open

Bazel 8 hangs during build event upload #25232

westrik opened this issue Feb 7, 2025 · 10 comments
Labels
more data needed team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged

Comments

@westrik
Copy link

westrik commented Feb 7, 2025

Description of the bug:

I'm seeing an intermittent issue where a build will fail to exit after uploading build events. Seems like Bazel is waiting for background upload tasks even though they have all completed.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This happens intermittently and I haven't found a straightforward repro.

Which operating system are you running Bazel on?

Amazon Linux 2023.6.20250115

What is the output of bazel info release?

release 8.0.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?


If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

Seems potentially related to #21626 but I'm not using remote execution.

Any other information, logs, or outputs that you want to share?

jstack dump

@fmeum
Copy link
Collaborator

fmeum commented Feb 7, 2025

@tjgq @coeuvre We have reports from multiple users about these kinds of hangs. They seem to be caused by --remote_cache_async. In this particular case, it looks like the RemoteExecutionService is waiting on the Phaser even though no uploads are running anymore.

@fmeum
Copy link
Collaborator

fmeum commented Feb 7, 2025

@bazel-io fork 8.1.0

@iancha1992 iancha1992 added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Feb 7, 2025
copybara-service bot pushed a commit that referenced this issue Feb 10, 2025
Users have reported hangs in Bazel's asynchronous remote cache uploads that may be happening because neither `onSuccess` nor `onError` is called on the observer.

Work towards #25232

Closes #25231.

PiperOrigin-RevId: 725235495
Change-Id: I20c3aaa2ee57a52041dea0b3c17356445f2bbc34
bazel-io pushed a commit to bazel-io/bazel that referenced this issue Feb 11, 2025
Users have reported hangs in Bazel's asynchronous remote cache uploads that may be happening because neither `onSuccess` nor `onError` is called on the observer.

Work towards bazelbuild#25232

Closes bazelbuild#25231.

PiperOrigin-RevId: 725235495
Change-Id: I20c3aaa2ee57a52041dea0b3c17356445f2bbc34
github-merge-queue bot pushed a commit that referenced this issue Feb 11, 2025
… uploads (#25251)

Users have reported hangs in Bazel's asynchronous remote cache uploads
that may be happening because neither `onSuccess` nor `onError` is
called on the observer.

Work towards #25232

Closes #25231.

PiperOrigin-RevId: 725235495
Change-Id: I20c3aaa2ee57a52041dea0b3c17356445f2bbc34

Commit
d4c9b92

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 8.1.0 RC4. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=8.1.0rc4. Thanks!

@iancha1992
Copy link
Member

@fmeum @westrik can you please check if Bazel 8.1.0 RC4 fixes the issue? Thanks

@connyay
Copy link

connyay commented Feb 12, 2025

We (@tanium) were seeing frequent hangs with 8.0.1. Hangs seem to be resolved with 8.1.0 rc4.

Thank you!

@iancha1992
Copy link
Member

Fixed. See #25232 (comment)

@westrik
Copy link
Author

westrik commented Feb 19, 2025

Unfortunately I'm still seeing this with 8.1.0rc4 (with --remote_cache_async).

jstack dump: https://gist.github.com/westrik/9da3128bfac9d59ef781681b664d9267

@meteorcloudy
Copy link
Member

/cc @coeuvre @tjgq

@fmeum
Copy link
Collaborator

fmeum commented Feb 20, 2025

Possibly related: #25308 (comment)

@fmeum fmeum reopened this Feb 21, 2025
@iancha1992
Copy link
Member

@bazel-io fork 8.2.0

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
more data needed team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged
Projects
None yet
Development

No branches or pull requests

8 participants