Deadlock detected with "-Z threads=8"

[0xdeafbeef]

Code

I have tried to measure #117755 on a 32 core machine and got this ice after several tries.
Steps to reproduce:

cd $(mktemp -d)
git clone https://github.com/rustsec/rustsec.git
cd rustsec/cargo-audit
git checkout b6baecc0ea4e2d115e4e10b10c2196b33d42c1da
RUSTFLAGS="-Z threads=8" cargo +nightly build --timings

I have got it 2 times in a row, so it should be reproducible.

Meta

rustc --version --verbose:

rustc 1.73.0 (cc66ad468 2023-10-03)
binary: rustc
commit-hash: cc66ad468955717ab92600c770da8c1601a4ff33
commit-date: 2023-10-03
host: x86_64-unknown-linux-gnu
release: 1.73.0
LLVM version: 17.0.2

Error output

   Compiling winnow v0.5.14
thread '<unnamed>' panicked at compiler/rustc_query_system/src/query/job.rs:543:9:
   Compiling aho-corasick v1.0.2
deadlock detected
stack backtrace:
   0:     0x7f4d92d69c9c - std::backtrace_rs::backtrace::libunwind::trace::h8709aeb3a36f95da
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x7f4d92d69c9c - std::backtrace_rs::backtrace::trace_unsynchronized::heb9f0713a26f337c
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f4d92d69c9c - std::sys_common::backtrace::_print_fmt::h1ed75bc02e1d9ee5
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7f4d92d69c9c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hddc1fcb1d86cbcd2
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f4d92dcbec0 - core::fmt::rt::Argument::fmt::h19337aa9a2c9da83
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/core/src/fmt/rt.rs:142:9
   5:     0x7f4d92dcbec0 - core::fmt::write::h2d162b4d9dda182b
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/core/src/fmt/mod.rs:1117:17
   6:     0x7f4d92d5dbbf - std::io::Write::write_fmt::hcca591aa27610910
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/io/mod.rs:1763:15
   7:     0x7f4d92d69a84 - std::sys_common::backtrace::_print::h90b2688e948a362e
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7f4d92d69a84 - std::sys_common::backtrace::print::h75b0cf3671567509
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7f4d92d6c717 - std::panicking::default_hook::{{closure}}::h6694607f9ccfc6ea
  10:     0x7f4d92d6c47f - std::panicking::default_hook::h4c80dc8b62887255
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/panicking.rs:292:9
  11:     0x7f4d95a8cee0 - std[d2e36247d228c058]::panicking::update_hook::<alloc[74ba046d4657710e]::boxed::Box<rustc_driver_impl[741ff592be011714]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7f4d92d6ce58 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hf3ff6ce13f580837
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/alloc/src/boxed.rs:2021:9
  13:     0x7f4d92d6ce58 - std::panicking::rust_panic_with_hook::h5ad0da38d014c68e
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/panicking.rs:783:13
  14:     0x7f4d92d6cb79 - std::panicking::begin_panic_handler::{{closure}}::hdc6d3bffdbbdcfdd
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/panicking.rs:649:13
  15:     0x7f4d92d6a166 - std::sys_common::backtrace::__rust_end_short_backtrace::h80bfdfe497702363
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys_common/backtrace.rs:170:18
  16:     0x7f4d92d6c912 - rust_begin_unwind
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/panicking.rs:645:5
  17:     0x7f4d92dc85e5 - core::panicking::panic_fmt::hf6ef87080f7e8766
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/core/src/panicking.rs:72:14
  18:     0x7f4d962c0b35 - rustc_query_system[7526d12203c43316]::query::job::deadlock
  19:     0x7f4d95a87edc - std[d2e36247d228c058]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[6c8618f6eb06e019]::util::run_in_thread_pool_with_globals<rustc_interface[6c8618f6eb06e019]::interface::run_compiler<core[916dc932853bc0a1]::result::Result<(), rustc_span[3b1f2782555e1c8c]::ErrorGuaranteed>, rustc_driver_impl[741ff592be011714]::run_compiler::{closure#1}>::{closure#0}, core[916dc932853bc0a1]::result::Result<(), rustc_span[3b1f2782555e1c8c]::ErrorGuaranteed>>::{closure#2}::{closure#1}, ()>
  20:     0x7f4d95a8ed71 - <<std[d2e36247d228c058]::thread::Builder>::spawn_unchecked_<rustc_interface[6c8618f6eb06e019]::util::run_in_thread_pool_with_globals<rustc_interface[6c8618f6eb06e019]::interface::run_compiler<core[916dc932853bc0a1]::result::Result<(), rustc_span[3b1f2782555e1c8c]::ErrorGuaranteed>, rustc_driver_impl[741ff592be011714]::run_compiler::{closure#1}>::{closure#0}, core[916dc932853bc0a1]::result::Result<(), rustc_span[3b1f2782555e1c8c]::ErrorGuaranteed>>::{closure#2}::{closure#1}, ()>::{closure#1} as core[916dc932853bc0a1]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
   Compiling object v0.31.1
  21:     0x7f4d92d77bc5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hee756f0d76156df1
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/alloc/src/boxed.rs:2007:9
  22:     0x7f4d92d77bc5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc26ba9eda2105aea
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/alloc/src/boxed.rs:2007:9
  23:     0x7f4d92d77bc5 - std::sys::unix::thread::Thread::new::thread_start::h9f04a0ee42338ce0
                               at /rustc/fdaaaf9f923281ab98b865259aa40fbf93d72c7a/library/std/src/sys/unix/thread.rs:108:17
  24:     0x7f4d92b43897 - start_thread
  25:     0x7f4d92bca6bc - clone3
  26:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please attach the file at `/home/odm3n/.cargo/registry/src/index.crates.io-6f17d22bba15001f/canonical-path-2.0.2/rustc-ice-2023-11-09T16_59_40-696374.txt` to your bug report

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C linker=/usr/bin/clang -Z threads=8

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
deadlock handler panicked, aborting process
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling ring v0.16.20
   Compiling uluru v3.0.0
   Compiling backtrace v0.3.68
   Compiling rustls-pemfile v1.0.3
   Compiling clap_lex v0.2.4
   Compiling unicode-normalization v0.1.22
   Compiling indexmap v2.0.0
   Compiling bstr v1.6.0
   Compiling crossbeam-channel v0.5.8
   Compiling rustls-native-certs v0.6.3
   Compiling tracing-log v0.1.3
error: could not compile `canonical-path` (lib)

Caused by:
  process didn't exit successfully: `/home/odm3n/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name canonical_path --edition=2018 /home/odm3n/.cargo/registry/src/index.crates.io-6f17d22bba15001f/canonical-path-2.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=198 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=0e29fdb8b83c99cf -C extra-filename=-0e29fdb8b83c99cf --out-dir /tmp/tmp.2hv7B581Tu/rustsec/target/debug/deps -C linker=/usr/bin/clang -L dependency=/tmp/tmp.2hv7B581Tu/rustsec/target/debug/deps --cap-lints allow -Z threads=8` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
      Timing report saved to /tmp/tmp.2hv7B581Tu/rustsec/target/cargo-timings/cargo-timing-20231109T165939Z.html

rustc-ice-2023-11-09T17_04_29-702009.txt
rustc-ice-2023-11-09T16_59_40-696374.txt

[0xdeafbeef]

searched nightlies: from nightly-2023-11-03 to nightly-2023-11-09
regressed nightly: nightly-2023-11-07
searched commit range: fee5518...189d6c7
regressed commit: 189d6c7

bisected with cargo-bisect-rustc v0.6.7

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

RUSTFLAGS="-Z threads=8" cargo bisect-rustc --start=2023-11-03 --end=2023-11-09 --prompt

[0xdeafbeef]

@rustbot modify labels: +A-parallel-queries

[mucinoab]

@rustbot modify labels: +WG-compiler-parallel

[SparrowLii]

Can you reproduce this bug using the latest nightly version to get the new error output?

[0xdeafbeef]

Can you reproduce this bug using the latest nightly version to get the new error output?

rustc-ice-2023-11-29T09_41_43-1429537.txt

[0xdeafbeef]

@SparrowLii I can build rustc from the sources if you need to check something.

[SparrowLii]

That's all right. It's a same error with other issues