Skip to content

tests/run-make: failing tests get incorrectly ignored on rerun #129971

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
jieyouxu opened this issue Sep 5, 2024 · 0 comments · Fixed by #131681
Closed

tests/run-make: failing tests get incorrectly ignored on rerun #129971

jieyouxu opened this issue Sep 5, 2024 · 0 comments · Fixed by #131681
Labels
A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@jieyouxu
Copy link
Member

jieyouxu commented Sep 5, 2024 •
edited
Loading

First run (fails on native windows because obviously make not found):

running 362 tests
iii
[run-make] tests\run-make\cat-and-grep-sanity-check ... F
.iii.......ii..iii...........i...ii....i
[run-make] tests\run-make\emit-to-stdout ... F
....i........i..........ii...ii.ii.i.......  88/362
i.......ii..i...i....ii...i.....i..i....ii......ii.......i.i......i......i.....i......i. 176/362
..i....iii.iii...........i.i.iii..i....i.iii.i.......i..iiii.....i..................iii. 264/362
......ii.
[run-make] tests\run-make\split-debuginfo ... F
...i........i...........iii.ii.....ii.iiiiiiiiiiii..i..i...................... 352/362
..........

Second run (no changes):

running 362 tests
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii  88/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 176/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 264/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 352/362
iiiiiiiiii

test result: ok. 0 passed; 0 failed; 362 ignored; 0 measured; 0 filtered out; finished in 2.44ms

The same tests do fail with --force-rerun. Something about the ignore logic is incorrect.

We should not be ignoring failing tests.
@jieyouxu jieyouxu added A-testsuite Area: The testsuite used to check the correctness of rustc T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-bug Category: This is a bug. A-run-make Area: port run-make Makefiles to rmake.rs labels Sep 5, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Sep 5, 2024
@jieyouxu jieyouxu removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Sep 5, 2024
@jieyouxu jieyouxu mentioned this issue Oct 14, 2024
Merged
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Oct 14, 2024
@matthiaskrgr
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
55016c8 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Oct 14, 2024
@matthiaskrgr
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
ba74130 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
compiler-errors added a commit to compiler-errors/rust that referenced this issue Oct 15, 2024
@compiler-errors
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
71f2254 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
@bors bors closed this as completed in 258c177 Oct 15, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Oct 15, 2024
@rust-timer
Unrolled build for rust-lang#131681
5a95ee3 
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jieyouxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix up-to-date checking for run-make tests Zalathar/rust
2 participants
@jieyouxu @rustbot