Skip to content

Do not run cargotest on quickcheck/proptest tests #73514

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
Manishearth opened this issue Jun 19, 2020 · 7 comments · Fixed by #99602
Closed

Do not run cargotest on quickcheck/proptest tests #73514

Manishearth opened this issue Jun 19, 2020 · 7 comments · Fixed by #99602
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@Manishearth
Copy link
Member

As part of our CI we run "cargotest" on a variety of ecosystem crates, making sure they successfully build and run tests.

Unfortunately, some of use stochastic tests like quickcheck, which means that a crate passing our CI one day may not necessarily mean it passes CI the next. Perhaps we should audit the tests being run and filter out ones like these?

See #73511 (comment) and BurntSushi/xsv#227

cc @rust-lang/release

@Manishearth Manishearth added C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 19, 2020
@Mark-Simulacrum
Copy link
Member

Yeah I think we've discussed this to some extent previously within infra team as part of our "what to do about cargotest" discussions. cc @nellshamrell in particular

Not sure that we came to any conclusions though.

@jonas-schievink jonas-schievink added the A-testsuite Area: The testsuite used to check the correctness of rustc label Jun 19, 2020
@XAMPPRocky
Copy link
Member

Related: #70659

@RalfJung
Copy link
Member

This just bit me and took a lot of time as I triple-checked that after bors failing twice (!) in xsv, the same test actually works fine on my system.

@RalfJung
Copy link
Member

The xsv test suite seems to be full of quickcheck tests, so I think the only way to fix this is to remove xsv from cargotest (or to only run a short list of curated tests).

@ehuss
Copy link
Contributor

ehuss commented Jul 22, 2022

@RalfJung See #70659 where there is some discussion fo changing the cargotest set. I think the last comment implies dropping xsv.

@RalfJung
Copy link
Member

Hm, that discussion seems to have stalled though.

So to fix the immediate problem, I propose we do something like #99602.

@bors bors closed this as completed in 0549516 Jul 23, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants