Skip to content

More systematic compile/run markup for "pass" UI tests #61712

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
petrochenkov opened this issue Jun 10, 2019 · 3 comments
Closed

More systematic compile/run markup for "pass" UI tests #61712

petrochenkov opened this issue Jun 10, 2019 · 3 comments
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@petrochenkov
Copy link
Contributor

TLDR:

In test files:

// check-pass
// build-pass
// run-pass

In `x.py` command line (forcing):

--force =check
--force >=check
--force <=check
--force =build
--force >=build
--force <=build
--force =run
--force >=run
--force <=run

In `x.py` command line (running a subset):

--select =check
--select >=check
--select <=check
--select =build
--select >=build
--select <=build
--select =run
--select >=run
--select <=run

TODO:
More detailed description, motivation, etc.

cc @Centril
@petrochenkov petrochenkov added A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. labels Jun 10, 2019
@Centril
Copy link
Contributor

Centril commented Jun 10, 2019

I've added:

    /// Only run tests that match any of these modes.
    pub filter_mode: Option<Vec<Mode>>,

to Config.

The thinking is that you pass --filter-mode run-pass,compile-fail,... and have only those tests run.

A --filter-mode scheme seems easiest to implement given the current compiletest infrastructure and doesn't require us to rename a bunch of annotations.

@Centril Centril added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Jun 10, 2019
@petrochenkov
Copy link
Contributor Author

After some thought, perhaps filtering is not even necessary with --force check.
I'll make some experiments and report (sometime this week).

This was referenced Jun 10, 2019
Closed
Merged
@petrochenkov petrochenkov mentioned this issue Jun 12, 2019
Merged
Centril added a commit to Centril/rust that referenced this issue Jun 23, 2019
@Centril
Rollup merge of rust-lang#61778 - petrochenkov:pass, r=Mark-Simulacrum
6e06fa0 
compiletest: Introduce `// {check,build,run}-pass` pass modes

Pass UI tests now have three modes
```
// check-pass
// build-pass
// run-pass
```
mirroring equivalent well-known `cargo` commands.

`// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases).
`// build-pass` will compile and link the test without running it.
`// run-pass` will compile, link and run the test.
Tests without a "pass" annotation are still considered "fail" tests.

Most UI tests would probably want to switch to `check-pass`.
Tests validating codegen would probably want to run the generated code as well and use `run-pass`.
`build-pass` should probably be rare (linking tests?).

rust-lang#61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way.
Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test.
Perhaps some secondary CI can verify this invariant, but that's not super urgent.

`// compile-pass` still works and is equivalent to `build-pass`.
Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command.
It can be removed some time in the future in a separate PR.

cc rust-lang#61712
bors added a commit that referenced this issue Jun 23, 2019
@bors
Auto merge of #61778 - petrochenkov:pass, r=Mark-Simulacrum
2cd5ed4 
compiletest: Introduce `// {check,build,run}-pass` pass modes

Pass UI tests now have three modes
```
// check-pass
// build-pass
// run-pass
```
mirroring equivalent well-known `cargo` commands.

`// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases).
`// build-pass` will compile and link the test without running it.
`// run-pass` will compile, link and run the test.
Tests without a "pass" annotation are still considered "fail" tests.

Most UI tests would probably want to switch to `check-pass`.
Tests validating codegen would probably want to run the generated code as well and use `run-pass`.
`build-pass` should probably be rare (linking tests?).

#61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way.
Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test.
Perhaps some secondary CI can verify this invariant, but that's not super urgent.

`// compile-pass` still works and is equivalent to `build-pass`.
Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command.
It can be removed some time in the future in a separate PR.

cc #61712
Centril added a commit to Centril/rust that referenced this issue Jun 28, 2019
@Centril
Rollup merge of rust-lang#61755 - Centril:compiletest-force-check, r=…
49c2a2a 
…petrochenkov

Add `--pass $mode` to compiletest through `./x.py`

Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`.

When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file.

The modes are explained in rust-lang#61778:
- `check` has the same effect as `cargo check`
- `build` or `compile` have the same effect as `cargo build`
- `run` has the same effect as `cargo run`

On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`.

cc rust-lang#61712

r? @petrochenkov
Centril added a commit to Centril/rust that referenced this issue Jun 28, 2019
@Centril
Rollup merge of rust-lang#61755 - Centril:compiletest-force-check, r=…
fea6402 
…petrochenkov

Add `--pass $mode` to compiletest through `./x.py`

Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`.

When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file.

The modes are explained in rust-lang#61778:
- `check` has the same effect as `cargo check`
- `build` or `compile` have the same effect as `cargo build`
- `run` has the same effect as `cargo run`

On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`.

cc rust-lang#61712

r? @petrochenkov
Centril added a commit to Centril/rust that referenced this issue Jun 28, 2019
@Centril
Rollup merge of rust-lang#61755 - Centril:compiletest-force-check, r=…
e775a5d 
…petrochenkov

Add `--pass $mode` to compiletest through `./x.py`

Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`.

When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file.

The modes are explained in rust-lang#61778:
- `check` has the same effect as `cargo check`
- `build` or `compile` have the same effect as `cargo build`
- `run` has the same effect as `cargo run`

On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`.

cc rust-lang#61712

r? @petrochenkov
Centril added a commit to Centril/rust that referenced this issue Jun 29, 2019
@Centril
Rollup merge of rust-lang#61755 - Centril:compiletest-force-check, r=…
0721364 
…petrochenkov

Add `--pass $mode` to compiletest through `./x.py`

Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`.

When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file.

The modes are explained in rust-lang#61778:
- `check` has the same effect as `cargo check`
- `build` or `compile` have the same effect as `cargo build`
- `run` has the same effect as `cargo run`

On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`.

cc rust-lang#61712

r? @petrochenkov
@petrochenkov
Copy link
Contributor Author

Addressed by #61778 and #61755.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Centril @petrochenkov