'cargo test --release' doesn't use release profile

[bcmills]

Problem
cargo test --release does not appear to respect the profile.release configuration in Cargo.toml.

I configured the profile.release section of my Cargo.toml file to set overflow-checks = true. (See full listings below.)

I wanted to confirm that I got the configuration right, so I wrote a test that intentionally overflows. I first ran cargo test to ensure that it panics, then ran cargo test --release to confirm that it continues to panic using the release configuration. I was surprised to see that it did not.

I moved the function up into a fn main and confirmed that it panics there (as expected) when using cargo run --release.

Steps
Cargo.toml:

[package]
name = "bug"
version = "0.1.0"
authors = ["Bryan C. Mills <bryan@bcmills.net>"]
edition = "2018"

[dependencies]

[profile.release]
overflow-checks = true

src/main.rs:

fn overflow_outside_test() {
    let a: u32 = 0xFFFFFFFF;
    let b = a + 1;
    assert_eq!(b, 0);
}

fn main() {
    overflow_outside_test()
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn outside() {
        overflow_outside_test()
    }

    #[test]
    fn inside() {
        let a: u32 = 0xFFFFFFFF;
        let b = a + 1;
        assert_eq!(b, 0);
    }
}

~/projects/bug$ cargo run -v
   Compiling bug v0.1.0 (/home/bryan/projects/bug)
     Running `rustc --edition=2018 --crate-name bug src/main.rs --color always --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=cf719392cc9fdf3e -C extra-filename=-cf719392cc9fdf3e --out-dir /home/bryan/projects/bug/target/debug/deps -C incremental=/home/bryan/projects/bug/target/debug/incremental -L dependency=/home/bryan/projects/bug/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.46s
     Running `target/debug/bug`
thread 'main' panicked at 'attempt to add with overflow', src/main.rs:3:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

~/projects/bug$ cargo run -v --release
   Compiling bug v0.1.0 (/home/bryan/projects/bug)
     Running `rustc --edition=2018 --crate-name bug src/main.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C overflow-checks=on -C metadata=e8a696542bb52eba -C extra-filename=-e8a696542bb52eba --out-dir /home/bryan/projects/bug/target/release/deps -L dependency=/home/bryan/projects/bug/target/release/deps`
    Finished release [optimized] target(s) in 0.31s
     Running `target/release/bug`
thread 'main' panicked at 'attempt to add with overflow', src/main.rs:3:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

~/projects/bug$ cargo test -v
   Compiling bug v0.1.0 (/home/bryan/projects/bug)
     Running `rustc --edition=2018 --crate-name bug src/main.rs --color always --emit=dep-info,link -C debuginfo=2 --test -C metadata=bb3179c6754bf8e8 -C extra-filename=-bb3179c6754bf8e8 --out-dir /home/bryan/projects/bug/target/debug/deps -C incremental=/home/bryan/projects/bug/target/debug/incremental -L dependency=/home/bryan/projects/bug/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.46s
     Running `/home/bryan/projects/bug/target/debug/deps/bug-bb3179c6754bf8e8`

running 2 tests
test tests::inside ... FAILED
test tests::outside ... FAILED

failures:

---- tests::inside stdout ----
thread 'tests::inside' panicked at 'attempt to add with overflow', src/main.rs:23:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- tests::outside stdout ----
thread 'tests::outside' panicked at 'attempt to add with overflow', src/main.rs:3:13


failures:
    tests::inside
    tests::outside

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--bin bug'

~/projects/bug$ cargo test -v --release
   Compiling bug v0.1.0 (/home/bryan/projects/bug)
     Running `rustc --edition=2018 --crate-name bug src/main.rs --color always --emit=dep-info,link -C opt-level=3 --test -C metadata=adfa154307f54ab3 -C extra-filename=-adfa154307f54ab3 --out-dir /home/bryan/projects/bug/target/release/deps -L dependency=/home/bryan/projects/bug/target/release/deps`
    Finished release [optimized] target(s) in 0.43s
     Running `/home/bryan/projects/bug/target/release/deps/bug-adfa154307f54ab3`

running 2 tests
test tests::inside ... ok
test tests::outside ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


~/projects/bug$

Note that cargo run and cargo run --release both panic, and cargo test reports that both tests fail.

cargo test --release unexpectedly reports that both tests passed.

Notes

Output of cargo version:

cargo 1.31.0 (339d9f9c8 2018-11-16)

[bcmills]

See previously #831.

[ehuss]

Tests use the test/bench profiles instead of dev/release (so test --release uses "bench"). This only affects the final artifact, dependencies still use the dev/release profiles.

[bcmills]

In that case, I think there may be two distinct issues at play:

1. If the --release flag to cargo test means “use the bench profile”, then cargo help test should state that explicitly. cargo help test currently says:

           --release                   Build artifacts in release mode, with optimizations
   

2. If the bench profile is intended to accurately reflect performance under the release profile, shouldn't the bench profile inherit all of the codegen options from release? Specifying the same options twice seems redundant and (as seen here) error-prone.

[bcmills]

(That second point seems to tie closely to #2085.)

[alexcrichton]

Yeah as mentioned by @ehuss this is "working as expected", albeit wonkily. Perhaps this should be closed in favor of #2085 which I think is probably the long-term solution to this issue?

[bcmills]

If #2085 is likely to happen, then I agree that this can be closed in favor of that.

If it will be a while, I'd still appreciate a clearer description in the output of cargo help test.

[dwijnand]

#2085 hasn't been particularly active, so I think we'd be happy to accept PRs that resolve this by bringing clarity to the help messaging.

[alexcrichton]

I would personally like to take the route of #2085, but it's obviously a large change and will take some time, and as a result needs to be prioritized against everything else

[ehuss]

Closing as fixed, as part of #9943, profile selection was simplified and test --release now uses the release profile instead of a mix of "bench" and "release".