Regression in code coverage (-Zprofile) at some point after nightly-2021-11-11

[fadeevab]

TL;DR:

Older nightly toolchain gives an expected code coverage: rustup default nightly-2021-11-11

UPDATE: Regression is reproduced below #100125 (comment)

Prerequisites

I have a GitHub workflow which generates a code coverage report with nightly toolchain +grcov, then sending it to https://coveralls.io.

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
export RUSTDOCFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"

rustup default nightly
cargo test --all-features
curl -L https://github.com/mozilla/grcov/releases/latest/download/grcov-x86_64-unknown-linux-gnu.tar.bz2 | tar jxf -

mkdir coverage
./grcov ./target/debug/ -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" \
                --excl-line '#\[|=> panic!|unreachable!|Io\(std::io::Error\)' \
                --excl-br-line '#\[|=> panic!|unreachable!|assert_..!' -o ./coverage/lcov.info

I expected to see 100% code coverage on a simple file, e.g. see older coverage result: https://coveralls.io/builds/43003664/source?filename=src%2Ferror.rs - error.rs has 100% code coverage.

Instead, this happened:

1. https://coveralls.io/builds/51398291/source?filename=src%2Ferror.rs - 9% coverage.
2. https://coveralls.io/builds/51398291/source?filename=src%2Flib.rs - coverage goes over comment lines

Version it worked on

At least nightly-2021-11-11 toolchain works:

rustup default nightly-2021-11-11

However, I have not narrowed down the most recent version this worked on.

Version with regression

rustc --version --verbose:

rustc 1.64.0-nightly (0f4bcadb4 2022-07-30)
binary: rustc
commit-hash: 0f4bcadb46006bc484dad85616b484f93879ca4e
commit-date: 2022-07-30
host: x86_64-unknown-linux-gnu
release: 1.64.0-nightly
LLVM version: 14.0.6

[apiraino]

Unless I am mistaken, grcov is a third-party tool, not officially under the Rust project umbrella.

In order to find an actionable here, a bisection and ideally a reproducible are probably the way to go.

@rustbot label E-needs-bisection E-needs-mcve

[fadeevab]

@apiraino I've spent a few hours yesterday, and that's my result:

1. latest grcov + latest rust nightly = bad
2. old grcov 0.8.2 (~July 2021) + latest rustc nightly = bad
3. old grcov + old rustc nightly-2021-11-11 = good
4. latest grcov + old nightly-2021-11-11 = good

Notes:

• "old" is for simplicity, it means the versions of the end of the 2021 year
• "latest" means the version at this moment, however, the it's not clear when the actual regression was introduced.
• "bad" means a wrong coverage
• "good" means an expected coverage

Case 1 and Case 2 above shows, that the newer nightly either has a regression, or the nightly introduces some profile format incompatibility with 3rd party tools.

[fanninpm]

You can help narrow down the regression even more by using cargo-bisect-rustc. The cargo-bisect-rustc tool is usually used to find compilation errors, but you can use a shell script to tell cargo-bisect-rustc what your regression is.

[fadeevab]

@fanninpm @apiraino

---

Regression in 936eba3

---

==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2022-05-16 to nightly-2022-05-20
regressed nightly: nightly-2022-05-19
searched commit range: https://github.com/rust-lang/rust/compare/4c5f6e6277b89e47d73a192078697f7a5f3dc0ac...cd282d7f75da9080fda0f1740a729516e7fbec68
regressed commit: https://github.com/rust-lang/rust/commit/936eba3b348e65b658b60218cc9237f02abdbeb4

<details>
<summary>bisected with <a href='https://github.com/rust-lang/cargo-bisect-rustc'>cargo-bisect-rustc</a> v0.6.4</summary>


Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc ./script.sh --start 2022-05-16 --end 2022-05-20 --preserve 

</details>

I actually bisected between 2021-11-11 and 2022-08-09.

[fanninpm]

regressed commit: 936eba3

Thank you! That points to #96867.

[fadeevab]

Examples of regression.

It checks random lines in the documentation comment:

---

It shows 3-5% of coverage on a little error.rs file (100% without regression).

[fadeevab]

@fanninpm By the way, you could probably change labels on this issue :)

[fanninpm]

By the way, you could probably change labels on this issue

Can I?

@rustbot label -E-needs-bisection

Nevertheless, it would be helpful to have a Minimal, Complete, and Verifiable Example.

[fadeevab]

@fanninpm

Steps:

1. Download https://github.com/fadeevab/cocoon/
2. Put script.sh into the project directory.

   #!/bin/sh
   
   export CARGO_INCREMENTAL=0
   export RUSTDOCFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
   export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
   
   cargo test --all-features
   mkdir -p ./${CARGO_TARGET_DIR}/coverage
   ./grcov ./${CARGO_TARGET_DIR}/${CARGO_BUILD_TARGET}/debug/ -s . -t html --llvm --branch --ignore-not-existing --ignore "/*"  --excl-line '#\[|=> panic!|unreachable!|Io\(std::io::Error\)'  --excl-br-line '#\[|=> panic!|unreachable!|assert_..!' -o ./${CARGO_TARGET_DIR}/coverage/lcov.html
   grep -e ">[89]" ./${CARGO_TARGET_DIR}/coverage/lcov.html/index.html

3. Run bisection

   cargo-bisect-rustc --script ./script.sh --start 2022-05-16 --end 2022-05-20 --preserve

Explanation: if grep finds 80-90% in a coverage report (it looks for a 99% or say 84%, which is also good), so it's just looking for '>9' or '>8', then the script returns 0, else it returns error 1.
I double checked the coverage reports: it really finds a regression between the commits.

[fadeevab]

@fanninpm

A Minimal, Complete, and Verifiable Example: https://github.com/fadeevab/rust-lang-rust-issues-100125

[fadeevab]

@fanninpm Could "regression-untriaged" be removed?

[fanninpm]

@fanninpm Could "regression-untriaged" be removed?

Not until it has been triaged by the appropriate team. @apiraino may be more privy to the triaging processes that go on around here.

[apiraino]

I've tried spending a bit of time on the differences between grcov reports, I'm unsure how to provide a clear actionable to act on for this behaviour. I checked the grcov issue tracker and found mozilla/grcov#725 which seems close to what is reported here. That issue lead to #91661 which has more insights and suggestions to help forming a diagnose.

For this reason I'd leave this issue marked as in need of some more accurate triaging than I can provide.

[federicomenaquintero]

I can reproduce this with librsvg. Both with grcov 0.8.11:

• Rust 1.62.0 - coverage seems OK, at 91.7%
• Rust 1.63.0 - coverage goes down to 76.85%, shows some toplevel comment lines as uncovered, use somecrate::Something; lines as uncovered, etc.

RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Clink-dead-code -Coverflow-checks=off"

(I'm using -Zprofile because -Cinstrument-coverage produces incorrect results when linking to C code)

[jieyouxu]

Triage: closing because -Zprofile is removed following #131829.

[fadeevab]

No body, no crime 😁

For the reference, here is what can be used instead of grcov: llvm-cov and the PR fadeevab/cocoon#31 how to make a transition from grcov to llvm-cov.

(Alternatively, see PR namib-project/dcaf-rs#27 how to continue using grcov with -Cinstrument-coverage instead of -Zprofile).