Skip to content

[DO NOT LAND] Regress the token-stream-stress benchmark. #67248

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed

Conversation

nnethercote
Copy link
Contributor

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See #66405 and #67079
for more details.

This commit deliberately causes a massive regression to the
token-stream-stress benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.

cc @Mark-Simulacrum
r? @ghost

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See rust-lang#66405 and rust-lang#67079
for more details.

This commit deliberately causes a massive regression to the
`token-stream-stress` benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.
@nnethercote
Copy link
Contributor Author

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion

@bors
Copy link
Collaborator

bors commented Dec 12, 2019

⌛ Trying commit 2d5843d with merge 5e1e02e...

bors added a commit that referenced this pull request Dec 12, 2019
[DO NOT LAND] Regress the `token-stream-stress` benchmark.

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See #66405 and #67079
for more details.

This commit deliberately causes a massive regression to the
`token-stream-stress` benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.

cc @Mark-Simulacrum
r? @ghost
@nnethercote
Copy link
Contributor Author

Here are the top 10 check-clean entries from a local run:

token-stream-stress-check
        avg: 1843.4%    min: 1843.4%    max: 1843.4%
helloworld-check
        avg: -0.1%      min: -0.1%      max: -0.1%
issue-46449-check
        avg: -0.1%      min: -0.1%      max: -0.1%
unify-linearly-check
        avg: -0.0%      min: -0.0%      max: -0.0%
html5ever-check
        avg: 0.0%       min: 0.0%       max: 0.0%
deeply-nested-check
        avg: -0.0%      min: -0.0%      max: -0.0%
await-call-tree-check
        avg: -0.0%      min: -0.0%      max: -0.0%
coercions-check
        avg: -0.0%?     min: -0.0%?     max: -0.0%?
serde-check
        avg: 0.0%       min: 0.0%       max: 0.0%
syn-check
        avg: -0.0%      min: -0.0%      max: -0.0%

It shows a huge regression for token-stream-stress, and negligible other changes. Let's see if the CI run matches that.

@bors
Copy link
Collaborator

bors commented Dec 12, 2019

☀️ Try build successful - checks-azure
Build commit: 5e1e02e (5e1e02e73da308be8b0908637628027265f123a7)

@rust-timer
Copy link
Collaborator

Queued 5e1e02e with parent de0abf7, future comparison URL.

@rust-timer
Copy link
Collaborator

Finished benchmarking try commit 5e1e02e, comparison URL.

@nnethercote
Copy link
Contributor Author

CI results match my local results. Top 10 check-clean results:

token-stream-stress-check
        avg: 1695.9%    min: 1695.9%    max: 1695.9%
helloworld-check
        avg: 0.4%       min: 0.4%       max: 0.4%
issue-46449-check
        avg: 0.2%       min: 0.2%       max: 0.2%
await-call-tree-check
        avg: 0.2%       min: 0.2%       max: 0.2%
unify-linearly-check
        avg: 0.2%       min: 0.2%       max: 0.2%
deeply-nested-check
        avg: 0.2%       min: 0.2%       max: 0.2%
ripgrep-check
        avg: 0.1%       min: 0.1%       max: 0.1%
futures-check
        avg: 0.1%       min: 0.1%       max: 0.1%
regression-31157-check
        avg: 0.1%       min: 0.1%       max: 0.1%
regex-check
        avg: 0.0%       min: 0.0%       max: 0.0% 

A bit more variation among the barely-changing ones, but it disproves the theory that the wrong revisions are being tested.

@nnethercote nnethercote deleted the regress-token-stream-stress branch December 12, 2019 09:59
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants