Skip to content

Add retry mechanism for rustdoc GUI tests to reduce flakyness #106449

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 1 commit into from
Jan 8, 2023

Conversation

GuillaumeGomez
Copy link
Member

@GuillaumeGomez GuillaumeGomez commented Jan 4, 2023

Part of #93784.

I added 3 retries for failing GUI tests. An important note: if more than half of total tests fail, I don't retry because it's very likely not flakyness anymore at this point but a missing update after changes.

@rustbot
Copy link
Collaborator

rustbot commented Jan 4, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Jan 4, 2023
@GuillaumeGomez GuillaumeGomez force-pushed the rustdoc-gui-retry-mechanism branch 2 times, most recently from 418cb4b to e36b5bf Compare January 4, 2023 15:44
@GuillaumeGomez GuillaumeGomez force-pushed the rustdoc-gui-retry-mechanism branch from e36b5bf to f902200 Compare January 4, 2023 15:45
@GuillaumeGomez
Copy link
Member Author

I added some documentation on the NB_RETRY constant too.

@jyn514 jyn514 mentioned this pull request Jan 5, 2023
9 tasks
@Mark-Simulacrum
Copy link
Member

@bors r+ rollup=iffy

@bors
Copy link
Collaborator

bors commented Jan 7, 2023

📌 Commit f902200 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 7, 2023
@bors
Copy link
Collaborator

bors commented Jan 8, 2023

⌛ Testing commit f902200 with merge cc47b06...

@bors
Copy link
Collaborator

bors commented Jan 8, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing cc47b06 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 8, 2023
@bors bors merged commit cc47b06 into rust-lang:master Jan 8, 2023
@rustbot rustbot added this to the 1.68.0 milestone Jan 8, 2023
@GuillaumeGomez GuillaumeGomez deleted the rustdoc-gui-retry-mechanism branch January 8, 2023 20:51
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (cc47b06): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.7% [0.5%, 1.1%] 8
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-1.5% [-1.7%, -1.3%] 2
All ❌✅ (primary) - - 0

Cycles

This benchmark run did not return any relevant results for this metric.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants