Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

bench: add repro test #10167

Closed
wants to merge 2 commits into from
Closed

bench: add repro test #10167

wants to merge 2 commits into from

Conversation

dberenbaum
Copy link
Collaborator

@dberenbaum dberenbaum commented Dec 14, 2023

Closes iterative/dvc-bench#468

Edit: the need for this is to help track performance for issues like #9085

@dberenbaum dberenbaum requested a review from a team December 14, 2023 19:21
Copy link

codecov bot commented Dec 14, 2023

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (f3fa674) 90.62% compared to head (d9abbe7) 90.60%.
Report is 2 commits behind head on main.

Files Patch % Lines
dvc/testing/benchmarks/cli/commands/test_repro.py 0.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10167      +/-   ##
==========================================
- Coverage   90.62%   90.60%   -0.02%     
==========================================
  Files         500      501       +1     
  Lines       37852    37879      +27     
  Branches     5503     5508       +5     
==========================================
+ Hits        34305    34322      +17     
- Misses       2908     2917       +9     
- Partials      639      640       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@skshetry
Copy link
Member

It's going to be hard to write a fair benchmark for dvc repro.

I understand the motivation, but what you are testing here is dvc commit and that too is a no-op here because all of those were run already with run_copy() and is cached (mtimes, md5 hashes and the actual files).

Even if you made all of them --no-exec and tried to benchmark dvc repro, there is I/O happening in the subprocess cmd, which will be a significant part of the runtime.

@dberenbaum dberenbaum marked this pull request as draft December 19, 2023 13:51
@dberenbaum dberenbaum closed this Feb 29, 2024
@skshetry skshetry deleted the bench-repro branch March 5, 2024 08:49
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add pipelines use case
3 participants