Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add some functionality to misc/perf_compare.py #18471

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

wyattscarpenter
Copy link
Contributor

@wyattscarpenter wyattscarpenter commented Jan 15, 2025

While working on determining the performance impact of #17875, I discovered misc/perf_compare.py. I extended its functionality slightly, introducing an -r flag to check a foreign repo instead of mypy itself. I also added a --dont-setup flag, to save time on recompiling mypy. I added several helpful printouts: the standard deviations of the samples, an epilog reminding you to probably specific master, a line at the bottom listing the total time taken by the whole benchmarking program.

As a caveat, I should flag: when running the benchmark on the foreign repo, it typically spits out a lot of errors, presumably because the libraries aren't installed. That actually also happens to me when running the mypy self-check, though, for whatever reason, so I didn't worry about it.

Another caveat I'll flag on the off-chance it's important: on my machine, this script seems constitutionally incapable of deleting the tmp dirs it makes. It fails with a permissions error. But it was like that when I got it.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant