Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Optimize recomputing definition #1232

Merged
merged 9 commits into from
Nov 21, 2024

Conversation

qtomlinson
Copy link
Collaborator

@qtomlinson qtomlinson commented Nov 13, 2024

  • Fix utils/getLatestVersion and add unit tests
  • Allow injection of logger in AbstractAzBlobStore and AbstractFileStore
  • Introduce getAllLatest to harvest stores, AzHarvestBlobStore and FileHarvestStore, to retrieve the latest versions of harvest tool results. It will fall back to getAll results in case any error in calculating the latest files. Use getAllLatest in definition computation.
  • Add an integration test to ensure that the definition computed using the latest harvest results are the same as the definition computed using all the harvest results.
  • Add unit tests

Timing of computing definition for npm/npmjs/-/debug/3.1.0:
before optimization: retrieve blob: 3.2 sec (without 2.9.0+b1), calculating summary: 0.004 sec, curation: 1.2 sec
after optimization: retrieve blob: 1.3 sec, calculating summary: 0.02 sec, curation: 1.2 sec

Task: #1237

@qtomlinson qtomlinson force-pushed the qt/optimize_recompute branch from bb0dfcd to bdfc3ae Compare November 13, 2024 23:38
@qtomlinson qtomlinson force-pushed the qt/optimize_recompute branch from 999b3aa to 8ce337c Compare November 16, 2024 01:51
@qtomlinson qtomlinson marked this pull request as ready for review November 18, 2024 20:31
@qtomlinson qtomlinson force-pushed the qt/optimize_recompute branch from 155b6ae to a83fe72 Compare November 18, 2024 22:27
Copy link
Collaborator

@elrayle elrayle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent PR. Always happy to see performance improvements.

@qtomlinson qtomlinson merged commit f79a63c into clearlydefined:master Nov 21, 2024
4 checks passed
@qtomlinson qtomlinson deleted the qt/optimize_recompute branch November 21, 2024 05:01
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants