profiles: slsa: Remove quadratic complexity in SRC_URI iteration #1399
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SLSA provenance generation iterates over $A (which is a subset of $SRC_URI) and for each of those tries to find a match in $SRC_URI. That's quadratic complexity, and the performance impact is bad because we shell out to a helper utility (basename) for every entry. This is leading to long stalls when generating SLSA for packages with long distfile lists, like go and rust packages. Iterate over SRC_URI once and create a dictionary to speed up subsequent lookups. dev-db/etcdctl is a good candidate for testing. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
|
Build action triggered: https://github.com/flatcar/scripts/actions/runs/7118559873 |
|
@krnowak lets cherry-pick this into your PR then |
|
Cherry picked to #1157. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
speed up slsa provenance generation for some packages
SLSA provenance generation iterates over $A (which is a subset of $SRC_URI) and for each of those tries to find a match in $SRC_URI. That's quadratic complexity, and the performance impact is bad because we shell out to a helper utility (basename) for every entry. This is leading to long stalls when generating SLSA for packages with long distfile lists, like go and rust packages. Iterate over SRC_URI once and create a dictionary to speed up subsequent lookups. dev-db/etcdctl is a good candidate for testing.
How to use
Testing done
Before: >10min on my laptop
After: 1min on my laptop (includes build time)
[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]
changelog/directory (user-facing change, bug fix, security fix, update)/bootand/usrsize, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.