fix: better clean up of file handles #2823
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The elf-binary-package-cataloger does its own file IO to account for the possibility of a logical ELF package being broken across multiple physical files. However, this casued it to skip the normal invocation pattern in the generic cataloger code that prevented file leaks. Ensure this cataloger always closes its file handles. Signed-off-by: Will Murphy <will.murphy@anchore.com>
Otherwise, a panicking cataloger could leak file handles. Signed-off-by: Will Murphy <will.murphy@anchore.com>
spiffcs
approved these changes
Apr 29, 2024
wagoodman
reviewed
Apr 29, 2024
|
|
||
| log.WithFields("path", location.RealPath).Trace("parsing file contents") | ||
| invokeParser := func(location file.Location, parser Parser) ([]pkg.Package, []artifact.Relationship, error) { |
There was a problem hiding this comment.
could this be a static function over a closure? it would be simpler to reason about.
There was a problem hiding this comment.
Agreed. It will have a lot more parameters (env and logger), but it will be easier to reason about, so that's probably the right trade off. I'll try to push up a rev.
Signed-off-by: Will Murphy <will.murphy@anchore.com>
Signed-off-by: Will Murphy <will.murphy@anchore.com>
|
@wagoodman / @spiffcs - refactored and added a unit test for not leaking file on generic cataloger panic. |
willmurphyscode
commented
Apr 30, 2024
Signed-off-by: Will Murphy <will.murphy@anchore.com>
wagoodman
reviewed
May 1, 2024
| @@ -122,17 +123,9 @@ func (c *Cataloger) Catalog(ctx context.Context, resolver file.Resolver) ([]pkg. | |||
|
|
|||
| log.WithFields("path", location.RealPath).Trace("parsing file contents") | |||
There was a problem hiding this comment.
optional opportunity for unrelated cleanup -- this should be using logger not log
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the ELF binary package cataloger would leak file handles, and the generic cataloger loop would leak file handles if parsing the file panicked. Instead, ensure both situations
defer closenew file readers as soon as they are made.The incorrect behavior had been happening for some time, but was probably exposed by #2814 which caused the ELF binary package cataloger to rely more on file handles and less on in-memory buffers.
Fixes #2819 - in my testing, syft 1.3.0 needs about 1500 concurrent file handles to parse
pytorch/pytorch:latest; after this change, the number is closer to 800.