Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Use single license scanner for all catalogers #3348

Merged
merged 2 commits into from
Oct 21, 2024
Merged

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Oct 18, 2024

Description

Extends work from #3290 by still not leveraging the static license change object (to prevent the GC from holding onto license classifier objects in static space) but also not creating the same scanner often (which results in poor CPU performance). Instead this approach uses a single scanner instance that is scoped to the context object created within the CreateSBOM() top-level API function. Since the licenses.Search() path is invoked several times, it's important to not create license scanner objects within that function, but instead, pass a new scanner object. This allows for the caller to swap strategies and provide configuration to get different behaviors in license scanning. That being said -- this configuration has NOT been exposed on the API, as the new licenses.Scanner is in an internal package, however, in the future we can elevate this to the pkg/license package and expose a new builder function on the top-level API CreateSBOMConfig, but this has been deferred until later.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Performance (make Syft run faster or use less memory, without changing visible behavior much)

Checklist:

  • I have added unit tests that cover changed behavior (updated existing tests)
  • I have tested my code in common scenarios and confirmed there are no regressions
  • I have added comments to my code, particularly in hard-to-understand sections

@wagoodman wagoodman added bug Something isn't working performance labels Oct 18, 2024
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman wagoodman force-pushed the single-license-scanner branch from 7431129 to 8f58814 Compare October 18, 2024 20:58
@popey
Copy link
Contributor

popey commented Oct 18, 2024

The fix is certainly comparable to the performance of the previous release.

for f in grype-0810 grype-0820 grype-0821 grype-fix; do time ./$f jenkins/jenkins:latest -o json=$f.json; done

grype v0.81.0

 ✔ Vulnerability DB                [no update available]
 ✔ Parsed image sha256:f78b5b90f20c751402aee62cca2342ed20abb9af0d06e45cdc666918ad9f9a00
 ✔ Cataloged contents 62fe8db1f064ce5bfd13cf87d2dcc9083d129b083a43009dde3551393ca9f4fa
   ├── ✔ Packages                        [497 packages]
   ├── ✔ File digests                    [8,512 files]
   ├── ✔ File metadata                   [8,512 locations]
   └── ✔ Executables                     [931 executables]
 ✔ Scanned for vulnerabilities     [156 vulnerability matches]
   ├── by severity: 4 critical, 15 high, 22 medium, 8 low, 92 negligible (15 unknown)
   └── by status:   13 fixed, 143 not-fixed, 0 ignored

real	0m55.700s
user	0m15.853s
sys	0m2.046s

grype v0.82.0

 ✔ Parsed image sha256:f78b5b90f20c751402aee62cca2342ed20abb9af0d06e45cdc666918ad9f9a00
 ✔ Cataloged contents 62fe8db1f064ce5bfd13cf87d2dcc9083d129b083a43009dde3551393ca9f4fa
   ├── ✔ Packages                        [497 packages]
   ├── ✔ File digests                    [8,878 files]
   ├── ✔ File metadata                   [8,878 locations]
   └── ✔ Executables                     [931 executables]
 ✔ Scanned for vulnerabilities     [156 vulnerability matches]
   ├── by severity: 4 critical, 15 high, 22 medium, 8 low, 92 negligible (15 unknown)
   └── by status:   13 fixed, 143 not-fixed, 0 ignored
[0058]  WARN unable to discover java packages from opener: unable to process nested java archive (WEB-INF/lib/javax.inject-1.jar): /usr/share/je
[0067]  WARN unable to parse cpe attributes for elf binary package error=unable to parse Attributes string: failed to parse Attributes="": wfn:

real	1m8.772s
user	0m52.209s
sys	0m2.892s

grype v0.82.1

 ✔ Parsed image sha256:f78b5b90f20c751402aee62cca2342ed20abb9af0d06e45cdc666918ad9f9a00
 ✔ Cataloged contents 62fe8db1f064ce5bfd13cf87d2dcc9083d129b083a43009dde3551393ca9f4fa
   ├── ✔ Packages                        [496 packages]
   ├── ✔ File digests                    [8,878 files]
   ├── ✔ File metadata                   [8,878 locations]
   └── ✔ Executables                     [931 executables]
 ✔ Scanned for vulnerabilities     [156 vulnerability matches]
   ├── by severity: 4 critical, 15 high, 22 medium, 8 low, 92 negligible (15 unknown)
   └── by status:   13 fixed, 143 not-fixed, 0 ignored
[0071]  WARN unable to discover java packages from opener: unable to process nested java archive (WEB-INF/lib/javax.inject-1.jar): /usr/share/je

real	1m28.341s
user	0m54.368s
sys	0m3.399s

grype fix - pr 3348

 ✔ Parsed image sha256:f78b5b90f20c751402aee62cca2342ed20abb9af0d06e45cdc666918ad9f9a00
 ✔ Cataloged contents 62fe8db1f064ce5bfd13cf87d2dcc9083d129b083a43009dde3551393ca9f4fa
   ├── ✔ Packages                        [496 packages]
   ├── ✔ File digests                    [8,878 files]
   ├── ✔ File metadata                   [8,878 locations]
   └── ✔ Executables                     [931 executables]
 ✔ Scanned for vulnerabilities     [156 vulnerability matches]
   ├── by severity: 4 critical, 15 high, 22 medium, 8 low, 92 negligible (15 unknown)
   └── by status:   13 fixed, 143 not-fixed, 0 ignored
[0053]  WARN unable to discover java packages from opener: unable to process nested java archive (WEB-INF/lib/javax.inject-1.jar): /usr/share/je

real	0m55.420s
user	0m17.839s
sys	0m2.131s

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman wagoodman enabled auto-merge (squash) October 21, 2024 16:10
@wagoodman wagoodman merged commit e4e985b into main Oct 21, 2024
12 checks passed
@wagoodman wagoodman deleted the single-license-scanner branch October 21, 2024 16:17
@wagoodman wagoodman changed the title Create single license scanner for all catalogers Use single license scanner for all catalogers Oct 21, 2024
HeyeOpenSource added a commit to HeyeOpenSource/syft that referenced this pull request Oct 22, 2024
…gers (anchore#3348)](anchore@e4e985b) by Alex Goodman <wagoodman@users.noreply.github.com>

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants