Add annotations for evidence on package locations #1723
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Benchmark Test ResultsBenchmark results from the latest changes vs base branch |
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
wagoodman
changed the title
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
wagoodman
force-pushed
the
add-location-annotations
branch
from
7b3870b to
bf30d08
Compare
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
|
JSON schema diff for reviewers: # ❯ diff schema/json/schema-7.1.1.json schema/json/schema-7.1.2.json
817,818c817,823
< "virtualPath": {
< "type": "string"
---
> "annotations": {
> "patternProperties": {
> ".*": {
> "type": "string"
> }
> },
> "type": "object"
946c951
< "$ref": "#/$defs/Coordinates"
---
> "$ref": "#/$defs/Location" |
kzantow
reviewed
Apr 12, 2023
syft/pkg/cataloger/binary/package.go
Outdated
| "github.com/anchore/syft/syft/source" | ||
| ) | ||
|
|
||
| func newPackage(classifier classifier, location source.Location, matchMetadata map[string]string) []pkg.Package { |
There was a problem hiding this comment.
I think this should continue to be named singlePackage, as it's returning a slice, not just a package. If it was returning a pkg.Package, definitely named newPackage but I wanted it to be very clear it wasn't returning more than one.
There was a problem hiding this comment.
You're right this is a little confusing. I was trying to bring the package constructor naming and file organization in line with the other catalogers. I think having something named singlePackage and return []pkg.Package is confusing and seems to be more of a convenience for the caller. I think to make this more clear on all fronts changing the signature to newPackage(...) *pkg.Package makes the most sense, and the caller would optionally return the slice result to fulfill it obligations as a parser function.
| Coordinates `cyclonedx:""` // Empty string here means there is no intermediate property name, e.g. syft:locations:0:path without "coordinates" | ||
| // note: it is IMPORTANT to ignore anything but the coordinates for a Location when considering the ID (hash value) | ||
| // since the coordinates are the minimally correct ID for a location (symlinks should not come into play) | ||
| VirtualPath string `hash:"ignore" json:"virtualPath,omitempty"` // The path to the file which may or may not have hardlinks / symlinks | ||
| ref file.Reference `hash:"ignore"` // The file reference relative to the stereoscope.FileCatalog that has more information about this location. | ||
| VirtualPath string `hash:"ignore" json:"-"` // The path to the file which may or may not have hardlinks / symlinks |
There was a problem hiding this comment.
Why are we excluding VirtualPath from JSON output?
There was a problem hiding this comment.
Now that the source.Location is used instead of source.Coordinates in the json model, I updated the json struct tags to reflect the same data shape that is supported by the current schema. Coordinates don't convey the virtual path https://github.com/anchore/syft/blob/v0.77.0/syft/formats/syftjson/model/package.go#L29 so to be consistent this struct tag was changed to -.
We could change this, but I'd recommend that in a follow up PR.
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
spiffcs
reviewed
Apr 13, 2023
spiffcs
approved these changes
Apr 13, 2023
|
This one looks good to me - no other comments to add besides what already exists on the PR. I've started incorporating the location changes into the license evidence PR so 👍 thanks for making this clear and easy to follow |
spiffcs
added a commit
that referenced
this pull request
Apr 17, 2023
* main: (35 commits) Fix kernel cataloger test fixtures (#1742) feat: Support scanning license files in golang packages over the network (#1630) Add package-to-file location evidence relationships (#1698) Add Linux Kernel cataloger (#1694) Add annotations for evidence on package locations (#1723) add format make target (#1733) Update tests to not fail on Mac M1's. (#1730) chore(deps): update bootstrap tools to latest versions (#1728) Add support for nar files. (#1727) add highlevel details about catalogers (#1726) chore(deps): bump golang.org/x/net from 0.8.0 to 0.9.0 (#1722) chore(deps): update stereoscope to e95d60a265e384df29b7a139f5c5402d6ad72e06 (#1721) feat: gradle lockfile support (#1719) chore(deps): bump github.com/docker/docker (#1715) chore(deps): bump golang.org/x/mod from 0.9.0 to 0.10.0 (#1713) chore(deps): bump golang.org/x/term from 0.6.0 to 0.7.0 (#1714) chore(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0 (#1716) chore(deps): bump peter-evans/create-pull-request from 4 to 5 (#1712) chore: update tools-golang to v0.5.0 (#1717) Add Nix cataloger (#1696) ... Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
This was referenced Apr 18, 2023
This was referenced Apr 18, 2023
GijsCalis
pushed a commit
to GijsCalis/syft
that referenced
this pull request
Feb 19, 2024
* add location annotations + deb evidence annotations Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename LocationData struct and Annotation helper function Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add failing integration test for evidence coverage Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add evidence to aplm cataloger locations Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * change location annotation helper to return a location copy Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add evidence to binary cataloger locations Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * updated remaining catalogers with location annotations Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix unit tests Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump json schema Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * partial addressing of review comments Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename location.WithAnnotation Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the concept of location annotations, allowing arbitrary key-value pairs to be listed onto
source.Locationobjects. This PR also shows the first example of this with the dpkg cataloger:[ { "id": "3e9282034226b93f", "name": "adduser", "version": "3.118", "type": "deb", "foundBy": "dpkgdb-cataloger", "locations": [ { "path": "/usr/share/doc/adduser/copyright", "layerID": "sha256:ec09eb83ea031896df916feb3a61cefba9facf449c8a55d88667927538dca2b4", "annotations": { "evidence": "supporting" } }, { "path": "/var/lib/dpkg/info/adduser.conffiles", "layerID": "sha256:ec09eb83ea031896df916feb3a61cefba9facf449c8a55d88667927538dca2b4", "annotations": { "evidence": "supporting" } }, { "path": "/var/lib/dpkg/info/adduser.md5sums", "layerID": "sha256:ec09eb83ea031896df916feb3a61cefba9facf449c8a55d88667927538dca2b4", "annotations": { "evidence": "supporting" } }, { "path": "/var/lib/dpkg/status", "layerID": "sha256:ec09eb83ea031896df916feb3a61cefba9facf449c8a55d88667927538dca2b4", "annotations": { "evidence": "primary" } } ], "licenses": [ "GPL-2" ], ...Notes:
source.LocationSetrequires that the existingsource.Locationis hashable as to enable it's use as a key in a map. This isn't possible since annotations are themselves maps and are directly attached to the Location struct. For this reason I've split out position information and metadata within the Location struct and added those as embeddings.source.LocationSetmerges metadata for any two locations where the position information is the same, essentially merging the locations. This seems like the right call relative to the package struct, but this does not account nicely for duplicate keys with differing values, which will log a warning.