Skip to content

Refactor lineage JSON serilization with a root element including version, kind and spec properties #6075

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented May 14, 2025

This PR refactors the Lineage data encoder to serialize lineage records using the following structure:

{
  "version": <lineage version>,
  "kind": <lineage record class>,
  "spec": <record description>
}

For instance a FileOutput type is serialized as:

{
  "version": "lineage/v1beta1",
  "kind": "FileOutput",
  "spec": {
    "path": "/home/jorgee/nextflow_tests/provenance-test/results/samples.json",
    "checksum": {
      "value": "30a926dd315d4f4b9d6389803a1324ff",
      "algorithm": "nextflow",
      "mode": "standard"
    },
    "source": "lid://31e092e42c71b9545a9c5fa91689340c",
    "workflowRun": "lid://31e092e42c71b9545a9c5fa91689340c",
    "taskRun": null,
    "size": 201,
    "createdAt": "2025-05-14T10:25:51.563182873+02:00",
    "modifiedAt": "2025-05-14T10:25:51.563182873+02:00",
    "labels": [
      "foo"
    ]
  }
}

jorgee added 2 commits May 14, 2025 09:53
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Copy link

netlify bot commented May 14, 2025

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 9f7c0fc
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6825a02197b0a000086ce18d
😎 Deploy Preview https://deploy-preview-6075--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@pditommaso
Copy link
Member

This looks nice, likely we needed to update the version to v1beta2 if we go ahead with this. Also in a ideal world, it should be possible to keep backward compatibility with previous spec version, but I can understand that cannot be that simple

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>
@jorgee
Copy link
Contributor Author

jorgee commented May 15, 2025

For these PR, as there aren't changes in the model I think there is no need to update the version. I can make able to read both with spec and without with a small change. However, with changes such as in #6069 a mechanism for version compatibility is required.

@bentsherman
Copy link
Member

It might be enough to simply ignore records that have an incompatible version and warn about them so that the user is aware. At least while the version is in beta. Maybe in the future we could add backwards compatibility, e.g. v1 / v2 data models, but hopefully we can iron out any changes in the beta period

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants