Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

ingest: Split monolith transform rule #44

Open
joverlee521 opened this issue Jun 8, 2022 · 0 comments
Open

ingest: Split monolith transform rule #44

joverlee521 opened this issue Jun 8, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@joverlee521
Copy link
Contributor

Context

Currently in the ingest pipeline, there is a single transform rule that runs a shell pipeline of multiple Python scripts. This works in the automated ingest pipeline, but may be tedious to debug when developing or when there's an error in the pipeline.

Description

We can split up the single rule into multiple rules by using Snakemake's piped outputs feature. I don't think anyone in the group has used this feature, so we don't know the pitfalls.

@joverlee521 joverlee521 added the enhancement New feature or request label Jun 8, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
No open projects
Development

No branches or pull requests

1 participant