Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

soorgeon cut #48

Open
edublancas opened this issue May 22, 2022 · 0 comments
Open

soorgeon cut #48

edublancas opened this issue May 22, 2022 · 0 comments

Comments

@edublancas
Copy link
Contributor

edublancas commented May 22, 2022

We should add one command to cut existing tasks into smaller pieces so users can take advantage of incremental builds.

soorgeon cut {task-name}

For example, if a user has a pipeline.yaml:

# pipeline.yaml
tasks:
  - source: script.py
     product: report.htm

We could run soorgeon refactor on script.py (but using the Python API directly), and break it into smaller pieces. Then manipulate the pipeline.yaml

# pipeline.yaml
tasks:
  - source: load.py
     product: load.html

  - source: clean.py
     product: clean.html

soorgeon already contains all the necessary elements to perform the refactoring so it's just a matter of experimenting a bit. The idea is that sometimes notebooks grow, so having an easy way to refactor them is great.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant