Skip to content

Add Pipeline and Step Comments #1556

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
wants to merge 1 commit into from

Conversation

goodwillpunning
Copy link
Contributor

Changes

This PR adds a comment attribute to the PipelineConfig and Step classes.

What does this PR do?

Adds the ability for pipeline authors to document what a pipeline and/or step does for future maintainers (including themselves).

Relevant implementation details

For pipelines containing many steps, it becomes unclear what effect a pipeline or step has. To discern what a step does, for example, a pipeline maintainer must look at the extract_source attribute, navigate to the Python or SQL file, and then read through the file to determine what a step does.

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #1470.

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs remorph ...
  • Augments the PipelineConfig and Step classes with a metadata attribute.

Tests

  • manually tested
  • added unit tests
  • added integration tests

Copy link

github-actions bot commented May 2, 2025

✅ 15/15 passed, 1 skipped, 10s total

Running from acceptance #534

@goodwillpunning
Copy link
Contributor Author

Closing in favor of #1557 (has a signed commit this time).

@goodwillpunning goodwillpunning deleted the feature/add_pipeline_comments branch May 2, 2025 16:36
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE]: Schedule job to run extraction
3 participants