Skip to content

Add Pipeline and Step Comment Attributes #1557

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 2 commits into from
May 7, 2025

Conversation

goodwillpunning
Copy link
Contributor

@goodwillpunning goodwillpunning commented May 2, 2025

Changes

This PR adds a comment attribute to the PipelineConfig and Step classes.

What does this PR do?

Adds the ability for pipeline authors to document what a pipeline and/or step does for future maintainers (including themselves).

Relevant implementation details

For pipelines containing many steps, it becomes unclear what effect a pipeline or step has. To discern what a step does, for example, a pipeline maintainer must look at the extract_source attribute, navigate to the Python or SQL file, and then read through the file to determine what a step does.

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #1468.

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs remorph ...
  • Augments the PipelineConfig and Step classes with a metadata attribute.

Tests

  • manually tested
  • added unit tests
  • added integration tests

Copy link

github-actions bot commented May 2, 2025

✅ 15/15 passed, 1 skipped, 15s total

Running from acceptance #565

Copy link
Collaborator

@sundarshankar89 sundarshankar89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@gueniai gueniai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@goodwillpunning goodwillpunning added this pull request to the merge queue May 7, 2025
Merged via the queue into main with commit 78ce9ec May 7, 2025
7 checks passed
@goodwillpunning goodwillpunning deleted the feature/add_pipeline_step_comments branch May 7, 2025 11:44
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE]: Generate local Profiler reports
4 participants