Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Document how parent context fields are included in the child record #301

Closed
MeltyBot opened this issue Jan 7, 2022 · 2 comments · Fixed by #2436
Closed

Document how parent context fields are included in the child record #301

MeltyBot opened this issue Jan 7, 2022 · 2 comments · Fixed by #2436
Labels
Accepting Pull Requests Documentation Improvements or additions to documentation good first issue Good for newcomers kind/Bug Something isn't working migrated from gitlab valuestream/SDK

Comments

@MeltyBot
Copy link
Contributor

MeltyBot commented Jan 7, 2022

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/303

Originally created by @edgarrmondragon on 2022-01-07 17:23:58


When user parent-child streams with the SDK as documented in https://sdk.meltano.com/en/latest/parent_streams.html

class GitlabStream(RESTStream):
    # Base stream definition with auth and pagination logic 
    # This logic works for other base classes as well, including Stream, GraphQLStream, etc.


class EpicsStream(GitlabStream):

    name = "epics"

    # ...

    def get_child_context(self, record: dict, context: Optional[dict]) -> dict:
        """Return a context dictionary for child streams."""
        return {
            "group_id": record["group_id"],
            "epic_id": record["id"],
            "epic_iid": record["iid"],
        }


class EpicIssuesStream(GitlabStream):
    # Note that this class inherits from the GitlabStream base class, and not from 
    # the EpicsStream class.

    name = "epic_issues"

    # EpicIssues streams should be invoked once per parent epic:
    parent_stream_type = EpicsStream  

    # Assume epics don't have `updated_at` incremented when issues are changed:
    ignore_parent_replication_keys = True

    # Path is auto-populated using parent context keys:
    path = "/groups/{group_id}/epics/{epic_iid}/issues"

    # ...

the docs fail to mention that group_id, epic_id and epic_iid are all added to the child epic_issues records and should the dev want them, they need to be included in the stream schema.

@MeltyBot
Copy link
Contributor Author

@stale
Copy link

stale bot commented Jul 18, 2023

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label Jul 18, 2023
@edgarrmondragon edgarrmondragon added the good first issue Good for newcomers label Jul 20, 2023
@stale stale bot removed the stale label Jul 20, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Accepting Pull Requests Documentation Improvements or additions to documentation good first issue Good for newcomers kind/Bug Something isn't working migrated from gitlab valuestream/SDK
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants