Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Refactor Pipeline #8

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Conversation

DannyGJdeJong
Copy link

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.

  • Python version update
    • Update all the pipeline components to run using Python 3.13
    • Update development version to Python 3.13
    • Update dependencies
  • Schema refactor
    • The transform component now uses a common schema
    • The CLI uses the same common schema
  • Transform component
    • Transform components are now merged between platforms
    • The timestamp of an event is now included in the hash
  • Add CLI
    • Generate BigQuery schema definitions
    • Update the field hashes
    • Update table schemas
  • Add a script to populate the firestore with dummy data for easy testing

Type of change

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

A full recompute of all hashes is required with this release

@DannyGJdeJong DannyGJdeJong added the enhancement New feature or request label Jan 13, 2025
@DannyGJdeJong DannyGJdeJong requested a review from jorisbr January 13, 2025 14:35
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants