Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Enable column casting #62

Merged
merged 1 commit into from
Jan 13, 2022
Merged

Conversation

MichailParaskevopoulos
Copy link
Contributor

@MichailParaskevopoulos MichailParaskevopoulos commented Jan 12, 2022

Changes:

  1. Table config now takes an argument column_casting.
  2. Schema parsing replaces the target table's schema based on the provided column_casting.
  3. The ODS/HDS upsert SQL-builder classes use the provided BigQuery function in the column casting when inserting the values from the landing zone table.

Column casting is currently supported for ODS (both full and incremental ingestions) and HDS (only snapshot).

Example column_casting:

column_casting:
  metrics_clicks: {function: "CAST({column} AS INTEGER)", output_type: INTEGER}

Use similar syntax for any BigQuery function. For instance:

column_casting:
  metrics_clicks: {function: "TIMESTAMP_MILLIS({column})", output_type: TIMESTAMP}

Resolves Jira issue: AF-73

Changes:
1) Table config now takes an argument column_casting.
2) Schema parsing replaces the target table's schema based on the provided column_casting.
3) The ODS/HDS upsert SQL-builder classes use the provided BigQuery function in the column casting when inserting the values from the landing zone table.

Resolves Jira issue: AF-73
@MichailParaskevopoulos MichailParaskevopoulos merged commit 99a7f0a into main Jan 13, 2022
@MichailParaskevopoulos MichailParaskevopoulos deleted the feature/column-casting-AF-73 branch January 13, 2022 15:16
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
area:core kind:enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants