Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Redshift web: Change SORTKEY encoding to RAW #129

Closed
mark-walle opened this issue May 30, 2022 · 0 comments
Closed

Redshift web: Change SORTKEY encoding to RAW #129

mark-walle opened this issue May 30, 2022 · 0 comments

Comments

@mark-walle
Copy link
Contributor

According to https://github.com/awslabs/amazon-redshift-utils/blob/master/src/Investigations/EarlyMaterialization.md it is possible that encoded sorkeys can cause "sortkey skew" and lead to excessive block I/O with early materialization.

In:

  • web/v1/redshift/sql-runner/sql/standard/02-page-views/01-main/00-setup-page-views.sql
  • web/v1/redshift/sql-runner/sql/standard/03-sessions/01-main/00-setup-sessions.sql
  • web/v1/redshift/sql-runner/sql/standard/04-users/01-main/00-setup-users.sql

The tables:

  • {{.output_schema}}.page_views{{.entropy}}
  • {{.output_schema}}.sessions{{.entropy}}
  • {{.output_schema}}.users{{.entropy}}

are created with explicit ZSTD encoding on the columns defined as primary sortkeys.

I believe these columns should have their encodings changed from ZSTD to RAW.

@colmsnowplow colmsnowplow changed the title Primary SORTKEY columns encoded with ZSTD could lead to sortkey skew Change SORTKEY encoding to RAW Jun 3, 2022
@colmsnowplow colmsnowplow changed the title Change SORTKEY encoding to RAW Redshift/web: Change SORTKEY encoding to RAW Jun 3, 2022
@colmsnowplow colmsnowplow changed the title Redshift/web: Change SORTKEY encoding to RAW Redshift web: Change SORTKEY encoding to RAW Jun 3, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant