Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[Feature Request]: IcebergIO should support table create-on-write #32677

Closed
2 of 17 tasks
kellen opened this issue Oct 7, 2024 · 1 comment
Closed
2 of 17 tasks

[Feature Request]: IcebergIO should support table create-on-write #32677

kellen opened this issue Oct 7, 2024 · 1 comment
Assignees
Labels
IcebergIO IcebergIO: can only be used through ManagedIO io java new feature P2

Comments

@kellen
Copy link
Contributor

kellen commented Oct 7, 2024

What would you like to happen?

BigQueryIO supports a CreateDisposition (NEVER, CREATE_IF_NEEDED) and any file-based IO requires no additional create step. Our (Spotify's) internal users are comfortable with and indeed expect this behavior.

We have added iceberg support to in scio spotify/scio#5494 but for example in the integration test we must issue createTable requests before the test is run:
https://github.com/spotify/scio/pull/5494/files#diff-5116ce37fed90178a3919b87a160d5795f51beb644117d63340ba55ffbf45b46R66-R71

I think it is reasonable to expect the catalog/namespace to exist but table creation could be automatically supported in the IO

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@ahmedabu98
Copy link
Contributor

Fixed by #32686

@github-actions github-actions bot added this to the 2.61.0 Release milestone Oct 9, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
IcebergIO IcebergIO: can only be used through ManagedIO io java new feature P2
Projects
None yet
Development

No branches or pull requests

3 participants