Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Support External Catalogs in DBT Adapter #22

Open
JadCham opened this issue Sep 13, 2023 · 3 comments
Open

Support External Catalogs in DBT Adapter #22

JadCham opened this issue Sep 13, 2023 · 3 comments

Comments

@JadCham
Copy link

JadCham commented Sep 13, 2023

Feature request

Is your feature request related to a problem? Please describe.

No, but a blocker to use the DBT adapter.

Describe the solution you'd like

The SR DBT Connector supports reading from external tables/catalogs and writing materialized tables into SR internal storage.

We plan to use SR for querying our external Hive tables with disk caching instead of Presto/Athena. We have many processes around using DBT to transform data in Athena. It would be great to be able to replace them with SR instead.

Describe alternatives you've considered

No alternatives at the moment, DBT runs happen in Athena and we read the resulting data in SR.
@Astralidea Astralidea self-assigned this Sep 13, 2023
@geoHeil
Copy link

geoHeil commented Feb 8, 2024

Ideally nosql systems such as Elastic or Kafka can also be supported and even the ingestion from kafka controlled form DBT

@alberttwong
Copy link
Contributor

@wangyouzhuo for visibility

@alberttwong alberttwong transferred this issue from StarRocks/starrocks Mar 11, 2024
@alberttwong
Copy link
Contributor

I saw that there was a readme section on supporting hive external catalog. Can you try it out?

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@alberttwong @geoHeil @Astralidea @JadCham