Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[Enhancemant](HLL) support export doris hll/bitmap data type #153

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

GoGoWen
Copy link

@GoGoWen GoGoWen commented Nov 3, 2023

Proposed changes

when we export data from table which has HLL/BITMAP to iceberg table, we can not export the HLL and BITMAP column. we just want to export, as we have another way to query the HLL/BITMAP in Doris by give the column type HLL and BITMAP.

fix issue: apache/doris#26757

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JNSimba
Copy link
Member

JNSimba commented Nov 6, 2023

Can you post a usage example for this? Because I saw that Doris does not support converting Bitmap and HLL type data into arrow for spark to read ?
https://github.com/apache/doris/blob/master/be/src/util/arrow/row_batch.cpp#L47-L128

@GoGoWen
Copy link
Author

GoGoWen commented Nov 8, 2023

Can you post a usage example for this? Because I saw that Doris does not support converting Bitmap and HLL type data into arrow for spark to read ? https://github.com/apache/doris/blob/master/be/src/util/arrow/row_batch.cpp#L47-L128

Later we can make a pr to support doris query Bitmap and HLL type, now we want to export the data to iceberg table because of disk capacity limitation
, and which also need to support query in doris with iceberg table.

@GoGoWen
Copy link
Author

GoGoWen commented Nov 9, 2023

doris related issue: apache/doris#26757

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants