Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[pixels-core] add support for general data compression #16

Open
ray6080 opened this issue Jul 21, 2019 · 1 comment
Open

[pixels-core] add support for general data compression #16

ray6080 opened this issue Jul 21, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@ray6080
Copy link
Collaborator

ray6080 commented Jul 21, 2019 •
edited by bianhq
Loading

Although we have implemented basic encodings (like RLE) in pixels, it's also common to use general compressions (like gzip) along with encoding in columnar storage. However, we haven't implemented this yet.

This is a highly likely feature in the near future.
@ray6080
Copy link
Collaborator Author

ray6080 commented Jul 25, 2019

Shall we consider lz4 and zstd?
lz4 has a very fast decompression speed (but lower compression ratio) while zstd has a very high compression ratio and reasonable compression/decompression speed. They are both under BSD License and supported in Parquet.

Yes, you're right.
We should look into these two first when compression is on our plan table.

@bianhq bianhq changed the title [pixels-core] add support for basic data compression [pixels-core] add support for general data compression Oct 10, 2019
@bianhq bianhq added the enhancement New feature or request label Oct 21, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bianhq @ray6080