Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

arrow2_convert primitive (de)serialization performance issues #1712

Closed
Tracked by #1899
teh-cmc opened this issue Mar 27, 2023 · 3 comments
Closed
Tracked by #1899

arrow2_convert primitive (de)serialization performance issues #1712

teh-cmc opened this issue Mar 27, 2023 · 3 comments
Labels
🏹 arrow Apache Arrow 🪳 bug Something isn't working 🚀 performance Optimization, memory use, etc

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Mar 27, 2023

Serializing and (maybe?) deserializing primitive types through arrow2-convert's APIs is much slower than doing so using raw arrow2 arrays directly.

See #1742 for detailed benchmarks.

@teh-cmc teh-cmc added 🪳 bug Something isn't working 🏹 arrow Apache Arrow 🚀 performance Optimization, memory use, etc labels Mar 27, 2023
@teh-cmc teh-cmc changed the title Creating primitive component arrays through high-level APIs is extremely slow arrow2_convert primitive (de)serialization performance issues Mar 31, 2023
@teh-cmc
Copy link
Member Author

teh-cmc commented Apr 1, 2023

  • Thankfully only really impacts serialization, not deserialization
  • Need to introduce specialized fast paths for serializing control & component columns that are backed by primitive types, which shouldn't be much of an issue

@teh-cmc
Copy link
Member Author

teh-cmc commented Apr 4, 2023

@teh-cmc
Copy link
Member Author

teh-cmc commented Oct 9, 2023

Closing: we're moving away from arrow2-convert.

@teh-cmc teh-cmc closed this as not planned Won't fix, can't repro, duplicate, stale Oct 9, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
🏹 arrow Apache Arrow 🪳 bug Something isn't working 🚀 performance Optimization, memory use, etc
Projects
None yet
Development

No branches or pull requests

1 participant