-
Notifications
You must be signed in to change notification settings - Fork 853
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Allow to create record batch without validation #7071
Comments
Evaluating the expressions in the comparator will be extremely inefficient:
I'm not sure your precise use-case, but typically one would evaluate the expressions on the input batches and then process the outputs in one shot. I struggle to think of an example where slicing in a hot loop is not a code smell. |
I could materialise every transformation on each item before hand (when possible)
The use case is similar to array_sort with custom compare function in spark |
https://docs.rs/arrow-ord/latest/arrow_ord/ord/type.DynComparator.html Is how I would recommend providing a similar interface in a columnar execution environment (spark is not a vectorized engine and instead relies on JIT compilation). So users would instead provide a function with a signature similar to https://docs.rs/arrow-ord/latest/arrow_ord/ord/fn.make_comparator.html |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to sort a list based on custom comparator function that is made out of data fusion expressions
To do a comparison sort we obviously need 2 items from the list, so I created the following schema with 2 identical columns:
and then I create a record batch with the schema above for every
sort_by
callas you can see I call
try_new
a lot of times which does a lot of checks that will evaluate to the same thing over and over againDescribe the solution you'd like
It would be helpful to have a function called
new_unchecked
that isunsafe
and doesn't do any checksDescribe alternatives you've considered
Just calling
try_new
over and over againThe text was updated successfully, but these errors were encountered: