Faster flywire_partner_summary queries #205
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
jefferis
commented
Aug 12, 2024
jefferis
commented
- chunking eg 20 neurons at a time and sending to cave can give 10x speedups
- but need to be careful about CAVE row limits
- awkwardly you can't just increase your offset and get more rows
- in the process implemented offset/limit arguments for generic CAVE queries
- ... and the ability to fetch all rows of a query (but this doesn't work for the synapse table!)
* instead of doing one neuron at a time, idea is to make some large chunks * plan was then to repeat these chunk requests if they overflowed * however then realised that I can't tell if there's an overflow because some rows are filtered out before returning * instead I think I should add logic to the basic cave_query to fetch more rows using the offset option
* but this doesn't work for synapse table (since it returns rows in random order)
* some testing is still in order when row limits are exceeded
* otherwise looks concerning
* but probably need to sort results at the end to be sure
|
this has been failing to run due to auth issues that are unrelated to content of PR |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.