chore: cleanup decoder benches #3118
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3118 +/- ##
=======================================
Coverage 77.19% 77.19%
=======================================
Files 240 240
Lines 81517 81523 +6
Branches 81517 81523 +6
=======================================
+ Hits 62927 62935 +8
- Misses 15383 15384 +1
+ Partials 3207 3204 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
westonpace
force-pushed
the
chore/cleanup-decoder-benches
branch
from
ef58054
to
236c728
Compare
| .unwrap(); | ||
| const NUM_ROWS: u64 = 10000; | ||
| // Randomly generated strings are always 12 characters (at the moment) | ||
| const NUM_BYTES: u64 = NUM_ROWS * 16; |
There was a problem hiding this comment.
nit: maybe a comment 16 comes from 12 bytes data + 4 bytes offset?
|
|
||
| let lance_schema = | ||
| Arc::new(lance_core::datatypes::Schema::try_from(data.schema().as_ref()).unwrap()); | ||
| let encoding_strategy = default_encoding_strategy(LanceFileVersion::default()); |
There was a problem hiding this comment.
I think fixed size binary encoding is not enabled in LanceFileVersion::default(Lance 2.0)
| .unwrap(); | ||
| let lance_schema = | ||
| Arc::new(lance_core::datatypes::Schema::try_from(data.schema().as_ref()).unwrap()); | ||
| let encoding_strategy = default_encoding_strategy(LanceFileVersion::default()); |
There was a problem hiding this comment.
is this intentional that we only run bench_decode against 2.0(LanceFileVersion::default)?
|
related to #2725 |
broccoliSpicy
approved these changes
Nov 14, 2024
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This ensures that the setup routines happen inside of the benchmark function. Otherwise datagen will be run for all benchmarks during the collection phase and the system will run out of RAM.
This also changes the routines to use
FileMetadataCache::no_cacheas it turns outmokadoesn't immediately free its cache space on drop and, when used in a tight benchmark loop, this leads to running out of memory.