Skip to content

Commit

Permalink
Fix inaccurate reporting of append dimension size in metadata. (#286)
Browse files Browse the repository at this point in the history
I noticed an issue where the shape of the array in the final dimension
was always reported as the number of frames acquired, which, in the case
of nontrivial internal dimensions, is just wrong. Opening up such a
dataset in napari will confirm.

---------

Co-authored-by: Justin Eskesen <jeskesen@chanzuckerberg.com>
  • Loading branch information
aliddell and jeskesen authored Aug 9, 2024
1 parent 80782e3 commit c54c8f3
Show file tree
Hide file tree
Showing 4 changed files with 387 additions and 2 deletions.
10 changes: 9 additions & 1 deletion src/writers/zarrv2.array.writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,15 @@ zarr::ZarrV2ArrayWriter::write_array_metadata_()

std::vector<size_t> array_shape, chunk_shape;

array_shape.push_back(frames_written_);
size_t append_size = frames_written_;
for (auto dim = config_.dimensions.begin() + 2;
dim < config_.dimensions.end() - 1;
++dim) {
CHECK(dim->array_size_px);
append_size = (append_size + dim->array_size_px - 1) / dim->array_size_px;
}
array_shape.push_back(append_size);

chunk_shape.push_back(config_.dimensions.back().chunk_size_px);
for (auto dim = config_.dimensions.rbegin() + 1;
dim != config_.dimensions.rend();
Expand Down
10 changes: 9 additions & 1 deletion src/writers/zarrv3.array.writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,15 @@ zarr::ZarrV3ArrayWriter::write_array_metadata_()

std::vector<size_t> array_shape, chunk_shape, shard_shape;

array_shape.push_back(frames_written_);
size_t append_size = frames_written_;
for (auto dim = config_.dimensions.begin() + 2;
dim < config_.dimensions.end() - 1;
++dim) {
CHECK(dim->array_size_px);
append_size = (append_size + dim->array_size_px - 1) / dim->array_size_px;
}
array_shape.push_back(append_size);

chunk_shape.push_back(config_.dimensions.back().chunk_size_px);
shard_shape.push_back(config_.dimensions.back().shard_size_chunks);
for (auto dim = config_.dimensions.rbegin() + 1;
Expand Down
1 change: 1 addition & 0 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ else ()
external-metadata-with-whitespace-ok
restart-stopped-zarr-resets-threadpool
repeat-start
metadata-dimension-sizes
write-zarr-v2-raw
write-zarr-v2-raw-chunk-size-larger-than-frame-size
write-zarr-v2-raw-with-even-chunking
Expand Down
Loading

0 comments on commit c54c8f3

Please # to comment.