Does Zarr support direct reading of data from a chunk?

[Peterpan-Coder]

Hi,
Zarr supports several index APIs. For distributed cloud storage, I want to directly obtain the data stored in a chunk file based on the storage path name of the chunk file on the cloud, rather than decompressing the binary block file. I don't know if Zarr has the corresponding API.

For example, the following chunk files exists: 0.0, 0.1, 1.0, 1.1
Then I want to get the data stored in chunk 1.0 directly, rather than decompressing it through the compressor and reading it through numpy. Does zarr provide the above methods?

[rabernat]

Yes that is certainly possible. You could use the Storage API directly.

[Peterpan-Coder]

Thank you so much for your reply, I think I can do this:

chunk_data = store['0.0.0']   # type: bytes
buffer = zlib.decompress(chunk)    # if compressor is zlib
frombuffer = np.frombuffer(buffer, dtype='i4')    # if dtype is 'i4'

[rabernat]

Yes you can. My question is...why? 🙃 Why are you determined to bypass Zarr's own decoding pipeline? Just curious...

[Peterpan-Coder]

Thank you very much for your patient response.I just started learning zarr not long ago.
You said "Zarr's own decoding pipeline". Actually, I have been looking for it in the document, but I cannot find it. Specifications v2.0 has an example: Manually decompress a single chunk for illustration . Therefore, I mistakenly think that Zarr does not support it.I didn't find an example of a decoding pipeline.I hope I can get your guidance.

[rabernat]

The recommended way to get data out of a zarr array is by slicing, i.e.

array = zarr.open("/path/to/array")
data = array[:]  # get all the data
data = array[:10, 20:30]  # get a piece of data via multidimentional slicing

In this case, Zarr handles all of the i/o, decompression, etc.

Your requirement to

directly obtain the data stored in a chunk file based on the storage path name of the chunk file

is not supported by the public API. That's why I'm curious to understand more about your use case.

[Peterpan-Coder]

Thank you so much for receiving your reply again. Let me describe our idea.
Currently, our team has multi-dimensional big data that needs to be stored in the cloud. Because of the good features of Zarr, we began to learn about Zarr. One of the features of Zarr is flexible chunking and re-chunking. When data is flexibly read by block from the cloud storage, we understand that there may be APIs to read data from chunk files.