Question about experimental V3 support in zarr-python

[maxrjones]

First, thanks for this community's work on Zarr!

I've been using both zarr-python and zarrita to experiment with V3 data and noticed some differences in the output paths (MVCE below), which seems to reflect some of the more recent changes made to the V3 spec (e.g., zarr-developers/zarr-specs#175). Given that the V3 support is experimental, I wasn't sure if this would be an appropriate issue. My specific question is whether a PR would be welcome for removing the /root prefix for paths for V3 data?

I'm also hoping someone could clarify whether this restriction could be looser after zarr-developers/zarr-specs#177?:

zarr-python/zarr/_storage/store.py

Lines 212 to 215 in 4132f36

	In spec v3, keys can only start with the prefix meta/, data/ or be
	exactly zarr.json and should not end with /. This should not be exposed
	to the user, and is a store implementation detail, so this method will
	raise a ValueError in that case.

Zarr

import os
import numpy as np

os.environ["ZARR_V3_EXPERIMENTAL_API"] = "1"
os.environ["ZARR_V3_SHARDING"] = "1"

import zarr
from zarr._storage.v3 import DirectoryStoreV3

print(f"Version: {zarr.__version__}")

shape=(4,2)

store = DirectoryStoreV3(f"zarr_python.zarr")
arr = zarr.create(shape=shape, path="a", chunks=(2,2), dtype="i4", store=store)
arr[:] = np.ones(shape)

Version: 2.14.2

tree zarr_python.zarr

zarr_python.zarr
├── data
│   └── root
│       └── a
│           ├── c0
│           │   └── 0
│           └── c1
│               └── 0
├── meta
│   └── root
│       └── a.array.json
└── zarr.json

8 directories, 4 files

Zarrita

import zarrita
import numpy as np
store = zarrita.LocalStore('zarrita.zarr')

shape = (4,2)

a = zarrita.Array.create(
    store,
    path="a",
    shape=shape,
    dtype='int32',
    chunk_shape=(2,2),
    codecs=[zarrita.codecs.gzip_codec(level=1)],
)
a[:] = np.ones(shape, dtype='int32')

tree zarrita.zarr

zarrita.zarr
└── a
    ├── c
    │   ├── 0
    │   │   └── 0
    │   └── 1
    │       └── 0
    └── zarr.json

[jbms]

zarr-python currently implements only an earlier draft of the zarr v3 specification, which is quite different from the finalized v3 spec.

I think PRs to make zarr-python conforming with the v3 spec would be welcome, though.

[rabernat]

Hi Max! 👋

As Jeremey said, unfortunately we have so far not been able keep up with the changes in V3 spec. However, we are making a big push to catch up. I invited you to a zarr python dev meeting today in case you're interested in participating.

[maxrjones]

Great, thanks both! I'll plan to attend today

[grimbough]

Thanks for this thread. I ran into the same inconsistencies between the V3 spec and the output of zarr-python when I started to create some example V3 files for testing my Rarr implementation. Maybe beyond the scope of this discusion, but do you have a feeling for which implementation is currently closest to the published spec?

[maxrjones]

We've found zarrita to be closest to the published spec. See https://github.com/freeman-lab/zarr-js/blob/main/generate/v3.py for the code used to generate datasets for testing the zarr-js implementation. As mentioned in their README, zarrita's experimental and changes really rapidly so we eventually found it necessary to freeze at a specific commit for testing (b88ae5ca02917e24e716855f15ffaf020b95bb1a for the data in https://github.com/freeman-lab/zarr-js/tree/main/data/v3)

[grimbough]

Thanks @maxrjones That's really helpful to see the code you're using to generate the test data. That will save me a lot of time in getting some test cases working.

[jhamman]

As mentioned by @jbms, Zarr-python does not currently have a V3 spec compliant implementation. We are about to kick off a focused effort to fix that! #1480. We can keep this thread up to date as that process moves forward.