Tree option to omit array metadata (shape, dtype)

[alimanfoo]

When using the tree() function/method, currently arrays are printed with shape and dtype. This is useful diagnostic information but requires that the .zarray resource is retrieved and read for every array in the tree. This is not an issue for data stored locally, but can be an issue for remote storage as retrieving each .zarray resource will require a network round-trip.

Proposed to add an option meta=True to the tree() function/method, which if set to meta=False will omit the array metadata in the output, and thus building the tree representation will require only retrieving the list of keys from the store.

[alimanfoo]

cc @jakirkham - I was playing around with S3 storage and noticed that tree() can take quite a few seconds even for a relatively small store with only a few arrays - my fault for wanting the array metadata in there. For remote storage I though this could be a useful option to exclude array metadata and build tree representation pretty much as you had originally proposed without any metadata.

[jakirkham]

What if this metadata is also cached?

[alimanfoo]

Caching would help the second time you want to view the tree, but it would still be slow to build the tree first time round.

[jakirkham]

It’s true. Though tree isn’t the only thing that may use shape and dtype.

Any thoughts on making meta=False by default? Just thinking about simplifying what sounds like the common case (particularly for cloud storage).

[alimanfoo]

For caching metadata, the new LRUStoreCache class can be used, e.g.:

store = ...  # whatever underlying store
cache = zarr.LRUStoreCache(store, max_size=2**28)
root = zarr.group(store=cache)

If you have a lot of data and want to prevent chunk data from evicting metadata from the cache, then you could do this:

store = ...  # whatever underlying store
data_cache = zarr.LRUStoreCache(store, max_size=2**28)
meta_cache = zarr.LRUStoreCache(store, max_size=None)
root = zarr.group(store=meta_cache, chunk_store=data_cache)

Yes tempted to make meta=False the default, although the common case for me and my colleagues is fast local storage, so it's nice to see the shape and dtype.

[alimanfoo]

I'm coming around to having the default be as simple as possible, I think it is a good principle if the default is the cheapest/fastest thing to compute. I started trying to code this up but then realised that even if we don't show any array properties, the current implementation of the tree traversal still retrieves the metadata for each array and group in the hierarchy, because it relies on instantiating Array and Group objects, which always retrieve their metadata. It would be possible to implement the initial construction of the tree purely by scanning the store keys and looking for keys ending in '.zarray' and '.zgroup', but is a bit more of a change than I was originally thinking. Will mull on it for a bit.

…

On Wednesday, January 3, 2018, jakirkham ***@***.***> wrote: It’s true. Though tree isn’t the only thing that may use shape and dtype. Any thoughts on making meta=False by default? Just thinking about simplifying what sounds like the common case (particularly for cloud storage). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/224#issuecomment-354915846>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qk-Ek_2eUJIt71Vn0sMJCn78UJ-Iks5tGtChgaJpZM4RQjNG> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[jakirkham]

Should we make this a release blocker and/or should we add a note in tree’s docs that it’s behavior is still evolving?

[alimanfoo]

I'm inclined to push this to the next release, I'd really like to get 2.2 out. I'll add a note to the docstring if you're agreeable.

…

On Wednesday, January 3, 2018, jakirkham ***@***.***> wrote: Should we make this a release blocker and/or should we add a note in tree’s docs that it’s behavior is still evolving? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QnZJwyegt2pFZ2WERSA9t8bu_gfuks5tG44GgaJpZM4RQjNG> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[jakirkham]

Totally agree.

[PaulJWright]

Has there been any progress on this? I am noticing very large wall times (currently at ~6 min) with data stored on GCP. I am new to zarr in general, so any advice to reduce this would be great too!

gcs = gcsfs.GCSFileSystem(access='read_only')
store = gcsfs,GCSMap('file.zarr', gcs=gcs, check=False)
root = zarr.group(store)

%time print(root.tree())
/
 ├── 2010
 │   ├── 131A (47116, 512, 512) float32
 │   ├── 1600A (47972, 512, 512) float32
 │   ├── 1700A (46858, 512, 512) float32
 │   ├── 171A (47186, 512, 512) float32
 │   ├── 193A (47134, 512, 512) float32
 │   ├── 211A (47186, 512, 512) float32
 │   ├── 304A (47131, 512, 512) float32
 │   ├── 335A (47187, 512, 512) float32
 │   └── 94A (46930, 512, 512) float32
 ├── 2011
 │   ├── 131A (75200, 512, 512) float32
 │   ├── 1600A (75814, 512, 512) float32
 │   ├── 1700A (74839, 512, 512) float32
 │   ├── 171A (75660, 512, 512) float32
 │   ├── 193A (75664, 512, 512) float32
 │   ├── 211A (75678, 512, 512) float32
 │   ├── 304A (74199, 512, 512) float32
 │   ├── 335A (75624, 512, 512) float32
 │   └── 94A (75138, 512, 512) float32
 ├── 2012
 │   ├── 131A (76849, 512, 512) float32
 │   ├── 1600A (76630, 512, 512) float32
 │   ├── 1700A (69091, 512, 512) float32
 │   ├── 171A (76750, 512, 512) float32
 │   ├── 193A (76852, 512, 512) float32
 │   ├── 211A (76870, 512, 512) float32
 │   ├── 304A (76851, 512, 512) float32
 │   ├── 335A (76855, 512, 512) float32
 │   └── 94A (76878, 512, 512) float32
 ├── 2013
 │   ├── 131A (82719, 512, 512) float32
 │   ├── 1600A (83001, 512, 512) float32
 │   ├── 1700A (74989, 512, 512) float32
 │   ├── 171A (82633, 512, 512) float32
 │   ├── 193A (82716, 512, 512) float32
 │   ├── 211A (82746, 512, 512) float32
 │   ├── 304A (82715, 512, 512) float32
 │   ├── 335A (82723, 512, 512) float32
 │   └── 94A (82746, 512, 512) float32
 └── 2014
     ├── 131A (73605, 512, 512) float32
     ├── 1600A (73390, 512, 512) float32
     ├── 1700A (66326, 512, 512) float32
     ├── 171A (73487, 512, 512) float32
     ├── 193A (73603, 512, 512) float32
     ├── 211A (73617, 512, 512) float32
     ├── 304A (73602, 512, 512) float32
     ├── 335A (73604, 512, 512) float32
     └── 94A (73618, 512, 512) float32
CPU times: user 1min 11s, sys: 1.9 s, total: 1min 13s
Wall time: 6min 14s

[jakirkham]

This may be a different issue. Would suggest looking into consolidated metadata