FileNotFoundError when using FsspecStore.from_url with a sync filesystem #2808

maxrjones · 2025-02-07T14:46:42Z

Zarr version

'3.0.3.dev9+g90cc08d4.d20250206'

Numcodecs version

'0.15.0'

Python Version

3.13

Operating System

Mac

Installation

Following https://zarr.readthedocs.io/en/latest/developers/contributing.html#creating-a-development-environment

Description

Writing an array to FsspecStore containing a AsyncFileSystemWrapper object does not currently work.

Steps to reproduce

import zarr
from zarr.storage import FsspecStore
from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
import tempfile
import numpy as np

with tempfile.TemporaryDirectory() as tmp_path:
    store = FsspecStore.from_url(f"local://{tmp_path}")
    assert isinstance(store.fs, AsyncFileSystemWrapper)
    assert store.fs.async_impl
    z = zarr.create_array(
        store=store,
        shape=(100, 100),
        chunks=(10, 10),
        dtype="f4"
    )
    # write to the array
    z[:, :] = np.random.random((100, 100))

Additional output

I added some print statements to check that the mode is correct for all the fsspec opener functions, which is consistently 'wb'. Here's the full traceback excluding those print statements:

  File "/Users/max/Documents/Code/zarr-developers/zarr-python/.vscode/test-async-wrapper.py", line 18, in <module>
    z[:, :] = np.random.random((100, 100))
    ~^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 2523, in __setitem__
    self.set_orthogonal_selection(pure_selection, value, fields=fields)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 2979, in set_orthogonal_selection
    return sync(
        self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype)
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/sync.py", line 142, in sync
    raise return_result
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/sync.py", line 98, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 1413, in _set_selection
    await self.codec_pipeline.write(
    ...<11 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 468, in write
    await concurrent_map(
    ...<6 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 68, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 66, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 418, in write_batch
    await concurrent_map(
    ...<8 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 68, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 66, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 416, in _write_key
    await byte_setter.set(chunk_bytes)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/storage/_common.py", line 144, in set
    await self.store.set(self.path, value)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/storage/_fsspec.py", line 276, in set
    await self.fs._pipe_file(path, value.to_bytes())
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/asyn_wrapper.py", line 27, in wrapper
    return await asyncio.to_thread(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/miniforge3/envs/zarr-python-tests/lib/python3.13/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/miniforge3/envs/zarr-python-tests/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/spec.py", line 787, in pipe_file
    with self.open(path, "wb", **kwargs) as f:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        path,
    ...<4 lines>...
        **kwargs,
    )
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 200, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 364, in __init__
    self._open()
    ~~~~~~~~~~^^
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 369, in _open
    self.f = open(self.path, mode=self.mode)
             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/70/hc_nynms54d8lp67z4rsfctc0000gp/T/tmpjmu3h688/c/0/2'

martindurant · 2025-02-07T14:56:24Z

The parent folder didn't get created, maybe?

maxrjones · 2025-02-07T15:00:25Z

The parent folder didn't get created, maybe?

Yeah I cannot say I understand it, but fwiw the metadata file gets created properly so my best guess is that there's an issue with the "c" chunks subdirectory.

martindurant · 2025-02-14T18:46:35Z

Try passing auto_mkdir=True to the filesystem, or explicitly create the directories first.

will-moore · 2025-02-18T10:11:07Z

I've come across the same error, originally coming from dask.array to_zarr() at dask/dask#11691 but now with updated zarr and removed the dask call to give this:

zarr==v3.03
fsspec==2024.12.0

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False)

z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

Similar stack trace as above

  File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 17, in <module>
    z = zarr.create(
        ^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 708, in create
    sync(
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 163, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1044, in create
    return await AsyncArray._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 611, in _create
    result = await cls._create_v3(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 754, in _create_v3
    await array._save_metadata(metadata, ensure_parents=True)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 1369, in _save_metadata
    await gather(*awaitables)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/abc/store.py", line 487, in set_or_delete
    await byte_setter.set(value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_common.py", line 144, in set
    await self.store.set(self.path, value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_fsspec.py", line 277, in set
    await self.fs._pipe_file(path, value.to_bytes())
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/asyn_wrapper.py", line 27, in wrapper
    return await asyncio.to_thread(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 787, in pipe_file
    with self.open(path, "wb", **kwargs) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 200, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 364, in __init__
    self._open()
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 369, in _open
    self.f = open(self.path, mode=self.mode)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.zarr/zarr.json'

will-moore · 2025-03-12T17:54:23Z

Just an update... Still seeing this with:

fsspec==2025.3.0
zarr==3.0.5

I don't see that I can use auto_mkdir=True in this example:

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False)
z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

martindurant · 2025-03-12T21:03:22Z

The following does work:

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False, storage_options={"auto_mkdir": True})
z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

Note that the fsspec config allows you to define that auto_mkdir should always be True for local FS, if you want.

will-moore · 2025-03-13T07:23:23Z

@martindurant Thanks for that. I've closed my original issue at dask/dask#11691 since it works with storage_options={"auto_mkdir": True}, but I won't close this issue just yet as it's not "my" issue originally. cc @maxrjones

maxrjones added the bug Potential issues with the zarr-python library label Feb 7, 2025

maxrjones mentioned this issue Feb 7, 2025

Support async FSMap objects in zarr.open #2774

Open

6 tasks

This was referenced Mar 1, 2025

Monthly issue metrics report #2877

Closed

Monthly issue metrics report sanketverma1704/zarr-python#8

Open

Monthly issue metrics report enthusiastdev121/zarr-python#17

Open

will-moore mentioned this issue Mar 13, 2025

Errors with Zarr v3 and da.to_zarr() dask/dask#11691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FileNotFoundError when using FsspecStore.from_url with a sync filesystem #2808

FileNotFoundError when using FsspecStore.from_url with a sync filesystem #2808

maxrjones commented Feb 7, 2025

martindurant commented Feb 7, 2025

maxrjones commented Feb 7, 2025

martindurant commented Feb 14, 2025

will-moore commented Feb 18, 2025

will-moore commented Mar 12, 2025

martindurant commented Mar 12, 2025

will-moore commented Mar 13, 2025

FileNotFoundError when using FsspecStore.from_url with a sync filesystem #2808

FileNotFoundError when using FsspecStore.from_url with a sync filesystem #2808

Comments

maxrjones commented Feb 7, 2025

Zarr version

Numcodecs version

Python Version

Operating System

Installation

Description

Steps to reproduce

Additional output

martindurant commented Feb 7, 2025

maxrjones commented Feb 7, 2025

martindurant commented Feb 14, 2025

will-moore commented Feb 18, 2025

will-moore commented Mar 12, 2025

martindurant commented Mar 12, 2025

will-moore commented Mar 13, 2025