Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Intake_esm feature: Support for Kerchunk (reference) on s3 file system #603

Open
dwest77a opened this issue May 31, 2023 · 1 comment
Open

Comments

@dwest77a
Copy link

dwest77a commented May 31, 2023

Related Issue:
Intake_esm catalog hosted on an s3 file system which points to kerchunk files also hosted on the same file system cannot be accessed properly by existing intake_esm scripts.

Solution:
A small change to line 53 onwards in intake_esm/source.py:

if data_format == 'reference':
    if 's3://' in urlpath:
        import s3fs
        xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = s3fs.S3FileSystem(**kwargs).open(urlpath, **kwargs)
    else:
        xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = urlpath

Alternatives:
There might be other options to consider/features in intake_esm I'm not taking advantage of, if anyone has any insight let me know.
The above change relates to a later section:
fsspec.get_mapper('reference://', fo=ref ...)
Where for a kerchunk file on s3, ref needs to be the S3FileSystem object not the string path.

@agstephens
Copy link
Contributor

Hi @andersy005, hope you are well.
Does this fix look okay to you? It would help us with providing access to Kerchunk files over S3 (with authz). Thanks

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants