Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Function to create zarr from netcdf #1034

Closed
mnurisso opened this issue Mar 27, 2024 · 3 comments · Fixed by #1068
Closed

Function to create zarr from netcdf #1034

mnurisso opened this issue Mar 27, 2024 · 3 comments · Fixed by #1068
Assignees
@oloapinivad
Labels
framework functions common to all diagnostics improvement Improvement to existing functionality

Comments

@mnurisso
Copy link
Collaborator

Feature request

We've on the AVISO catalogue folder of lumi catalogue a utility used to create a json file from a list of netcdf.
We want to adapt it to speed up data of ERA5 and maybe to long LRA.
@mnurisso mnurisso added the improvement Improvement to existing functionality label Mar 27, 2024
@mnurisso mnurisso self-assigned this Mar 27, 2024
@mnurisso mnurisso added the framework functions common to all diagnostics label Mar 27, 2024
@oloapinivad oloapinivad mentioned this issue Mar 27, 2024
Closed
10 tasks
@oloapinivad
Copy link
Collaborator

I have been doing some tests and there seem to be an important limitation in zarr which might be a bottleneck for our implementation. The size of the chunks must be always the same for the MultiZarr, which means that we cannot create a single file for the LRA which has both 1 timestep and 12 timesteps files.

I am investigating this more...

@oloapinivad
Copy link
Collaborator

A possible solution is to split in two blocks the json creation, one for the yearly files and one for the monthly files...

@oloapinivad oloapinivad mentioned this issue Apr 9, 2024
Merged
5 tasks
@oloapinivad
Copy link
Collaborator

fsspec/kerchunk#447 here some investigation on the limitation of zarr

@mnurisso mnurisso assigned oloapinivad and unassigned mnurisso Apr 30, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@oloapinivad oloapinivad

Labels
framework functions common to all diagnostics improvement Improvement to existing functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Create Zarr function to create json references and LRA feature DestinE-Climate-DT/AQUA
2 participants
@oloapinivad @mnurisso