Preparations for ESGF phase out of CMIP3 and CMIP5 data

[rswamina]

ESGF nodes may not host CMIP5 and CMIP3 data from early 2025. This has implications for several of our recipes that use this data. The proto steering , tech lead and science lead groups have discussed this and some ideas were put forth for discussion:

• Remove recipes using this data vs keeping them for scientific continuity.
• Keep an archive of original recipes especially if used in published outputs.
• Try to convince the ESGF developers that they should keep the data available because it is necessary for reproducible science. Papers like GMD demand robust archives for publications and it looks like ESGF is not capable of develivering this: https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
• Use the (fraction of?) the data that is archived at local centers, though it is not yet clear how this will be made available.
• Re-publish relevant data on a more reliable repository, e.g. Zenodo, and update ESMValCore so it can download from there
• Update recipes to CMIP6 where it makes sense to do so (e.g. general recipes, example recipes)
• Retire recipes reproducing specific papers

This is a community discussion for the @ESMValGroup/esmvaltool-developmentteam to contribute thoughts and ideas on this issue so please post them here.

[nhsavage]

Ideally we would persuade ESGF developers to at least retain CMIP5 data. We typically use a combination of CMIP5 and CMIP6 for our analysis and if ESMValTool were no longer to support CMIP5 it would mean we would have to reconsider adoption of ESMValTool for some of our work. This might change when CMIP7 fast track is published as we might then want to use CMIP6+7 but that is some way off.

I am not sure that Zenodo is a suitable solution as the size of the data sets is fairly limited by our standards: https://support.zenodo.org/help/en-gb/1-upload-deposit/80-what-are-the-size-limitations-of-zenodo 50GB per record with a maximum of 100 files. We grant a one-time quota increase of up to 200GB. We can use multiple records but I think that might fall foul of the "fair use" case. For context MPI-ESM-MR daily data for 1 variable for the historical period is 4Gb. If we restricted ourselves to a subset of variables, we might fit each model/experiment/realisation in a single record but we'd probably need to ask for permission.

More generally on recipies used for papers - I'd suggest that as long as the code is on the trunk, then the requirement to publish code is satisified if you get it in a published release and then you cite a specific release. If future releases remove that recipie then it is not relevant as the code is still available in a release. The idea that the code behind a paper can somehow magically work forever despite constant changes in IT is hugely optimistic.