ESMValTool long-term strategy

[hb326]

We have talked from time to time over the last year about coming up with a strategy (short-, mid- and long-term) for ESMValTool and ESMValCore developments. Until we have a Consortium Agreement in place, and a Steering Committee installed, this will be the place to introduce and collect ideas and visions.

Latest draft: https://docs.google.com/document/d/1bDl9YTw00bZp1YW0WfM8qLRzM4UrA5hRAPTbRAgY_z4/edit?usp=sharing

We are sharing while still in quite rough draft form, in order to capture wider perspectives before it is too mature. Please suggest changes or additions to the text, or add comment. To do so please # with a google / gmail account so that we can discuss ideas further with you.

[hb326]

@axel-lauer, @alistairsellar, @zklaus, @rswamina: I did it! It is online!
Feel free to add, change, update...

[bouweandela]

Let's tag the @ESMValGroup/esmvaltool-developmentteam to draw some attention to this admirable initiative.

[Peter9192]

I was playing around with the updated ESMValCore API and I think that's a great feature. It would be awesome if it could be extended to the point that you eventually can generate a complete recipe programmatically. Would something like that fit on the roadmap? Another feature that might be interesting is to have ESMValCore running as a web service so you could retrieve pre-processed data via an API call.

[axel-lauer]

Cool! Eventually, we could think about either merging this with the "roadmap" we started on the ESMValTool website or removing the table on the website if this becomes our new roadmap.

[remi-kazeroni]

Indeed, it would be nice to have only one version of the "roadmap" or "strategy" and keep the content of the 2 lists synchronized. I think the roadmap on the website might be more visible, in particular for outsiders. I got asked in a project if we plan to extend our roadmap beyond end of 2022. But it's nice to have this page for discussion 👍

[valeriupredoi]

now, if only we had a nice document, that sets up groups of people to lead/do/steer/brainstorm stuff, and helps setting up objectives and means to achieve them and such...I wonder what that sort of document would be called - Magna Carta, Constitution, Hitchhiker's Guide to the Galaxy? 😁

[schlunma]

Couple of ideas that came to my mind or have been discussed during the workshop:

1. Add the option to use a "generic" preprocessor, i.e., a custom function to the preprocessing pipeline. This would increase the flexibility of our preprocessing pipeline dramatically and could make the life of advanced users much easier.
2. Add a "plot" preprocessor to quickly create plots without having to configure the diagnostic. Maybe the iris.quickplot would be a good starting point.
3. In the context of monitoring a running simulation where more and more data is available over time: allow re-using preprocessed files from earlier runs. This is highly non-trivial and probably also not necessary for non high-resolution simulations, so something for the farer future.
4. Add an option to specify the path of the final index.html (with correct links to the files in it), this can make live monitoring a bit easier.
5. Group recipes (in doc + code) into overarching categories (e.g., papers for recipes that "just" reproduce papers, ipcc for IPCC-related recipes, general for general-purpose recipes, ...) to make it easier for new users which recipes are actually interesting.

[alistairsellar]

Thanks for everyone's input to long-term strategy thinking, either via the ESMValTool teams, or this thread. The proto-Steering Group have developed these into a draft document, which we would now like to share with you for comment: https://docs.google.com/document/d/1bDl9YTw00bZp1YW0WfM8qLRzM4UrA5hRAPTbRAgY_z4/edit?usp=sharing

We are sharing while still in quite rough draft form, in order to capture wider perspectives before it is too mature. Please suggest changes or additions to the text, or add comment. To do so please # with a google / gmail account so that we can discuss ideas further with you.