Data provenance and preparation

[abelsiqueira]

Description

Based on the discussion that we had on 14/Oct/2024 in person, here is an overview of the things we need.
Notice that these happen before TulipaEnergyModel or even TulipaClustering, in general.

Sub issues

• UI-Friendly TulipaIO
  There needs to be a way to convert from a user-oriented data file (e.g., wide format) to a Tulipa-oriented format
• Find a data storage solution
  The EU data is too large and needs to be moved to some place to be easier to download and to prevent downloading it with every clone.
  Furthermore, we want to allow users to easily download the data for examples
• Limit data-loading to only the necessary queries
  Somewhere (where?), the whole data will be stored (e.g., 50 years of data). We only want to load into memory a selection of this data (e.g., 1 year). How to do it? Does this imply a file format or style or whatever for expected data storage?
• "Source of truth"
  How can we set up a source of truth kind of data? Consider the situation where the correct data is made available by some "higher" entity. This data is collected by some TNO employee, and then moved into another place/format that is considered the "truth".
  What can be used for this? Microsoft Access is the current solution.

Related:

• Define scenario specification workflow #414

[clizbe]

For data storage I think we should consider Zenodo. It takes data in all forms, complete or in-progress. We can also have it be private at first we want, and only share it with certain people.