Improve the INPUT workflow

[clizbe]

The input workflow (raw to model) needs improvements and consolidation. This will take many iterations, but an initial effort should improve the user experience in the short term as our community grows.
This epic focuses mainly on the user-facing features and workflow structure, not what's under the hood (not all TIO should be listed here).

Workflow Overview

Phase 1: Clean & Documented (but manual)

• Put the defaults in a JSON file #1039
• Define & document the User Format #1038
• Define scenario specification workflow #414
  • Create switches that explicitly enable/disable features #996
• Document all modelling features and required data (decision tree) #997
• Add a description of the tables available in the connection for the docs #941
• How does the user specify that an energy carrier acts like gas/electric?

Phase 2: A Bit Friendlier

• Implement data validation #461
• Create function(s) in TIO to convert UF tables into MF tables
• Desired data transformations in analysis workflow TulipaIO.jl#11
• Add function add_asset(type, args) that requires data but puts it into all the right tables?
• Add function add_profile?
• Add function add_flow?

Phase 3: Clear, Structured, and Smooth

• Create a site/notebook/template/repo for the whole pipeline documentation #685
• (Probably) Create a wrapper function to convert all tables UF -> MF at once
  • Once this is done, if it's completely automatic, consider moving ModelFormat inside TEM and just show UserFormat to the user - then conversion happens as pre-processing when they call the model
• Invent a way to help the user organize/track scenarios
• Programmatically generate partition data for editing TulipaIO.jl#29
• Data provenance and preparation #875
• Define as input the resolution input data in the rep_periods_data from somewhere in the pipeline #686

Related

• OBZ repo: https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudy/tree/main
• Improve the OUTPUT workflow #975
• Review if it makes sense to split "user input files" into information for new capacity and existing capacity #774
• Create argument to function that create DB connection to allow a name to be given #635
• New user experience #916
• Draft User Workflow #700
• How should we handle data duplication, provenance, metadata? #499
• Simple cases/convenience of missing info #664
• Explicitly list the permissions of each workflow #68

[clizbe]

I've been trying to imagine how an analyst would start from scratch.
Below is a pseudocode of my imagined workflow, using TIO to generate templates that include the columns the user needs.
We could also consider a template repo, but then I think the templates would be set and not dynamic to the user needs.

Not sure if this would all become a nightmare with versioning...

# Example Workflow (with imaginary parts)

using TulipaIO: TIO
using DuckDB: DDB
using TulipaEnergyModel: TEM

## Create Templates
# - I think it's necessary to first start something in DuckDB?
connection = DBInterface.connect(DuckDB.DB)
# - This function would create tables in DuckDB according to the choices in the analysis_features 'file'
TIO.create_template_tables("analysis_features.csv", connection; table = all)
# - But these tables will be empty except the selection of column names, 
#   so maybe they also need some more input like assets or something

# - This function would be the same as create_template_tables but works on an existing table to add/remove columns
#   Theoretically allowing the analyst to adjust their analysis features on the fly 
TIO.change_feature_columns("analysis_features_v2.csv", connection)

## Import raw data

## Manipulate data into User Format
# - Function to map a column of raw data onto the corresponding UserFormat column 
#   (Maybe this could enable more automatic mapping scripts...)
TIO.map_column(col_raw, col_UF; key_raw, key_UF)
# - Function that fills in defaults - either with auto defaults or specified by user 
TIO.fill_defaults(; defaults_file)
# - Other stuff

## Scenarios?
# - Some way of setting Base scenario and creating Alternatives?
# - For now it seems this is entirely on the user to run the workflow for each scenario...

## Preprocessing?
# - TulipaClustering
# - Time partitions? Created somehow in templates?

## Convert to Model Format
TIO.userformat_to_modelformat(connection)

## Run Model
TEM.run_scenario(connection; optimizer, params...)

This idea is taking shape for me, but I need feedback in case it's the wrong direction.
Let's discuss in person in the new year!

[clizbe]

If we built a template repo, it could have a few basic files and a Pluto workflow that walks them through...

1. Fork this repo
2. Change the analysis_features and defaults files as necessary
3. Open the Pluto notebook analysis template and follow the steps

And then maybe the Pluto notebook includes the template workflow above with instructions?
But I dunno if that's actually useful / maintainable.