Update signac's internal schema

[vyasr]

This issue is a meta-issue meant to encompass a few different points relating to changes to the internal schema that we're hoping to have in place for signac 2.0. Ideally, we'd like to synchronize updating from schema version 1 to version 2 with the 2.0 release of the package to simplify the migration for users. Resolution of this issue should imply resolution of at least #423, #390, #541, and #197. If possible, we may also want to resolve #283, but that decision can be made independently. Additionally, we will need to make a final decision on whether our config file should be signac.rc or .signac/config, for which I would like to have a vote of all committers; this can be handled at the end once most of the other issues have been tackled. By necessity, this issue involves changes to both signac and signac-flow, but hopefully it will move us a long way towards decoupling any dependencies.

The discussion on this thread indicated the need to implement the changes proposed in #197 via a proper schema migration using the tools implemented in #253. Additionally, it also highlighted the problems with how signac-flow uses the signac configuration files internally, which makes it difficult to independently make changes to either. As a result, I believe that we will need to make a number of changes before we can update our schemata in a consistent manner. I'm going to make a list of action items here, but anyone should feel free to edit it if they think I have any of the items wrong or in the wrong order. There are a few places where I haven't thought through this thoroughly yet and there will definitely be room for improvement.

Note: This checklist has been edited multiple times as plans have evolved.

• Remove signac-flow's dependence on signac's internal file schema and configuration. The principal connection that needs to be severed is flow's monkey-patching of the signac config for verifying flow config keys. Once that link is broken, signac-flow may rely on signac's config API alone without being tied to its internals. Completed in Replace monkey-patching of flow config with post-load validation signac-flow#573
• Make a new release of signac-flow that introduces this migration.
• Introduce a migration of signac to schema_version=2. Based on the list here as well as discussions on Refactor/reorganize root dir #517 and here, this process involves moving signac_shell_history and signac_sp_cache.json.gz, requiring the workspace to be an immediate subdirectory of the project called workspace (i.e. making workspace_dir no longer a mutable config variable), and possibly moving signac.rc to .signac/config.
• Reconsider moving from configobj to TOML or another format as discussed in Proposal: Revise the signac configuration system to use the TOML format #283. I think we have largely agreed in the past that such a move would be beneficial. One thing to keep in mind is that @csadorf mentioned keeping an eye on the overhead of supporting migrations. If we do choose to move formats, we will have to retain the code for the current configuration logic (including the configobj API) in perpetuity to support migrating, so that's something to keep in mind. We are no longer planning to pursue this for signac 2.0. See Proposal: Revise the signac configuration system to use the TOML format #283 for further discussion.

One way that we could make it easier to share config logic between signac and its dependents would be to modify the migration API in signac to work with an arbitrary namespace. The default namespace will be signac (for config files living in .signac as of schema_version 2, but could live elsewhere in the future if we change again), but it should support other namespaces as well, namely flow. That would be preferable to just copying the migration logic to signac-flow, if we can define this cleanly.

We could conceivably use TOML (or something else) for the flow config immediately upon switching; since we'd already be requiring a migration, it wouldn't be a huge issue to make that switch and test it in flow, then backport it to signac later.

[csadorf]

We could use one of the standard mechanisms for Python plugins to add migrations for auxiliary packages and then execute them through the primary migration pathway in signac-core.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vyasr]

With the merge of schema2 into next this issue can be closed.