Preprocess: Ability to append to a database

[msilvafe]

We want a way to add onto an existing database i.e. steps 1-30 in the preprocess config file are the same but we want to add step 31 we shouldn’t have to regenerate the whole database. Or say we want to keep steps 1-25, delete 25-30 and do them differently but just do a partial overwrite of an existing db.

[msilvafe]

In discussion with @susannaaz we agreed that a starting point for this could be running 2 preprocess databases. Loading (site_pipeline.preprocess_tod.load_preprocess_tod) from one database/config file and taking the output processed axis-manager and feeding into a new preprocess pipeline (i.e. generating a new database/config file using site_pipeline.preprocess_tod.preprocess_tod)

[msilvafe]

In discussion with @susannaaz we agreed that a starting point for this could be running 2 preprocess databases. Loading (site_pipeline.preprocess_tod.load_preprocess_tod) from one database/config file and taking the output processed axis-manager and feeding into a new preprocess pipeline (i.e. generating a new database/config file using site_pipeline.preprocess_tod.preprocess_tod)

I think we're largely moving this to issue #1003 but its still useful to extend the ability to append/modify some processes to an existing database.