Update S3 Ingestion workflow to add Alignment relationships

[manasaV3]

Motivation

As we will be supporting multiple tomograms for a run, this requires restructuring how we store data, and add additional relationships to them.

Definition of Done

Any new ingestion follows the new spec, such as but not limited to:

1. The tomograms, and annotations have a relationship to the alignment they use.
2. The tomograms and relevant metadata are renamed with identifiers
3. The annotations and relevant metadata are renamed to the new standard
4. The key photo generated include an identifier
5. The neuroglancer config is only generated for the tomogram with viz_default set to true

Tasks

Update the existing ingestion workflow to support capturing the new relevant metadata and updating the data structure as required.

The changes required for the entities are as follows:

KeyPhoto Generation

• Keyphoto generation is moved to <Dataset_Id>/<run_name>/Reconstructions/VoxelSpacing<vs_spacing>/Images folders from the prior KeyPhotos folder.
• The key photo name include the tomogram_id in the name.

Tomogram

• Update how the tomogram and related metadata files are named.
• The tomograms are to be saved in <Dataset_Id>/<run_name>/Reconstructions/VoxelSpacing<vs_spacing>/Tomograms.
• The tomograms will now have a unique identifier that is named by creating for a unique combination of: deposition_id, alignment_path, reconstruction method, post processing
• Capture the relationship between alignment and tomogram in metadata

Annotations

• Capture the relationship between alignment and annotation.
• Update how the annotation files and related metadata files are named. The unique identifiers is created for a unique combination of: deposition id, alignment path (new), description, object name, state (new), annotation method
• The annotations are to be saved in <Dataset_Id>/<run_name>/Reconstructions/VoxelSpacing<vs_spacing>/Annotation.
• Handle for annotations for standardized configuration, by updating their naming and adding is_standardized in the files object of the metadata.

Neuroglancer Config

• Only generate the neuroglancer config for the tomograms that have visualization default set to true
• Move neuroglancer_config.json to NeuroglancerPrecompute. <Dataset_Id>/<run_name>/Reconstructions/VoxelSpacing<vs_spacing>/NeuroglancerPrecompute.
• update the naming for the neuroglancer_config to include the tomogram_id

Notes

It is possible in some cases, that the annotations in neuroglancer config might not include the annotations in a different voxel spacing