Introduce Frames metadata and CTF

[manasaV3]

Rationale

Given CTF parameters, tilt order information from MDOC-files, along with the already surfaced tilt series alignment parameters and it is possible to compute average particle reconstructions from cropped tilt series stacks, massively reducing the amount of data required to download, and decreasing the time required to evaluate annotations by subtomogram averaging. This will enable more rigorous and automated benchmarking of particle picking algorithms by STA.

Related Tech Spec: https://docs.google.com/document/d/1etrUPRv_kaFAJlwDSCIuyLfxfku7185TB43zhpjXXn0/edit?tab=t.0

Tasks

Tech Spec

• Create a tech spec for updating Frames with ctf metadata #1367

Validation

• Create validations for source data #1527
• Fix existing source data with invalid mdoc or rawtlt
• Update output data validations for frames and ctf #1528
• Run validation for all ingested data

S3 Ingestion Workflow Update

• Update the ingestion configs to support frames and ctf #1054
• Update S3 Ingestion workflow for Frames #1055
• Update S3 Ingestion workflow to support PerSectionParameter #1056
• Update S3 Ingestion workflow for Gains #1516
• Update S3 ingestion for CollectionMetadata #1587
• Run S3 ingestion for all existing datasets

DB Ingestion Workflow

• Update DB ingest for FramesAcquisitionFile #1529
• Update DB ingestion for Frames #1522
• Update DB ingestion for PerSectionParameters table #1523
• Update DB Schema for Gains #1524
• Run DB ingestion for to update all datasets

Miscellaneous

• Python client update for Frames and CTF #1525
• Documentation for Frames and CTF #1526
• Migrate Validate data to production and update the production db