Added delay to trigger after block end and to prefer public/same block/same proposal calibrations. #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…k/same proposal calibrations.
mgdaily
approved these changes
Dec 11, 2024
sfoale
reviewed
Dec 12, 2024
jchate6
approved these changes
Dec 16, 2024
…ompatible with direct date differences.
…e.utc for compatability with python <=3.10
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds functionality to choose which calibration frames are chosen and when data is processed. This is mostly for BANZAI-FLOYDS. I don't expect BANZAI imaging to use this functionality.
- Delaying processing until after the block is complete.
I added functionality that would delay processing until after the block is done observing. This helps for FLOYDS when you want to use an arc and flat taken along with the current data without waiting for the full end of the night.
This is implemented by adding retry logic into the realtime utils, need_to_be_processed. In the case that the block end date is later than now, we use a celery task retry with a delay of 5 minutes after the end of the block. This will only affect certain observation types as set in the settings and has to be manually enabled from the command line.
- Prefering same block or same proposal calibrations
For imaging, all of the calibrations are taken under a public/observatory level program so we haven't had to worry about this. For FLOYDS it is common for the user to take their own calibrations. We then don't want other observations to use that calibration (unless it's public, see below). I have added in the db queries a chain of logic to first check for observations in the same block as the frame that is being processed, then the same proposal, then everything else. The knock on of this is that we now need to store the proposal and block id in the database. The database migration for this is not very complicated but will take a while to run at this point because we have so much data.
- Only use public calibrations
If calibrations from the same block or same proposal don't exist we want to fall back to public data, i.e. we don't want to process data with someone else's proprietary data. To enable this, we need to include the L1PUBDAT from the data in the database. This database migration can be easily completed with the one above.