Implement DQ support for HST

[bmorris3]

This PR adds DQ plugin support for L2 products from four HST instruments, with data quality flag mappings from these HST instrument docs pages (retrieved today):

• STIS: https://hst-docs.stsci.edu/stisdhb/chapter-2-stis-data-structure/2-5-error-and-data-quality-array
• ACS: https://www.stsci.edu/hst/instrumentation/acs/data-analysis/dq-flag-definitions
• WFC3: https://hst-docs.stsci.edu/wfc3dhb/chapter-2-wfc3-data-structure/2-2-wfc3-file-structure
• COS: https://hst-docs.stsci.edu/cosdhb/chapter-2-cos-data-files/2-7-error-and-data-quality-arrays

Example

Now using a MAST URI for an HST L2 product will work in the concept notebook imviz_dq_concept.ipynb. Two example HST products are:

    'mast:HST/product/hst_17183_02_wfc3_uvis_g280_iexr02mt_flt.fits', # HST/WFC3 (UVIS)
    'mast:HST/product/hst_16968_01_acs_wfc_f606w_jezz01l3_flt.fits' # HST/ACS

Here's the demo for the WFC3 observations:

imviz-dq-hst-wfc3.mov

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)? 🐱

[cshanahan1]

I'm a bit confused about what changing the dq mapping in the dropdown actually triggers to change.

For example, looking at mast:HST/product/hst_17183_02_wfc3_uvis_g280_iexr02mt_flt.fits, when i toggle between UVIS (the correct dq mapping) and IR (not correct, but apparently able to toggle to), the 32 bit flag retains the UVIS definition (CTE tail).

[bmorris3]

Thanks for catching that @cshanahan1! I simply hadn't tried to select a different mapping because the auto-detect has worked each time, but on changing the flag map, the definitions should change in the DQ list. I made a small update in bf1105f which enforces that change.

[gibsongreen]

This I think should be a separate ticket from this PR if it is applicable, when I am using DQ in Imviz, I must load_data() with the the SCI and DQ extension, parent child relationship for DQ to load. If I exclude this, we get the SCI data.

For Cubeviz, I think this should also be true, but the parser? will automatically load the DQ without the ext in load_data() being set (and will understandably throw an error).

So the separate ticket I will create if we agree would be to have consistency in the loading logic for DQ across plugins.

[cshanahan1]

in terms of the labels applied to each flag, i see that they each get labeled with their bit value but there is a second number which appears to be their index in the list of all flags

it appears that flag 8 (which would be labeled with a 4) is being skipped in both UVIS and IR. Hot pixel gets the label '4'

[bmorris3]

@gibsongreen:

This I think should be a separate ticket from this PR if it is applicable, when I am using DQ in Imviz, I must load_data() with the the SCI and DQ extension, parent child relationship for DQ to load. If I exclude this, we get the SCI data.

Correct – Imviz loads only the science/data extension by default. If you want all, you can list them explicitly like ext=('SCI', 'DQ') or ext='*' for all.

For Cubeviz, I think this should also be true, but the parser? will automatically load the DQ without the ext in load_data() being set (and will understandably throw an error).

Correct – Cubeviz always loads the science/data, error, and DQ extensions.

So the separate ticket I will create if we agree would be to have consistency in the loading logic for DQ across plugins.

Here's why it was designed to be intentionally inconsistent for these configs:

• Cubeviz used to have a flux viewer, a mask(/DQ) viewer, and an uncertainty viewer in the top row of image viewers. Loading the extension for each viewer was the default behavior for Cubeviz, so that each viewer could be populated. You could (and maybe should!) argue that we don't need DQ or ERR by default – I haven't seen a single use-case where ERR is useful to see with your eyes, BUT we do use the ERR array when we propagate the uncertainties during Cubeviz spectral extraction. And until the DQ plugin was introduced, there was no visualization support for the DQ extension, BUT we have a use for it now. So I agree that revisiting these defaults is reasonable, but I would argue they're more appropriate now than before.
• In Imviz on occasion, you may have a single image that is enormous, or a file with loads of extensions. Loading them all would be expensive in memory and file IO.

[bmorris3]

@cshanahan1:

I’m not sure I follow:

it appears that flag 8 (which would be labeled with a 4) is being skipped in both UVIS and IR. Hot pixel gets the label ‘4’

We can check that the DQ array for the WFC3 observation contains these values:

from astropy.nddata import NDDataArray
import numpy as np

dq = imviz.app.data_collection[1]
np.unique(dq.get_object(NDDataArray).data).astype(int)

which returns

[   4,   16,   20,   32,   48,  256,  272,  288,  304, 1024, 1028,
   1040, 1044, 1056, 1072, 1280, 4096, 4100, 4112, 4116, 4128, 4144,
   4352, 5120, 5124, 5136, 5140]

these are the bold values in the plugin:

The values in the parentheses are the flags decomposed into powers of two. So referring to the WFC3 flag mapping docs, if you have a DQ value of 4, that corresponds to "bad detector pixel," and in parentheses we have log2(4) = 2.

[cshanahan1]

i take back my comment! that makes perfect sense.

[cshanahan1]

Looks good to me, the only issue I found was the non-updating issue which you fixed

[gibsongreen]

This was the only case that I found that could be an issue (where B1 in the plugin is displayed as the child of A [excluding this example, the opacity works as expected and does get applied to B1 when you reselect B]), but it isn't easily reproducible. I've tried ~10 times this morning and only could force it to happen again once.

edge_case.mov

I feel like this has no meaning, but is there ever a case where a reverse bit filter would be desirable? I can make sure all the 2s are on, is there any utility in making sure all the 2s are off?

The turn around from ticket -> PR is awesome and super happy that this got in for 3.10!

[bmorris3]

Thanks @gibsongreen and @cshanahan1!

RE @gibsongreen:

I feel like this has no meaning, but is there ever a case where a reverse bit filter would be desirable? I can make sure all the 2s are on, is there any utility in making sure all the 2s are off?

It's possible this would be useful, and it's possible that it's easy to implement. But let's wait for someone to ask for it.