Add datasets to TrackBeam for variance between transits

[tristpinsm]

required by chime-experiment/ch_pipeline#3

[jrs65]

Is there a particular rationale behind having the components as real, imag and cross? Would amplitude and phase have made more sense?

[tristpinsm]

Is there a particular rationale behind having the components as real, imag and cross? Would amplitude and phase have made more sense?

I guess it is more straightforward to calculate the real and imaginary covariances? Maybe @ssiegelx can comment.

[ssiegelx]

Is there a particular rationale behind having the components as real, imag and cross? Would amplitude and phase have made more sense?

I guess the rationale was to provide a complete description of the covariance of the complex visibility in the units/basis that the visibility is saved in the container.

If someone is interested in the amplitude and phase stability of the instrument, then they can use the mean visibility to rotate this covariance matrix to the in-phase/quadrature basis and divide by the visibility amplitude squared. If we assume that the amplitude and phase fluctuations are small, then the resulting diagonal elements would give the variance of the amplitude (in fractional units) and variance of the phase (in radians). If you like, I can add properties to the container that perform this rotation and scaling automatically.

Or are you suggesting that for each observation, we calculate the amplitude and phase, and then calculate the variance of those two quantities? In this case we would need to be a bit careful with phase wrapping.

[ssiegelx]

@jrs65 @tristpinsm hopefully these last two commits address some of the issues that were raised.

It keeps the full complex covariance in the real/imag basis, but add an observed_variance_amp_phase property to rotate it to the amplitude/phase basis.

It also adds an observed_weight property that can be directly interchanged with the weight dataset. The observed_weight uses the largest eigenvalue of the complex covariance matrix as a conservative estimate of the variance.

example_holo_CAS_A_noise.pdf

Here are examples from a stack of ~40 Cas A observations. Shown is the co-polar measurements for a random frequency and input. First panel is amplitude and second panel is phase. The rainbow colors show 10 of the observations, and the black line is the stack of all 40 observations.

Third panel is the error relative to the peak. Rainbow colors are derived from the weight dataset for those 10 days. Black line is derived from the weight dataset of the stack (which is propagated from the individual days). The thick, dashed pink line is derived from the observed variance and corresponds to the component in phase with the visibility (i.e., amplitude direction). The thick, dashed blue line corresponds to the component out of phase with the visibility (i.e., phase direction). Perhaps surprisingly the observed amplitude noise is consistent with our expectation for thermal noise (black line) throughout the track. It does seem to be biased slightly low... will have to look into this. On the other hand, the phase noise exceeds the thermal noise by up to a factor of 3. The cyan, dotted line shows what one would get using the observed_weight. In this case it tracks the phase noise closely.

Note that this particular input is in a different hut from the 26m input. If you flip to the second page you can see the same plots for an input in the same hut as the 26m input. You'll notice that the phase noise is significantly reduced within the primary beam (although still above the thermal noise level). This leads me to believe that we are seeing the short-timescale timing variations, and should probably be applying the timing correction to the holography data for this next round of processing. As you move out into the far sidelobes the phase noise is roughly consistent for these two inputs, which makes me think we are dominated by drift in the phase of the coaxial cables relative to the calibration point at hour angle = 0.

Let me know if you have any suggestions of further improvements. If it looks reasonable to you, then perhaps @pboubel could implement something similar for the chimestack visibility stacking.

[ssiegelx]

example_holo_CAS_A_noise_v2.pdf

Here is an updated plot. I had to divide the noise from the weight dataset by sqrt(2) in order to compare to the amplitude and phase noise, since it is the quadrature sum of the noise from the real and imaginary component. Now it is clear that there is excess noise above thermal in the amplitude direction, particularly when Cas A is in the primary beam.

I now think it makes more sense to set observed_weight to be the trace of the covariance matrix for interchangeability with the weight dataset.

[tristpinsm]

@ssiegelx Thanks for putting this together, those plots are very interesting. I agree it would be good to apply the timing correction at some point in the holography pipeline. Taking a quick look at the existing task for this, it seems it might not need any modification to work on the 26m archive files directly (i.e. in the initial fringestopping/regridding pipeline). Do you think that's correct?

I'm also wondering if any of the existing RFI flagging tasks could be run on the holography transits prior to stacking. While testing Alex found some transits that had RFI-like features which propagated into the stack.

[ssiegelx]

Yeah that's right, it should work on the 26m archive files in the initial fringestopping and regrinding pipeline. The config file would look something like this:

...

    - type: draco.core.io.LoadFilesFromParams
      out: tcorr
      params:
        files: "/project/rpp-krs/chime/chime_processed/timing/rev_00/not_referenced/*_chimetiming_delay.h5"
        distributed: No

    - type: draco.core.misc.AccumulateList
      in: tcorr
      out: tcorrlist

    - type: draco.core.misc.WaitUntil
      requires: tcorrlist
      in: filelist
      out: filelist2

    - type: ch_pipeline.core.io.LoadCorrDataFiles
      requires: filelist2
      out: tstream

    - type: ch_pipeline.analysis.timing.ApplyTimingCorrection
      requires: tcorrlist
      in: tstream
      out: tstream_corr

...

I don't think it would work on the TrackBeam containers at the moment. There would need to be additional modification to the ApplyTimingCorrection task to handle the polarization axis and generate an equivalent prod index map.

[tristpinsm]

With a small tweak to ch_util those additions to the config are at least running without crashing. I can add this to the standard fringestopping pipeline from now on. We should just coordinate when revision 2 gets going where to get the timing files from.