mirror MdV beams?

[o-smirnov]

The archive keeps playing silly buggers with us. @sjperkins could we set up a mirror of the files from here on our OpenData S3 bucket? Then I could change the defult paths (https://github.com/ratt-ru/suricat-beams/blob/master/suricat-beams/suricat/beams.py#L38) to download from S3 instead.

[IanHeywood]

I think @SpaceMeerkat already set one up:

In the meantime an S3 publicly accessible endpoint for these is now here (once the copy finishes): https://ratt-public-data.s3.af-south-1.amazonaws.com/MeerKATbeams/

(Although the link doesn't go anywhere for me).

[sjperkins]

Indeed

$ aws s3 ls  --profile ratt-public-data --region af-south-1 s3://ratt-public-data/MeerKATbeams/
                           PRE beam-L.bds.zarr/
                           PRE beam-U.bds.zarr/
2024-11-15 15:21:55          0 
2024-11-15 15:32:55         81 .persistent_history
2024-11-15 15:34:17 34896620743 mdv-beams-L.npz
2024-11-15 15:34:17  235929600 mdv-beams-L.npzo9zca6jp.tmp
2024-11-15 15:34:17  477102080 mdv-beams-L.npzz4zroyyt.tmp
2024-11-15 15:34:17 34896620743 mdv-beams-U.npz

The http url for beams-L would be: https://ratt-public-data.s3.af-south-1.amazonaws.com/MeerKATbeams/mdv-beams-L.npz

[o-smirnov]

I can get L and U band beams via that link, but not S0 or S4. @sjperkins @SpaceMeerkat could you upload those as well please? And maybe rename everything back to match the filenames in https://archive-gw-1.kat.ac.za/public/repository/10.48479/wdb0-h061/index.html? I'll change the default paths in suricat to point to AWS then -- it just become awkward when the files themselves are named differently depending on mirror.

[sjperkins]

I can get L and U band beams via that link, but not S0 or S4. @sjperkins @SpaceMeerkat could you upload those as well please? And maybe rename everything back to match the filenames in https://archive-gw-1.kat.ac.za/public/repository/10.48479/wdb0-h061/index.html? I'll change the default paths in suricat to point to AWS then -- it just become awkward when the files themselves are named differently depending on mirror.

OK, busy working with the office server permissions (better to do it from the chpc I guess). Actually maybe a cheap EC2 instance is better.

[sjperkins]

Both the data and metric npz's have been uploaded. The MeerKATbeams folder now looks as follow:

$  aws s3 ls  --profile ratt-public-data --region af-south-1 s3://ratt-public-data/MeerKATbeams/
2024-11-15 15:21:55          0 
2025-01-21 15:34:41 34896620743 MeerKAT_L_band_primary_beam.npz.tar.gz
2025-01-21 15:35:15   28235140 MeerKAT_L_band_primary_beam_metrics.npz.tar.gz
2025-01-21 15:35:15 34896620743 MeerKAT_S0_band_primary_beam.npz.tar.gz
2025-01-21 15:35:15   28235140 MeerKAT_S0_band_primary_beam_metrics.npz.tar.gz
2025-01-21 15:35:15 34896620743 MeerKAT_S4_band_primary_beam.npz.tar.gz
2025-01-21 15:35:16   28235140 MeerKAT_S4_band_primary_beam_metrics.npz.tar.gz
2025-01-21 15:35:16 34896620743 MeerKAT_U_band_primary_beam.npz.tar.gz
2025-01-21 15:35:16   28235140 MeerKAT_U_band_primary_beam_metrics.npz.tar.gz

So the http url prefix that should be prepended to each file is https://ratt-public-data.s3.af-south-1.amazonaws.com/MeerKATbeams