Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Log level for data read is too low #343

Open
spassmoor opened this issue Jul 29, 2021 · 1 comment
Open

Log level for data read is too low #343

spassmoor opened this issue Jul 29, 2021 · 1 comment
Assignees

Comments

@spassmoor
Copy link
Contributor

When reading data that is not found in the archive the log level is set to debug and the missing data is filled in with zeros. This should be a warning level output
example
DEBUG: Starting new HTTPS connection (1): archive-gw-1.kat.ac.za:443
DEBUG: https://archive-gw-1.kat.ac.za:443 "GET /1537611365_sdp_l0/correlator_data/00004_01920_00000.npy HTTP/1.1" 404 230

@ludwigschwardt
Copy link
Contributor

ludwigschwardt commented Jul 29, 2021

First off, it's better to make SPR1 JIRA tickets if you are within SARAO to avoid falling off the radar :-D

Now, on missing data, there are three cases I am interested in:

  • The data was never in the archive. This happens at the start and end of nearly all observations and should not turn into a warning.
  • The data is in the archive but we struggled to retrieve it. This could become a warning, but I'm not keen on a sea of logs.
  • No data is in the archive, because the dataset has moved to tape. This needs an entirely different error message (SPR1-766).

There is a ticket (SPR1-891) to collect all the statistics of archive errors and present that at the end. Or do you want some kind of running commentary?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants