-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
NCT upload script #1
Conversation
Needs exeception handling and testing
Added logging to report progress
Some refactoring
Updated README
Clean up upload in case of exception
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sieht gut aus, aber ein local test mittel localstack, den man dann auch im CI laufen lassen kann, wäre schon gut.
Bezüglich dem cleanup bei Fehlern. Es wäre in der Tat ganz gut, wenn die bisher hochgeladenen Files dann gelöscht werden. Sonst haben wir ganz viele Dateileichen, die wir später nichtmer zuordenen können. Kann ja teil der Exception handling logic werden.
Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Found a typo
Regarding what Kersten mentioned: - We should always use S3-Testcontainers when using S3-Capabilities
- I feel like always using pycurl-requests for consistency, but we can debate this
In total, I think the script could be refactored for readability before we give it to a wider audience - as long as we only use it for the nct master dataset, we should be fine.
Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com>
Happy path test with output cleanup check
Now back to global SESSION object
Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com>
Do we need a test against a localstack Testcontainer for this?
Retries are handled through a request session object.
Multipart uploads are cleaned up, but if something happens during/after dowanload, object stays in storage.
Do we need to clean this up as well?