Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Handle out-of-order ERA5 time coord issue #189

Merged

Conversation

leifdenby
Copy link
Contributor

It appears that ECMWF's CDS has started returning data where the time-coordinate isn't monotonic anymore and the timesteps are randomly ordered. This commit adds sorting by time during load-time. This may slow down calculations and a fix where input files are overwritten could be made in future

Related to #183

It appears that ECMWF's CDS has started returning data where the
time-coordinate isn't monotonic anymore and the timesteps are randomly
ordered. This commit adds sorting by time during load-time. This may
slow down calculations and a fix where input files are overwritten could
be made in future

Related to EUREC4A-UK#183
@leifdenby leifdenby requested a review from sjboeing December 8, 2022 14:35
@leifdenby
Copy link
Contributor Author

@sjboeing I've manually verified that this works. We can't test it with our test-data because we don't download data from ECMWF during testing. Could you give it a try too and let me know what you think? Thanks :)

@sjboeing
Copy link
Contributor

Hi Leif,

This looks good and it seems to work for me. While working on this, I did run into an issue with domain versioning:
"Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download."
To make it work, I removed the version from:
lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml
Let me know if this sounds familiar, if not I will make an issue for us to investigate.

@leifdenby
Copy link
Contributor Author

"Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download."
To make it work, I removed the version from:
lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml

Hmm.. I think this means you downloaded this domain data before we added the versioning (#87). But this is fine :) Another way to solve it is just create a filename called VERSION in domains/eurec4a_circle_data which just contains 1.0.0

@leifdenby
Copy link
Contributor Author

I'll merge this in once the tests pass

@sjboeing with this change the changelog is getting quite long: https://github.com/leifdenby/lagtraj/blob/fix-era5-time-monotinicity-issue/CHANGELOG.md - how do you feel about me creating release v0.1.2 once this pull-request has been merged?

@leifdenby leifdenby merged commit 20bb79c into EUREC4A-UK:master Dec 12, 2022
@sjboeing
Copy link
Contributor

Ah, thanks Leif. merging this and creating a new release sounds good.

The other issue occurred with data downloaded just this week. I will check and see if it is something I missed and create an issue if it persists.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants