Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Can't load_remote_dataset in suite.py for OPeNDAP URL #500

Closed
mwengren opened this issue May 18, 2017 · 15 comments
Closed

Can't load_remote_dataset in suite.py for OPeNDAP URL #500

mwengren opened this issue May 18, 2017 · 15 comments

Comments

@mwengren
Copy link
Member

I recently updated my Compliance Checker to 3.0.3 and am unable to run tests against OPeNDAP URLs from PacIOOS.

Here's a few examples:

compliance-checker -t cf -f json http://oos.soest.hawaii.edu/thredds/dodsC/hioos/model/atm/ncep_pac/NCEP_Pacific_Atmospheric_Model_best.ncd
compliance-checker -t cf -f json http://oos.soest.hawaii.edu/thredds/dodsC/usgs_dem_10m_tinian

Output is similar:

Traceback (most recent call last):
  File "C:\Users\Micah.Wengren\AppData\Local\Continuum\Miniconda2\envs\IOOS_27\Scripts\compliance-checker-script.py", line 5, in <module>
    sys.exit(cchecker.main())
  File "C:\Users\Micah.Wengren\AppData\Local\Continuum\Miniconda2\envs\IOOS_27\Scripts\cchecker.py", line 73, in main
    args.format)
  File "C:\Users\Micah.Wengren\AppData\Local\Continuum\Miniconda2\envs\IOOS_27\lib\site-packages\compliance_checker\runner.py", line 50, in run_checker
    ds = cs.load_dataset(ds_loc)
  File "C:\Users\Micah.Wengren\AppData\Local\Continuum\Miniconda2\envs\IOOS_27\lib\site-packages\compliance_checker\suite.py", line 527, in load_dataset
    return self.load_remote_dataset(ds_str)
  File "C:\Users\Micah.Wengren\AppData\Local\Continuum\Miniconda2\envs\IOOS_27\lib\site-packages\compliance_checker\suite.py", line 537, in load_remote_dataset
    return Dataset(ds_str)
  File "netCDF4\_netCDF4.pyx", line 1811, in netCDF4._netCDF4.Dataset.__init__ (netCDF4\_netCDF4.c:13224)
IOError: NetCDF: Invalid argument

Apologies is this is user error or something I'm missing about running this on Windows (Python 2.7), but it does not seem able to read these OPeNDAP URLs. It is successful in the same environment with several NANOOS OPeNDAP services.

compliance-checker -t cf -f json http://ona.coas.oregonstate.edu:8080/thredds/dodsC/NANOOS/OCOS
@lukecampbell
Copy link
Member

I was able to execute all the commands successfully on MacOS, maybe this is isolated to windows? Does anyone else have a windows machine they can confirm on?

@daf
Copy link
Member

daf commented May 18, 2017

I am able to reproduce on my Windows 10 laptop.

@daf
Copy link
Member

daf commented May 18, 2017

At this stage, it's just a passthrough to netcdf's Dataset() initializer - it haves the same way in ipython:

In [4]: Dataset("http://oos.soest.hawaii.edu/thredds/dodsC/hioos/model/atm/ncep
   ...: _pac/NCEP_Pacific_Atmospheric_Model_best.ncd")
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-4-0da87aab71ef> in <module>()
----> 1 Dataset("http://oos.soest.hawaii.edu/thredds/dodsC/hioos/model/atm/ncep_pac/NCEP_Pacific_Atmospheric_Model_best.ncd")

netCDF4\_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__ (netCDF4\_netCDF4.c:13985)()

IOError: NetCDF: Invalid argument

So it seems like an upstream issue.

(ioos27) C:\Users\daf>conda list netCDF
# packages in environment at C:\Users\daf\Anaconda2\envs\ioos27:
#
libnetcdf                 4.4.1.1                   vc9_4  [vc9]  conda-forge
netcdf4                   1.2.7               np111py27_0    conda-forge

Should we file with unidata?

@ocefpaf
Copy link
Member

ocefpaf commented May 18, 2017 via email

@ocefpaf
Copy link
Member

ocefpaf commented May 18, 2017 via email

@daf
Copy link
Member

daf commented May 18, 2017

@ocefpaf downgrading to libnetcdf==4.4.0 works!

@mwengren
Copy link
Member Author

Yep, libnetcdf=4.4.0 did the trick! Good for the moment for me, thanks.

@ocefpaf
Copy link
Member

ocefpaf commented May 21, 2017

I believe we can close this because it is not a compliance-checker issue.
PR Unidata/netcdf-c#366 should have fixed this, but I did applied that as a patch in the latest conda-forge package. I will investigate if the patch is bad or if the issue is not really resolved upstream.

@ocefpaf ocefpaf closed this as completed May 21, 2017
@mwengren
Copy link
Member Author

@ocefpaf @daf So, for now if we stick with libnetcdf=4.4.0 as a workaround this is fixed, and if there is an issue with the conda-forge package you will resolve @ocefpaf and then at that point we can do a libnetcdf update to the latest release again? And if not, we'll wait for the upstream issue fix to occur?

I'm also running into problems related to #497, so I'm wondering if a bug fix release of compliance-checker would make sense, even if the above isn't resolved (since they are unconnected).

Since the prior release was only a few weeks ago and there have just been a few fixes since then, maybe not, but the cf debug output definitely is cramping my style. I've noticed it occurs for some URLs, but not all (which makes it impossible to workaround programmatically):

Mixing issue content here, sorry...

 compliance-checker -t cf -f json http://oos.soest.hawaii.edu/thredds/dodsC/pibhmc_bathy_40m_palmyra

(outputs cf table debug message)

 compliance-checker -t cf -f json http://hfrnet.ucsd.edu/thredds/dodsC/HFR/USHI/1km/hourly/RTV/HFRADAR,_US_Hawaii,_1km_Resolution,_Hourly_RTV_best.ncd

(does not output cf table debug message)

@ocefpaf
Copy link
Member

ocefpaf commented May 22, 2017

issue with the conda-forge package you will resolve @ocefpaf

I don't think the issue is with the package in conda-forge. I got my hands on a Windows machine today and I test libnetcdf 4.4.1.1 vc14_4 [vc14] conda-forge, on Python 3.6, and everything worked.

I can confirm that the vc9_4 version is not working though. Not sure why... I can see that the patch was applied and the test passed at build time, so I am clueless at this time. Unfortunately upstream does not have continuous integration tests for Windows so this is really hard to track.

PS: I am 👍 to a bugfix release. (Release early, release often!) Also, the issue here is unrelated to compliance-checker and there is nothing we can do besides workaround it. The only solution so far is to pin to libnetcdf 4.4.0 on Python 2.7.

@ocefpaf
Copy link
Member

ocefpaf commented May 22, 2017

FYI I have a PR to add continuous integration to the unidata netcdf-c repo in Unidata/netcdf-c#347

I added @mwengren two URLs to the test there, but you can see that the latest source does not even build on Python 2.7 (vc9) and 3.4 (vc10). Here are the results: https://ci.appveyor.com/project/ocefpaf/netcdf-c/build/1.0.23

@mwengren
Copy link
Member Author

mwengren commented May 23, 2017 •
edited
Loading

@ocefpaf @daf @jbosch-noaa @kknee who can/should handle a Compliance Checker release at this point? Is everything resolved from the PRs/commits since the last release?

@daf
Copy link
Member

daf commented May 23, 2017

I'm happy to do it, will get to it in an hour or so.

@ocefpaf
Copy link
Member

ocefpaf commented May 23, 2017 via email

@daf
Copy link
Member

daf commented May 23, 2017

@mwengren released 3.0.4 to pypi. @ocefpaf got started on a bump for conda-forge, conda-forge/compliance-checker-feedstock#18

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@daf @ocefpaf @lukecampbell @mwengren