netCDF4 conflicts with h5py on Windows

[akeeman]

The two libraries are not compatible with each other, as one cannot import them both within the same runtime.
I don't know whether h5py or netCDF4 is causing the problem. (same issue in h5py repo)

win10 + python 3.8.5 fails
win11 + python 3.8.10 fails
linux + python 3.8.15 works

Setup details

C:\path\to>python -m venv .venv

C:\path\to>.venv\Scripts\pip install h5py netCDF4
Collecting h5py
  Using cached h5py-3.7.0-cp38-cp38-win_amd64.whl (2.6 MB)
Collecting netCDF4
  Using cached netCDF4-1.6.1-cp38-cp38-win_amd64.whl (5.2 MB)
Collecting numpy>=1.14.5
  Using cached numpy-1.23.4-cp38-cp38-win_amd64.whl (14.7 MB)
Collecting cftime
  Using cached cftime-1.6.2-cp38-cp38-win_amd64.whl (171 kB)
Installing collected packages: numpy, h5py, cftime, netCDF4
Successfully installed cftime-1.6.2 h5py-3.7.0 netCDF4-1.6.1 numpy-1.23.4
WARNING: You are using pip version 20.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'c:\path\to\.venv\scripts\python.exe -m pip install --upgrade pip' command.

C:\path\to>.venv\Scripts\activate

(.venv) C:\path\to>

Import netCDF4, then h5py -> exits python

(.venv) C:\path\to>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
>>> import h5py
C:\path\to\.venv\lib\site-packages\h5py\__init__.py:36: UserWarning: h5py is running against HDF5 1.12.1 when it was built against 1.12.2, this may cause problems
  _warn(("h5py is running against HDF5 {0} when it was built against {1}, "
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.12.2, library is 1.12.1
        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.12.1
                  Configured on: 2022-03-04
                  Configured by: Ninja
                    Host system: Windows-10.0.17763
              Uname information: Windows
                       Byte sex: little-endian
             Installation point: D:/bld/hdf5_split_1646412547396/_h_env/Library

Compiling Options:
------------------
                     Build Mode: RELEASE
              Debugging Symbols: OFF
                        Asserts: OFF
                      Profiling: OFF
             Optimization Level: OFF

Linking Options:
----------------
                      Libraries:
  Statically Linked Executables: OFF
                        LDFLAGS: /machine:x64
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: D:/bld/hdf5_split_1646412547396/_h_env/Library/lib/libcurl.lib;D:/bld/hdf5_split_1646412547396/_h_env/Library/lib/libssl.lib;D:/bld/hdf5_split_1646412547396/_h_env/Library/lib/libcrypto.lib
                       Archiver: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.16.27023/bin/HostX64/x64/lib.exe
                         Ranlib: :

Languages:
----------
                              C: YES
                     C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.16.27023/bin/HostX64/x64/cl.exe 19.16.27045.0
                       CPPFLAGS:
                    H5_CPPFLAGS:
                    AM_CPPFLAGS:
                         CFLAGS:   /DWIN32 /D_WINDOWS
                      H5_CFLAGS: /W3;/wd4100;/wd4706;/wd4127
                      AM_CFLAGS:
               Shared C Library: YES
               Static C Library: YES

                        Fortran: OFF
               Fortran Compiler:
                  Fortran Flags:
               H5 Fortran Flags:
               AM Fortran Flags:
         Shared Fortran Library: YES
         Static Fortran Library: YES

                            C++: ON
                   C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.16.27023/bin/HostX64/x64/cl.exe 19.16.27045.0
                      C++ Flags:
                   H5 C++ Flags: /W3;/wd4100;/wd4706;/wd4127
                   AM C++ Flags:
             Shared C++ Library: YES
             Static C++ Library: YES

                            JAVA: OFF
                 JAVA Compiler:

Features:
---------
                   Parallel HDF5: OFF
Parallel Filtered Dataset Writes:
              Large Parallel I/O:
              High-level library: ON
                Build HDF5 Tests: ON
                Build HDF5 Tools: ON
                    Threadsafety: ON (recursive RW locks: )
             Default API mapping: v112
  With deprecated public symbols: ON
          I/O filters (external):  DEFLATE
                             MPE:
                      Direct VFD:
                      Mirror VFD:
              (Read-Only) S3 VFD: 1
            (Read-Only) HDFS VFD:
                         dmalloc:
  Packages w/ extra debug output:
                     API Tracing: OFF
            Using memory checker: OFF
 Memory allocation sanity checks: OFF
          Function Stack Tracing: OFF
                Use file locking: best-effort
       Strict File Format Checks: OFF
    Optimization Instrumentation:
Bye...

Import h5py, then netCDF4 -> raises ImportError

(.venv) C:\path\to>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> import netCDF4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\path\to\.venv\lib\site-packages\netCDF4\__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: DLL load failed while importing _netCDF4: The specified procedure could not be found.

[ocefpaf]

I'm not an expert on wheels but I'm pretty sure that is b/c we have different HDF5 builds for the h5py and the netCDF4 ones. @isuruf, is there something we can do to mitigate that?

For know I would recommend you to either build both from source, or to keep them in separate envs.

[akeeman]

Thanks for you recommendations. Can't keep the min separate envs, as they're needed in the same runtime. However, I could look into fording to build them from source.

[jswhit]

I'm getting ready to release new wheels for 1.6.2, so I could build with the same HDF5 version as h5py. What version is h5py using? Is it just a version issue, or is there something else incompatible in the build?

[jswhit]

AFAICT, hdf5 1.12.2 is used for all builds - so it must be more than just a version issue.

[isuruf]

@ocefpaf, the delvewheel relocated wheels shouldn't have this issue

[ocefpaf]

@ocefpaf, the delvewheel relocated wheels shouldn't have this issue

Good to know. We started using delvewheel in ocefpaf/netcdf4-win-wheels#4. If I did it correctly the 1.6.2 should be OK.

@akeeman can you try a manual wheel installation from the artifacts in https://github.com/ocefpaf/netcdf4-win-wheels/releases/tag/v1.6.2? You may need to download the right Python version, unzip, and pip install it. If those work the next versions should be OK too.

[jswhit]

1.6.2 is up on pypi now @akeeman, can you try it and see if the conflict is resolved?

[akeeman]

Works like a charm. Thanks!
So take home message is that I should pick versions where their hdf5 versions align.

[isuruf]

So take home message is that I should pick versions where their hdf5 versions align.

No, that's not it. 1.6.1 had wheels without proper renaming of DLLs. 1.6.2 wheels renamed the DLLs prpoerly.

[ocefpaf]

No, that's not it. 1.6.1 had wheels without proper renaming of DLLs. 1.6.2 wheels renamed the DLLs properly.

Indeed. We started using delveheels only for the second one. It should be ok for >=1.6.2 now.

@jswhit you may want to yank the 1.6.1. I can create a post release if necessary but I believe that the upgrade to 1.6.2 is better.

[jswhit]

the windows wheels for 1.6.1 have been deleted from pypi

[tacaswell]

Can you point me to where the wheel name mangling is done? If my understanding is correct this was a problem because neither h5py nor netcdf4 were doing the mangling?

[ocefpaf]

Can you point me to where the wheel name mangling is done? If my understanding is correct this was a problem because neither h5py nor netcdf4 were doing the mangling?

We are using delvewheel now and Windows wheels >1.6.1 do not seem to have this issue. Before we had a custom script. Maybe this diff has all you need: https://github.com/ocefpaf/netcdf4-win-wheels/pull/4/files#diff-54cf74e113dd3f6d11e092fdb1d888ec82c69bdafbb15cfb6570c83ecad28f33L14-L18