NetCDF's use of HDF5 HL DS APIs and parallel I/O performance. #1822
Description
I noticed that NetCDF calls the High-level DS APIs by all the processes doing I/O. For example, it has sync_netcdf5_file --> nc4_rec_write_metadata --> write_var --> (H5DSdetach_scale, H5DSattach_scale, etc...).
Has there been an investigation into the possibility of closing the file and reopening it with only one process and then doing all the metadata updating (.i.e. the DS calls)?
All the HL DS APIs are "serial," so all the processes are re-writing the same data, and this data also seems to be very small. Usually, doing many little writes results in poor parallel performance on a parallel file system, especially when you have tens of thousands of processes writing what amounts to the same data.
The HL APIs are not meant to be called by all the processes since this, for the most part, just duplicates the I/O. Also, HDF5 does not test any of the HL APIs in a parallel setting.
I could be missing something subtle that doesn't allow NetCDF to do this, so any insight/feedback would be much appreciated.