adios driver use BP5

[dqwu]

Move from BP3 to BP5 for adios driver to write.
BP3 read capability preserved for reading decomposition file.
With required metadata to use BP5 conversion tool of SCORPIO.

[khou2020]

Move from BP3 to BP5 for adios driver to write. BP3 read capability preserved for reading decomposition file. With required metadata to use BP5 conversion tool of SCORPIO.

The output file cannot be read by adios's bpls. Is it normal?

$ ~/.local/adios2/2.8.1/bin/bpls ~/FS_TEST/e3sm/adios/bp5/f_h0.bp
int64_t /pio/decomp/512
bpls caught an exception
[Thu Oct 27 01:49:38 2022] [ADIOS2 EXCEPTION] : Engine BP5Reader does not support DoAllStepsBlocksInfo

[dqwu]

Move from BP3 to BP5 for adios driver to write. BP3 read capability preserved for reading decomposition file. With required metadata to use BP5 conversion tool of SCORPIO.

The output file cannot be read by adios's bpls. Is it normal?

$ ~/.local/adios2/2.8.1/bin/bpls ~/FS_TEST/e3sm/adios/bp5/f_h0.bp int64_t /pio/decomp/512 bpls caught an exception [Thu Oct 27 01:49:38 2022] [ADIOS2 EXCEPTION] : Engine BP5Reader does not support DoAllStepsBlocksInfo

bpls works fine for me. I built ADIOS 2.8.3 with CMake settings like below.
Could you please try these settings?

cd /path/to/ADIOS2-2.8.3-source

mkdir build
cd build

CC=mpicc CXX=mpicxx FC=mpif90 \
CFLAGS="-g -O2" CPPFLAGS="-g -O2" CXXFLAGS="-g -O2" FCFLAGS="-g -O2" \
cmake \
-DCMAKE_INSTALL_PREFIX=/path/to/ADIOS2-2.8.3-installation \
-DBUILD_SHARED_LIBS=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DBUILD_TESTING=OFF \
-DADIOS2_USE_Blosc=OFF \
-DADIOS2_USE_BZip2=OFF \
-DADIOS2_USE_ZFP=OFF \
-DADIOS2_USE_SZ=OFF \
-DADIOS2_USE_MGARD=OFF \
-DADIOS2_USE_PNG=OFF \
-DADIOS2_USE_DataMan=OFF \
-DADIOS2_USE_DataSpaces=OFF \
-DADIOS2_USE_MHS=OFF \
-DADIOS2_USE_ZeroMQ=OFF \
-DADIOS2_USE_HDF5=OFF \
-DADIOS2_USE_Python=OFF \
-DADIOS2_USE_Fortran=OFF \
-DADIOS2_USE_Profiling=OFF \
..

make -j4
make install

[dqwu]

Move from BP3 to BP5 for adios driver to write. BP3 read capability preserved for reading decomposition file. With required metadata to use BP5 conversion tool of SCORPIO.

The output file cannot be read by adios's bpls. Is it normal?

$ ~/.local/adios2/2.8.1/bin/bpls ~/FS_TEST/e3sm/adios/bp5/f_h0.bp int64_t /pio/decomp/512 bpls caught an exception [Thu Oct 27 01:49:38 2022] [ADIOS2 EXCEPTION] : Engine BP5Reader does not support DoAllStepsBlocksInfo

Maybe ADIOS 2.8.1 has some issues in its BP5 engine code. You can build latest ADIOS 2.8.3 and use it to build and run E3SM-IO (also use bpls of ADIOS 2.8.3).

[wkliao]

@khou2020
I think the only issue left is bpstat.
Looks like calling adios2_inquire_blockinfo() causes errors due to invalid memcpy.

(gdb) where
#0  0x00007ffff5b9acf7 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#1  0x00007ffff79a0825 in adios2_inquire_blockinfo (engine=0x6b6ae0, variable=0x9edac0, step=0)
    at /files2/scratch/wkliao/ADIOS/ADIOS2-2.8.3/bindings/C/adios2/c/adios2_c_engine.cpp:686
#2  0x0000000000403f45 in bpstat_core (inpath="../test_output/adios_blob_map_f_case_16p_h0.bp", 
    cfg=..., stat=...) at ../../utils/bpstat.cpp:185
#3  0x0000000000404c0d in main (argc=3, argv=0x7fffffffdd08) at ../../utils/bpstat.cpp:427

[khou2020]

Calling adios2_inquire_blockinfo on ADIOS 2.8.3 gets segmentation fault. We fall back to iterating the blocks until we get an error.
BP5 does not allow reading individual subfile (subfiles are not valid BP file). For now, bpstat only support single process.

[wkliao]

@khou2020
bpstat prints a lot of warning messages like below. Can they be silenced?

CMD = utils/bpstat ./test_output/adios_blob_map_f_case_16p_h0.bp
^[[1;36m[Sun Nov  6 19:03:08 2022]^[[1;31m [ADIOS2 ERROR]^[[0m <Helper> <adiosSystem> <ExceptionToError> : adios2_selection_size: ^[[1;36m[Sun Nov  6 19:03:08 2022]^[[1;34m [ADIOS2 EXCEPTION]^[[0m <Core> <Variable> <DoCount> : blockID 1 from SetBlockSelection is out of bounds for available blocks size 1 for variable /__pio__/decomp/512 for step 0, in call to Variable<T>::Count()^[[0m
^[[0m

[wkliao]

bpstat hangs when input file does not exist.
Try command utils/bpstat non_exist

[khou2020]

@khou2020 bpstat prints a lot of warning messages like below. Can they be silenced?

CMD = utils/bpstat ./test_output/adios_blob_map_f_case_16p_h0.bp
^[[1;36m[Sun Nov  6 19:03:08 2022]^[[1;31m [ADIOS2 ERROR]^[[0m <Helper> <adiosSystem> <ExceptionToError> : adios2_selection_size: ^[[1;36m[Sun Nov  6 19:03:08 2022]^[[1;34m [ADIOS2 EXCEPTION]^[[0m <Core> <Variable> <DoCount> : blockID 1 from SetBlockSelection is out of bounds for available blocks size 1 for variable /__pio__/decomp/512 for step 0, in call to Variable<T>::Count()^[[0m
^[[0m

I am not aware of any API that can turn ADIOS error message off.

[wkliao]

Thanks, @dqwu