Deadlock in PIConGPU when instrumenting locks

[skyreflectedinmirrors]

To build, follow instructions in: #145
Use binary rewrite to instrument (no exclusions needed, as boost doesn't come in because of #144)

When running, it hangs at MPI_Init with:

0x00007ffff3d0b0ec in __lll_lock_wait_private () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff3d0b0ec in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007ffff3d83810 in malloc () from /lib64/libc.so.6
#2  0x00007ffff4142d6c in operator new(unsigned long) () from /lib64/libstdc++.so.6
#3  0x00007fffde74463d in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#4  0x00007fffde702e98 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#5  0x00007fffdda4b26f in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#6  0x00007fffde4d80e0 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#7  0x00007fffde5760d9 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#8  0x00007fffde578661 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#9  0x00007fffde55ef65 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#10 0x00007fffe9d45d48 in ucm_event_enter () at event/event.c:161
#11 0x00007fffe9d46acf in ucm_sbrk (increment=139264) at event/event.c:376
#12 0x00007ffff3d8528d in __default_morecore () from /lib64/libc.so.6
#13 0x00007ffff3d814db in sysmalloc () from /lib64/libc.so.6
#14 0x00007ffff3d82659 in _int_malloc () from /lib64/libc.so.6
#15 0x00007ffff3d84486 in calloc () from /lib64/libc.so.6
#16 0x00007fffea6a36c1 in opal_hash_table_init2 () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libopen-pal.so.80
#17 0x00007fffea7272c2 in mca_base_pvar_init () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libopen-pal.so.80
#18 0x00007fffea723f25 in mca_base_var_init () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libopen-pal.so.80
#19 0x00007fffea6abbd2 in opal_init_util () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libopen-pal.so.80
#20 0x00007ffff60c4c3f in ompi_mpi_init () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libmpi.so.80
#21 0x00007ffff60fe301 in PMPI_Init () from /share/modules/gcc-8_3_1/openmpi/5.0.0rc2-ucx1.11.2/lib/libmpi.so.80
#22 0x00007fffde4cec1d in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#23 0x0000000001a90d67 in ?? ()
#24 0x0000000002026010 in ?? ()
#25 0x00007fffdf5b2b60 in ?? () from /home/nicurtis/omnitrace-install/lib/libomnitrace.so
#26 0x00000000023b39e0 in ?? ()
#27 0x0000000000000000 in ?? ()

Disabling OMNITRACE_TRACE_THREAD_RW_LOCKS and OMNITRACE_TRACE_THREAD_SPIN_LOCKS allows progress

[jrmadsen]

Closed due to default behavior being changed.