Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Crash in libssl when opening connections in parallel on Windows #595

Closed
larskanis opened this issue Oct 16, 2024 · 0 comments · Fixed by #598
Closed

Crash in libssl when opening connections in parallel on Windows #595

larskanis opened this issue Oct 16, 2024 · 0 comments · Fixed by #598

Comments

@larskanis
Copy link
Collaborator

The fat binary gem for Windows pg-1.5.8-x64-mingw-ucrt.gem crashes when PG.connect is simultaneously called from multiple threads.

require "pg"

CONNECT_DB_MUTEX = Mutex.new

4.times.map do |idx|
  Thread.new do
    # CONNECT_DB_MUTEX.synchronize do
      p PG.connect dbname: 'postgres' #, sslmode: 'disable'
    # end
  end
end.each(&:join)
GDB output looks like so: 
warning: HEAP[ruby.exe]:
warning: Invalid address specified to RtlFreeHeap( 00000000006A0000, 0000000006128670 )

Thread 8 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 26424.0x4c10]
0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007fffebe21eef in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007fffebe553c8 in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007fffebe0cd18 in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007fffebd8c324 in ntdll!RtlGetCurrentServiceSessionId () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007fffebd8aff1 in ntdll!RtlFreeHeap () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007fffe994364b in ucrtbase!_free_base () from C:\Windows\System32\ucrtbase.dll
#7  0x000000006457ef13 in CRYPTO_free ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#8  0x000000006454c18f in err_clear ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#9  0x000000006454c322 in ERR_pop_to_mark ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#10 0x000000006444d461 in ssl_evp_cipher_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#11 0x00000000644374f9 in ssl_load_ciphers ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#12 0x000000006444593d in SSL_CTX_new_ex ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#13 0x0000000064445f93 in SSL_CTX_new ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#14 0x0000000064420a13 in initialize_SSL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#15 0x000000006441fcd3 in pgtls_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#16 0x000000006441b28c in pqsecure_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#17 0x00000000644070df in PQconnectPoll ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#18 0x000000007038143b in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#19 0x00007fff97bebfb5 in x64-ucrt-ruby330!rb_nogvl () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#20 0x00007fff97bec241 in x64-ucrt-ruby330!rb_thread_call_without_gvl ()
   from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#21 0x0000000070381a88 in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#22 0x000000007038ab3a in pg_ext!Init_pg_ext ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#23 0x00007fff97c2b410 in x64-ucrt-ruby330!rb_error_arity () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#24 0x00007fff97c3d950 in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#25 0x00007fff97c3de6b in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#26 0x00007fff97c49742 in x64-ucrt-ruby330!rb_ec_obj_respond_to () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#27 0x00007fff97c3ae45 in x64-ucrt-ruby330!rb_vm_exec () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#28 0x00007fff97c3f9c5 in x64-ucrt-ruby330!rb_vm_invoke_proc () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#29 0x00007fff97be8deb in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#30 0x00007fff97be9472 in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#31 0x00007fff97be9bcf in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#32 0x00007fffea79257d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#33 0x00007fffebdaaf28 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#34 0x0000000000000000 in ?? ()
(gdb)
or so: 
warning: HEAP[ruby.exe]:
warning: Invalid address specified to RtlReAllocateHeap( 0000000000720000, 00000000061648F0 )

Thread 11 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 27176.0x6790]
0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007fffebe5705f in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007fffebe21eef in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007fffebe55e99 in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007fffebe0eaa1 in ntdll!EtwLogTraceEvent () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007fffebd922c7 in ntdll!RtlReAllocateHeap () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007fffebd9218a in ntdll!RtlReAllocateHeap () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007fffe99418e9 in ucrtbase!_realloc_base () from C:\Windows\System32\ucrtbase.dll
#7  0x000000006457edc6 in CRYPTO_realloc ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#8  0x0000000064576eb9 in expand ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#9  0x0000000064576a67 in OPENSSL_LH_insert ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#10 0x00000000645798c2 in lh_NAMENUM_ENTRY_insert ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#11 0x000000006457a01a in namemap_add_name ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#12 0x000000006457a0ca in ossl_namemap_add_name ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#13 0x000000006457a4ea in get_legacy_evp_names ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#14 0x000000006457a56a in get_legacy_cipher_names ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#15 0x0000000064592a28 in do_all_fn ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#16 0x0000000064592a5d in lh_OBJ_NAME_doall_OBJ_DOALL_thunk ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#17 0x0000000064576d0c in doall_util_fn ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#18 0x0000000064576e4a in OPENSSL_LH_doall_arg_thunk ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#19 0x0000000064592a96 in lh_OBJ_NAME_doall_OBJ_DOALL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#20 0x0000000064592ae3 in OBJ_NAME_do_all ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#21 0x000000006457a748 in ossl_namemap_stored ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#22 0x0000000064557e3d in inner_evp_generic_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#23 0x00000000645582bc in evp_generic_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#24 0x0000000064557705 in EVP_CIPHER_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#25 0x000000006444d458 in ssl_evp_cipher_fetch ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#26 0x00000000644374f9 in ssl_load_ciphers ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#27 0x000000006444593d in SSL_CTX_new_ex ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#28 0x0000000064445f93 in SSL_CTX_new ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#29 0x0000000064420a13 in initialize_SSL ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#30 0x000000006441fcd3 in pgtls_open_client ()
--Type <RET> for more, q to quit, c to continue without paging--
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#31 0x000000006441b28c in pqsecure_open_client ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#32 0x00000000644070df in PQconnectPoll ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\x64-mingw-ucrt\libpq.dll
#33 0x000000007038143b in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#34 0x00007fff97bebfb5 in x64-ucrt-ruby330!rb_nogvl () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#35 0x00007fff97bec241 in x64-ucrt-ruby330!rb_thread_call_without_gvl ()
   from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#36 0x0000000070381a88 in ?? ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#37 0x000000007038ab3a in pg_ext!Init_pg_ext ()
   from C:\Ruby33-x64\lib\ruby\gems\3.3.0\gems\pg-1.5.8-x64-mingw-ucrt\lib\3.3\pg_ext.so
#38 0x00007fff97c2b410 in x64-ucrt-ruby330!rb_error_arity () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#39 0x00007fff97c3d950 in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#40 0x00007fff97c3de6b in x64-ucrt-ruby330!rb_vm_invoke_bmethod () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#41 0x00007fff97c49742 in x64-ucrt-ruby330!rb_ec_obj_respond_to () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#42 0x00007fff97c3ae45 in x64-ucrt-ruby330!rb_vm_exec () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#43 0x00007fff97c3f9c5 in x64-ucrt-ruby330!rb_vm_invoke_proc () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#44 0x00007fff97be8deb in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#45 0x00007fff97be9472 in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#46 0x00007fff97be9bcf in x64-ucrt-ruby330!rb_vm_proc_local_ep () from C:\Ruby33-x64\bin\x64-ucrt-ruby330.dll
#47 0x00007fffea79257d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#48 0x00007fffebdaaf28 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#49 0x0000000000000000 in ?? ()
(gdb)

It doesn't crash when sslmode: 'disable' is set, or when a Mutex avoids concurrent connections.

It also doesn't happen, when libpq and libssl is used from the MSYS2-MINGW packages like so:

$ pacman -S %MINGW_PACKAGE_PREFIX%-postgresql
$ gem uninst pg
$ gem inst pg --platform ruby
$ ruby test-crash-concurrent-ar.rb
#<PG::Connection:0x000001f05a279ab0 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27c710 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27b8b0 host=localhost port=5432 user=kanis>
#<PG::Connection:0x000001f05a27a9b0 host=localhost port=5432 user=kanis>

Possibly there's some relationship with the crash observed on Linux with binary gems in #551 .
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
4126827 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
@larskanis larskanis mentioned this issue Oct 24, 2024
Merged
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
abc13f4 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
larskanis added a commit to larskanis/ruby-pg that referenced this issue Oct 24, 2024
@larskanis
Enable thread safety in static OpenSSL build
716221a 
This is disabled by default due to the `-static` option.
The `thread` doesn't work with `-static`, that's why it is enabled by `-DOPENSSL_THREADS`.
The thread functions are used in libcrypto.a and libssl.a but only defined in libssl.a.

Fixes ged#595
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable thread safety in static OpenSSL build larskanis/ruby-pg
1 participant
@larskanis