Sync client assertion failure in unbinded session: Assertion failed: m_initiated

[kiburtse]

Expected results

Actual Results

src/realm/sync/client.cpp:1324: [realm-core-13.23.1] Assertion failed: m_initiated

stacktrace

[2023/10/20 17:37:18.596] 2: /System/Volumes/Data/data/mci/9e4d046c729b0e133e0154a0c04f0cfe/realm-core/src/realm/sync/client.cpp:1324: [realm-core-13.23.1] Assertion failed: m_initiated
[2023/10/20 17:37:18.596] 2: 0   realm-sync-tests                    0x00000001005a5f70 _ZN5realm4utilL18terminate_internalERNSt3__118basic_stringstreamIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 28
[2023/10/20 17:37:18.596] 2: 1   realm-sync-tests                    0x00000001005a5eb8 _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 400
[2023/10/20 17:37:18.596] 2: 2   realm-sync-tests                    0x00000001005a5d28 _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 0
[2023/10/20 17:37:18.596] 2: 3   realm-sync-tests                    0x000000010034fb74 _ZN5realm4sync14SessionWrapper14async_wait_forEbbNS_4util14UniqueFunctionIFvNS_6StatusEEEE + 260
[2023/10/20 17:37:18.596] 2: 4   realm-sync-tests                    0x000000010034bf0c _ZN5realm4sync14SessionWrapper43handle_pending_client_reset_acknowledgementEv + 344
[2023/10/20 17:37:18.596] 2: 5   realm-sync-tests                    0x000000010039db7c _ZN5realm4sync10ClientImpl7Session21receive_ident_messageENS0_15SaltedFileIdentE + 596
[2023/10/20 17:37:18.596] 2: 6   realm-sync-tests                    0x000000010039d87c _ZN5realm4sync10ClientImpl10Connection21receive_ident_messageEyNS0_15SaltedFileIdentE + 68
[2023/10/20 17:37:18.596] 2: 7   realm-sync-tests                    0x000000010039a824 _ZN5realm5_impl14ClientProtocol22parse_message_receivedINS_4sync10ClientImpl10ConnectionEEEvRT_NSt3__117basic_string_viewIcNS8_11char_traitsIcEEEE + 508
[2023/10/20 17:37:18.596] 2: 8   realm-sync-tests                    0x00000001003a530c _ZN5realm4sync10ClientImpl10Connection21WebSocketObserverShim33websocket_binary_message_receivedENS_4util4SpanIKcLm18446744073709551615EEE + 120
[2023/10/20 17:37:18.596] 2: 9   realm-sync-tests                    0x0000000100387d7c _ZN12_GLOBAL__N_19WebSocket17frame_reader_loopEv + 1180
[2023/10/20 17:37:18.596] 2: 10  realm-sync-tests                    0x00000001001c75c8 _ZN5realm4sync7network7Service14BasicStreamOpsINS1_6SocketEE16BufferedReadOperINS_4util14UniqueFunctionIFvNSt3__110error_codeEmEEEE19recycle_and_executeEv + 324
[2023/10/20 17:37:18.596] 2: 11  realm-sync-tests                    0x000000010037f2d0 _ZN5realm4sync7network7Service4Impl8run_implEb + 540
[2023/10/20 17:37:18.596] 2: 12  realm-sync-tests                    0x0000000100372bf8 _ZN5realm4sync9websocket21DefaultSocketProvider10event_loopEv + 220
[2023/10/20 17:37:18.596] 2: 13  realm-sync-tests                    0x0000000100374658 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN5realm4sync9websocket21DefaultSocketProviderEFvvEPSA_EEEEEPvSF_ + 64
[2023/10/20 17:37:18.596] 2: 14  libsystem_pthread.dylib             0x000000018f23f878 _pthread_start + 320
[2023/10/20 17:37:18.596] 2: 15  libsystem_pthread.dylib             0x000000018f23a5e0 thread_start + 8
[2023/10/20 17:37:18.597] 2: !!! IMPORTANT: Please report this at https://github.com/realm/realm-core/issues/new/choose
[2023/10/20 17:37:18.597] 1/1 Test #2: SyncTests ........................Subprocess aborted***Exception:   0.52 sec
[2023/10/20 17:37:18.598] 0% tests passed, 1 tests failed out of 1
[2023/10/20 17:37:18.598] Total Test time (real) =   0.52 sec

Steps & Code to Reproduce

happened during CI run for #6911

Core version

Core version: 13.23.1

[kiburtse]

still happens with an update of baas from #7042:

[2023/11/01 23:23:27.673] 2: /System/Volumes/Data/data/mci/06960e543696eba5dd591b72255dda2c/realm-core/src/realm/sync/client.cpp:1329: [realm-core-13.23.2] Assertion failed: m_initiated
[2023/11/01 23:23:27.673] 2: 0   realm-sync-tests                    0x00000001050a9d84 _ZN5realm4utilL18terminate_internalERNSt3__118basic_stringstreamIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 28
[2023/11/01 23:23:27.673] 2: 1   realm-sync-tests                    0x00000001050a9ccc _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 400
[2023/11/01 23:23:27.673] 2: 2   realm-sync-tests                    0x00000001050a9b3c _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 0
[2023/11/01 23:23:27.673] 2: 3   realm-sync-tests                    0x0000000104e5c094 _ZN5realm4sync14SessionWrapper14async_wait_forEbbNS_4util14UniqueFunctionIFvNS_6StatusEEEE + 260
[2023/11/01 23:23:27.673] 2: 4   realm-sync-tests                    0x0000000104e58398 _ZN5realm4sync14SessionWrapper43handle_pending_client_reset_acknowledgementEv + 344
[2023/11/01 23:23:27.673] 2: 5   realm-sync-tests                    0x0000000104eaa228 _ZN5realm4sync10ClientImpl7Session21receive_ident_messageENS0_15SaltedFileIdentE + 596
[2023/11/01 23:23:27.673] 2: 6   realm-sync-tests                    0x0000000104ea9f28 _ZN5realm4sync10ClientImpl10Connection21receive_ident_messageEyNS0_15SaltedFileIdentE + 68
[2023/11/01 23:23:27.673] 2: 7   realm-sync-tests                    0x0000000104ea6ed0 _ZN5realm5_impl14ClientProtocol22parse_message_receivedINS_4sync10ClientImpl10ConnectionEEEvRT_NSt3__117basic_string_viewIcNS8_11char_traitsIcEEEE + 508
[2023/11/01 23:23:27.673] 2: 8   realm-sync-tests                    0x0000000104eb1968 _ZN5realm4sync10ClientImpl10Connection21WebSocketObserverShim33websocket_binary_message_receivedENS_4util4SpanIKcLm18446744073709551615EEE + 120
[2023/11/01 23:23:27.673] 2: 9   realm-sync-tests                    0x0000000104e94384 _ZN12_GLOBAL__N_19WebSocket17frame_reader_loopEv + 1180
[2023/11/01 23:23:27.673] 2: 10  realm-sync-tests                    0x0000000104cd3430 _ZN5realm4sync7network7Service14BasicStreamOpsINS1_6SocketEE16BufferedReadOperINS_4util14UniqueFunctionIFvNSt3__110error_codeEmEEEE19recycle_and_executeEv + 324
[2023/11/01 23:23:27.673] 2: 11  realm-sync-tests                    0x0000000104e8b8f4 _ZN5realm4sync7network7Service4Impl8run_implEb + 540
[2023/11/01 23:23:27.674] 2: 12  realm-sync-tests                    0x0000000104e7f118 _ZN5realm4sync9websocket21DefaultSocketProvider10event_loopEv + 220
[2023/11/01 23:23:27.674] 2: 13  realm-sync-tests                    0x0000000104e80b78 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN5realm4sync9websocket21DefaultSocketProviderEFvvEPSA_EEEEEPvSF_ + 64
[2023/11/01 23:23:27.674] 2: 14  libsystem_pthread.dylib             0x000000019a32b878 _pthread_start + 320
[2023/11/01 23:23:27.674] 2: 15  libsystem_pthread.dylib             0x000000019a3265e0 thread_start + 8
[2023/11/01 23:23:27.675] 2: !!! IMPORTANT: Please report this at https://github.com/realm/realm-core/issues/new/choose
[2023/11/01 23:23:27.676] 1/1 Test #2: SyncTests ........................Subprocess aborted***Exception:   0.35 sec
[2023/11/01 23:23:27.676] 0% tests passed, 1 tests failed out of 1

@danieltabacaru could you guess where to look?

[danieltabacaru]

still happens with an update of baas from #7042:

Did you use Michael's PR?

[kiburtse]

No, i've just looked at latest CI run on #7042, and one builder has this failure. So, you're saying it should be the same issue as #6844?

Don't know if it's the exact same issue, but I think it's worth trying. I remember looking into the issue and thinking that some sort of race may be the cause.

[sync-by-unito]

➤ Jonathan Reams commented:

I've taken a look at this but haven't been able to reproduce it locally. This isn't dependent on baas since it's in the realm-sync-tests which talk to the C++ test server. I think this is happening in one of the client reset tests. I think this may be happening in one of the client reset tests.

[kiburtse]

Yes right, i think i forgot to post here back then that this exact assertion also happens with object store test and baas server on linux debug. And you are right, i must be client reset tests.

log and detailed stacktrace as of v13.25.1

curl_easy_perform() failed when sending request to 'http://localhost:9090/api/client/v2.0/app/test-iaiac/location' with body '': Couldn't connect to server                                                                                                                                                                                                                                                                                                                                               
Filters: [baas]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
Randomness seeded to: 407221112                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
Websocket redirect test: User logged out                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
Websocket redirect test: User logged out                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
/home/user/r/src/src/realm/sync/client.cpp:1337: [realm-core-13.25.1] Assertion failed: m_initiated                                                                                                                                                                                                                                                                                                                                                                                                       
realm::util::terminate_internal(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&) at terminate.cpp:129 (discriminator 1)
std::remove_reference<std::initializer_list<realm::util::Printable>&>::type&& std::move<std::initializer_list<realm::util::Printable>&>(std::initializer_list<realm::util::Printable>&) at move.h:97
realm::util::terminate_with_info(char const*, char const*, long, char const*, std::initializer_list<realm::util::Printable>&&) at terminate.cpp:155
realm::sync::SessionWrapper::async_wait_for(bool, bool, realm::util::UniqueFunction<void (realm::Status)>) at client.cpp:1338
realm::sync::SessionWrapper::handle_pending_client_reset_acknowledgement() at client.cpp:1802 (discriminator 4)
realm::sync::ClientImpl::Session::handle_pending_client_reset_acknowledgement() at client.cpp:790
realm::sync::ClientImpl::Session::activate() at client_impl_base.cpp:1700
realm::sync::ClientImpl::Connection::activate_session(std::unique_ptr<realm::sync::ClientImpl::Session, std::default_delete<realm::sync::ClientImpl::Session> >) at client_impl_base.cpp:333
realm::sync::SessionWrapper::actualize(realm::sync::ServerEndpoint) at client.cpp:1532 (discriminator 2)
realm::sync::ClientImpl::actualize_and_finalize_session_wrappers() at client.cpp:572 (discriminator 2)
realm::sync::ClientImpl::ClientImpl(realm::sync::ClientConfig)::{lambda(realm::Status)#1}::operator()(realm::Status) const at client_impl_base.cpp:231
realm::util::UniqueFunction<void (realm::Status)>::SpecificImpl<realm::sync::ClientImpl::ClientImpl(realm::sync::ClientConfig)::{lambda(realm::Status)#1}>::call(realm::Status&&) at functional.hpp:154 (discriminator 2)
?? ??:0
realm::util::UniqueFunction<void (realm::Status)>::SpecificImpl<realm::sync::Trigger<realm::sync::ClientImpl>::trigger()::{lambda(realm::Status)#1}>::call(realm::Status&&) at functional.hpp:154 (discriminator 2)
realm::util::UniqueFunction<void (realm::Status)>::operator()(realm::Status) const at functional.hpp:94
realm::sync::ClientImpl::post(realm::util::UniqueFunction<void (realm::Status)>&&)::{lambda(realm::Status)#1}::operator()(realm::Status) const at client_impl_base.cpp:250 (discriminator 2)
realm::util::UniqueFunction<void (realm::Status)>::SpecificImpl<realm::sync::ClientImpl::post(realm::util::UniqueFunction<void (realm::Status)>&&)::{lambda(realm::Status)#1}>::call(realm::Status&&) at functional.hpp:154 (discriminator 2)
realm::util::UniqueFunction<void (realm::Status)>::operator()(realm::Status) const at functional.hpp:94
realm::sync::network::Service::PostOper<realm::util::UniqueFunction<void (realm::Status)> >::recycle_and_execute() at network.hpp:2015 (discriminator 2)
realm::sync::network::Service::Impl::execute(std::unique_ptr<realm::sync::network::Service::AsyncOper, realm::sync::network::Service::LendersOperDeleter>&) at network.cpp:1647
realm::sync::network::Service::Impl::run_impl(bool) at network.cpp:1574 (discriminator 2)
realm::sync::network::Service::Impl::run_until_stopped() at network.cpp:1386
realm::sync::network::Service::run_until_stopped() at network.cpp:1777
void std::__invoke_impl<void, void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*>(std::__invoke_memfun_deref, void (realm::sync::websocket::DefaultSocketProvider::*&&)(), realm::sync::websocket::DefaultSocketProvider*&&) at invoke.h:74
std::__invoke_result<void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*>::type std::__invoke<void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*>(void (realm::sync::websocket::DefaultSocketProvider::*&&)(), realm::sync::websocket::DefaultSocketProvider*&&) at invoke.h:97
void std::thread::_Invoker<std::tuple<void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) at std_thread.h:292
std::thread::_Invoker<std::tuple<void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*> >::operator()() at std_thread.h:299
std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*> > >::_M_run() at std_thread.h:244
std::default_delete<std::thread::_State>::operator()(std::thread::_State*) const at unique_ptr.h:99 (discriminator 1)                                                                                                                                                                                                                                                                                                                                                                                                                                                
!!! IMPORTANT: Please report this at https://github.com/realm/realm-core/issues/new/choose                                                                                                                                                                                                                                                                                                                                                                                                                
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                                                                                                                                                                                                           
realm-object-store-tests is a Catch2 v3.3.2 host application.                                                                                                                                                                                                                                                                                                                                                                                                                                             
Run with -? for options                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
-------------------------------------------------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                           
flx: client reset                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  unsuccessful replay of local changes                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  Recover: unsuccessful recovery leads to a manual reset                                                                                                                                                                                                                                                                                                                                                                                                                                                  
-------------------------------------------------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                           
/home/user/r/src/test/object-store/sync/flx_sync.cpp:920                                                                                                                                                                                                                                                                                                                                                                                                                                                  
...............................................................................                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
/home/user/r/src/test/object-store/sync/flx_sync.cpp:920: FAILED:                                                                                                                                                                                                                                                                                                                                                                                                                                         
  {Unknown expression after the reported line}                                                                                                                                                                                                                                                                                                                                                                                                                                                            
due to a fatal error condition:                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  SIGABRT - Abort (abnormal termination) signal                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
===============================================================================                                                                                                                                                                                                                                                                                                                                                                                                                           
test cases:   55 |   54 passed | 1 failed                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
assertions: 5054 | 5053 passed | 1 failed          

[sync-by-unito]

➤ Jonathan Reams commented:

The only thing I can think of is that we're racing to read this variable

realm-core/src/realm/sync/client.cpp

Line 1275 in 822c29d

m_initiated = true;

in the event loop with the application's thread to actualize the sync session.