Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

SIGSEGV on OTLPAgent deletion when running test-nsolid-otlp-config-start.js #33

Closed
santigimeno opened this issue Nov 24, 2023 · 0 comments · Fixed by #34
Closed

SIGSEGV on OTLPAgent deletion when running test-nsolid-otlp-config-start.js #33

santigimeno opened this issue Nov 24, 2023 · 0 comments · Fixed by #34
Assignees

Comments

@santigimeno
Copy link
Member

Observed on a resource-constrained box under load.

(lldb) bt all
* thread #1, name = 'nsolid', stop reason = signal SIGSEGV
  * frame #0: 0x00005633ec739ead nsolid`std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(this=0x00007fa160004f00) at shared_ptr_base.h:168:16
    frame #1: 0x00005633ec738c8f nsolid`std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count(this=0x00007fa1600692b8) at shared_ptr_base.h:705:21
    frame #2: 0x00005633ee35f028 nsolid`std::__shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr(this=0x00007fa1600692b0) at shared_ptr_base.h:1154:7
    frame #3: 0x00005633ee35f048 nsolid`std::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::~shared_ptr(this=nullptr) at shared_ptr.h:122:11
    frame #4: 0x00005633ee360ab8 nsolid`opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::shared_ptr_wrapper::~shared_ptr_wrapper(this=0x00007fa1600692a8) at shared_ptr.h:44:36
    frame #5: 0x00005633ee35ebe3 nsolid`opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::~shared_ptr(this=0x00007fa1600692a8) at shared_ptr.h:112:48
    frame #6: 0x00005633ee35a7d5 nsolid`opentelemetry::v1::ext::http::client::curl::HttpClient::~HttpClient(this=0x00007fa1600690e0) at http_client_curl.cc:161:1
    frame #7: 0x00005633ee36daa7 nsolid`void __gnu_cxx::new_allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>::destroy<opentelemetry::v1::ext::http::client::curl::HttpClient>(this=0x00007fa1600690e0, __p=0x00007fa1600690e0) at new_allocator.h:168:13
    frame #8: 0x00005633ee36da27 nsolid`void std::allocator_traits<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >::destroy<opentelemetry::v1::ext::http::client::curl::HttpClient>(__a=0x00007fa1600690e0, __p=0x00007fa1600690e0) at alloc_traits.h:535:15
    frame #9: 0x00005633ee36d8a5 nsolid`std::_Sp_counted_ptr_inplace<opentelemetry::v1::ext::http::client::curl::HttpClient, std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>, (__gnu_cxx::_Lock_policy)2>::_M_dispose(this=0x00007fa1600690d0) at shared_ptr_base.h:528:35
    frame #10: 0x00005633ec739eb9 nsolid`std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(this=0x00007fa1600690d0) at shared_ptr_base.h:168:16
    frame #11: 0x00005633ec738c8f nsolid`std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count(this=0x00007fa16001f0c8) at shared_ptr_base.h:705:21
    frame #12: 0x00005633ee343a8a nsolid`std::__shared_ptr<opentelemetry::v1::ext::http::client::HttpClient, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr(this=0x00007fa16001f0c0) at shared_ptr_base.h:1154:7
    frame #13: 0x00005633ee343aaa nsolid`std::shared_ptr<opentelemetry::v1::ext::http::client::HttpClient>::~shared_ptr(this=nullptr) at shared_ptr.h:122:11
    frame #14: 0x00005633ee33fe20 nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpClient::~OtlpHttpClient(this=0x00007fa16001f020) at otlp_http_client.cc:685:1
    frame #15: 0x00005633ecb72998 nsolid`std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpClient>::operator(this=0x00007fa1600cb750, __ptr=0x00007fa16001f020)(opentelemetry::v1::exporter::otlp::OtlpHttpClient*) const at unique_ptr.h:85:2
    frame #16: 0x00005633ecb68b34 nsolid`std::unique_ptr<opentelemetry::v1::exporter::otlp::OtlpHttpClient, std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpClient> >::~unique_ptr(this=0x7fa16001f020) at unique_ptr.h:361:17
    frame #17: 0x00005633ecb5ff10 nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpExporter::~OtlpHttpExporter(this=0x00007fa1600cb6e0) at otlp_http_exporter.h:29:28
    frame #18: 0x00005633ecb5ff4c nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpExporter::~OtlpHttpExporter(this=0x00007fa1600cb6e0) at otlp_http_exporter.h:29:28
    frame #19: 0x00005633ecb5ff85 nsolid`std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpExporter>::operator(this=0x00005633f30d6790, __ptr=0x00007fa1600cb6e0)(opentelemetry::v1::exporter::otlp::OtlpHttpExporter*) const at unique_ptr.h:85:2
    frame #20: 0x00005633ecb5853a nsolid`std::unique_ptr<opentelemetry::v1::exporter::otlp::OtlpHttpExporter, std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpExporter> >::~unique_ptr(this=0x7fa1600cb6e0) at unique_ptr.h:361:17
    frame #21: 0x00005633ecb4e452 nsolid`node::nsolid::otlp::OTLPAgent::~OTLPAgent(this=0x00005633f30d6120) at otlp_agent.cc:79:1
    frame #22: 0x00007fa17f0d8495 libc.so.6`__run_exit_handlers(status=0, listp=0x00007fa17f2ac838, run_list_atexit=true, run_dtors=true) at exit.c:113:8
    frame #23: 0x00007fa17f0d8610 libc.so.6`__GI_exit(status=<unavailable>) at exit.c:143:3
    frame #24: 0x00007fa17f0bcd97 libc.so.6`__libc_start_call_main(main=(nsolid`main at node_main.cc:96:34), argc=2, argv=0x00007fff22194d28) at libc_start_call_main.h:74:3
    frame #25: 0x00007fa17f0bce40 libc.so.6`__libc_start_main_impl(main=(nsolid`main at node_main.cc:96:34), argc=2, argv=0x00007fff22194d28, init=(_rtld_global), fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fff22194d18) at libc-start.c:392:3
    frame #26: 0x00005633ec731355 nsolid`_start + 37
  thread #2, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #3, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #4, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #5, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #6, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #7, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #8, stop reason = signal 0
    frame #0: 0x00007fa17f1b8d58 libc.so.6`__GI_epoll_pwait(epfd=23, events=0x00007fa177ffbbd0, maxevents=1024, timeout=100, set=0x0000000000000000) at epoll_pwait.c:40:10
    frame #1: 0x00005633edf14d1f nsolid`uv__io_poll(loop=0x00005633f30e9048, timeout=100) at linux.c:1360:12
    frame #2: 0x00005633edef84c4 nsolid`uv_run(loop=0x00005633f30e9048, mode=UV_RUN_DEFAULT) at core.c:453:5
    frame #3: 0x00005633ecc1e6c8 nsolid`node::nsolid::EnvList::env_list_routine_((null)=0x00005633f30e94d8, envlist=0x00005633f30e9020) at nsolid_api.cc:1488:14
    frame #4: 0x00005633ecc330d2 nsolid`void nsuv::ns_thread::create_proxy_<void (*)(nsuv::ns_thread*, node::nsolid::EnvList*), node::nsolid::EnvList>(arg=0x00005633f30e94d8) at nsuv-inl.h:1829:6
    frame #5: 0x00007fa17f127ac3 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:442:8
    frame #6: 0x00007fa17f1b8bf4 libc.so.6`__clone at clone.S:100
  thread #9, stop reason = signal 0
    frame #0: 0x00007fa17f124117 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x00007fa17f1240e9 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x00007fa17f1240e9 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x00005633f30ea0e0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x00007fa17f12fc78 libc.so.6`__new_sem_wait_slow64(sem=0x00005633f30ea0e0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x00005633edf0d508 nsolid`uv__sem_wait(sem=0x00005633f30ea0e0) at thread.c:639:9
    frame #5: 0x00005633edf0d67f nsolid`uv_sem_wait(sem=0x00005633f30ea0e0) at thread.c:695:5
    frame #6: 0x00005633ecc5ae8c nsolid`node::inspector::(anonymous namespace)::StartIoThreadMain(unused=0x0000000000000000) at inspector_agent.cc:85:16
    frame #7: 0x00007fa17f127ac3 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:442:8
    frame #8: 0x00007fa17f1b8bf4 libc.so.6`__clone at clone.S:100
@santigimeno santigimeno self-assigned this Nov 24, 2023
santigimeno added a commit that referenced this issue Nov 24, 2023
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
santigimeno added a commit that referenced this issue Dec 6, 2023
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
PR-URL: #34
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
trevnorris pushed a commit that referenced this issue Dec 7, 2023
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
PR-URL: #34
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant