Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

node v4.2.2 app finalize abruptly signaling SIGILL - Illegal instruction #4121

Closed
diegosanchez opened this issue Dec 2, 2015 · 20 comments
Closed

node v4.2.2 app finalize abruptly signaling SIGILL - Illegal instruction #4121

diegosanchez opened this issue Dec 2, 2015 · 20 comments
Labels
openssl Issues and PRs related to the OpenSSL dependency. tls Issues and PRs related to the tls subsystem.

Comments

@diegosanchez
Copy link

Having built node v4.2.2 I get a segmentation fault.

The binary shasum is: 6ec12774e7b80f3bc391ed0ebdc6890d9660907f v4.2.2/bin/node.

Because of this I recompile de binary using the command ./configure --prefix=<path> --gdb --debug in order to perform a backtrace.

Bellow the backtrace:

coredumpctl gdb 20259
           PID: 20259 (node)
           UID: 1000 (diegos)
           GID: 1000 (diegos)
        Signal: 4 (ILL)
     Timestamp: Mon 2015-11-30 18:49:56 ART (1 day 20h ago)
  Command Line: node index
    Executable: /home/diegos/bin/node-4.2.2/bin/node
 Control Group: /user.slice/user-1000.slice/session-c4.scope
          Unit: session-c4.scope
         Slice: user-1000.slice
       Session: c4
     Owner UID: 1000 (diegos)
       Boot ID: 258848166ac845eaa4916bab77534870
    Machine ID: 85f0595d909245e7b2a1a41e3752b3e6
      Hostname: tao
      Coredump: /var/lib/systemd/coredump/core.node.1000.258848166ac845eaa4916bab77534870.20259.1448920196000000.lz4
       Message: Process 20259 (node) of user 1000 dumped core.

GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/diegos/bin/node-4.2.2/bin/node...bt
done.
[New LWP 20259]
[New LWP 20267]
[New LWP 20261]
[New LWP 20263]
[New LWP 20268]
[New LWP 20262]
[New LWP 20269]
[New LWP 20260]
[New LWP 20270]

warning: Could not load shared library symbols for 4 libraries, e.g. linux-vdso.so.1.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `node index'.
Program terminated with signal SIGILL, Illegal instruction.
#0  aesni_cbc_sha256_enc () at ../deps/openssl/asm/x64-elf-gas/aes/aesni-sha256-x86_64.s:29
29      ud2
[Current thread is 1 (Thread 0x7f1dda383740 (LWP 20259))]
(gdb) bt
#0  aesni_cbc_sha256_enc () at ../deps/openssl/asm/x64-elf-gas/aes/aesni-sha256-x86_64.s:29
#1  0x0000000000c8a259 in aesni_cbc_hmac_sha256_cipher (ctx=0x2a227c0, 
    out=0x2abe268 "\332\367\331\065\370h\210\366_g({\272\226>\231POST /agent_listener/invoke_raw_method?marshal_format=json&protocol_version=14&license_key=insertKeyHere&method=get_redirect_host HTTP/1.1\r\nHost: collector.newrelic.com\r\nUser-Agent: Ne"..., 
    in=0x2abe268 "\332\367\331\065\370h\210\366_g({\272\226>\231POST /agent_listener/invoke_raw_method?marshal_format=json&protocol_version=14&license_key=insertKeyHere&method=get_redirect_host HTTP/1.1\r\nHost: collector.newrelic.com\r\nUser-Agent: Ne"..., len=400) at ../deps/openssl/openssl/crypto/evp/e_aes_cbc_hmac_sha256.c:506
#2  0x0000000000c92059 in EVP_Cipher (ctx=0x2a227c0, 
    out=0x2abe268 "\332\367\331\065\370h\210\366_g({\272\226>\231POST /agent_listener/invoke_raw_method?marshal_format=json&protocol_version=14&license_key=insertKeyHere&method=get_redirect_host HTTP/1.1\r\nHost: collector.newrelic.com\r\nUser-Agent: Ne"..., 
    in=0x2abe268 "\332\367\331\065\370h\210\366_g({\272\226>\231POST /agent_listener/invoke_raw_method?marshal_format=json&protocol_version=14&license_key=insertKeyHere&method=get_redirect_host HTTP/1.1\r\nHost: collector.newrelic.com\r\nUser-Agent: Ne"..., inl=400) at ../deps/openssl/openssl/crypto/evp/evp_lib.c:203
#3  0x0000000000beb5c1 in tls1_enc (s=0x294fac0, send=1) at ../deps/openssl/openssl/ssl/t1_enc.c:880
#4  0x0000000000bca693 in do_ssl3_write (s=0x294fac0, type=23, 
    buf=0x2b05d70 "POST /agent_listener/invoke_raw_method?marshal_format=json&protocol_version=14&license_key=insertKeyHere&method=get_redirect_host HTTP/1.1\r\nHost: collector.newrelic.com\r\nUser-Agent: NewRelic-NodeAgent"..., len=341, create_empty_fragment=0) at ../deps/openssl/openssl/ssl/s3_pkt.c:1042
#5  0x0000000000bca08d in ssl3_write_bytes (s=0x294fac0, type=23, buf_=0x2b05d70, len=341) at ../deps/openssl/openssl/ssl/s3_pkt.c:830
#6  0x0000000000bc88e1 in ssl3_write (s=0x294fac0, buf=0x2b05d70, len=341) at ../deps/openssl/openssl/ssl/s3_lib.c:4399
#7  0x0000000000bdf728 in SSL_write (s=0x294fac0, buf=0x2b05d70, num=341) at ../deps/openssl/openssl/ssl/ssl_lib.c:1046
#8  0x00000000015d8f47 in node::TLSWrap::ClearIn (this=0x28331a0) at ../src/tls_wrap.cc:467
#9  0x00000000015da611 in node::TLSWrap::Cycle (this=0x28331a0) at ../src/tls_wrap.h:100
#10 0x00000000015d993f in node::TLSWrap::DoRead (this=0x28331a0, nread=300, buf=0x7ffedb271570, pending=UV_UNKNOWN_HANDLE) at ../src/tls_wrap.cc:700
#11 0x00000000015d96d0 in node::TLSWrap::OnReadImpl (nread=300, buf=0x7ffedb271570, pending=UV_UNKNOWN_HANDLE, ctx=0x28331a0) at ../src/tls_wrap.cc:640
#12 0x000000000155f3fb in node::StreamResource::OnRead (this=0x294f6d0, nread=300, buf=0x7ffedb271570, pending=UV_UNKNOWN_HANDLE) at ../src/stream_base.h:160
#13 0x00000000015af24c in node::StreamWrap::OnReadCommon (handle=0x294f720, nread=300, buf=0x7ffedb271570, pending=UV_UNKNOWN_HANDLE) at ../src/stream_wrap.cc:250
#14 0x00000000015af2e4 in node::StreamWrap::OnRead (handle=0x294f720, nread=300, buf=0x7ffedb271570) at ../src/stream_wrap.cc:265
#15 0x0000000001619409 in uv__read (stream=0x294f720) at ../deps/uv/src/unix/stream.c:1178
#16 0x0000000001619695 in uv__stream_io (loop=0x1e925c0 <default_loop_struct>, w=0x294f7a8, events=1) at ../deps/uv/src/unix/stream.c:1241
#17 0x000000000161f169 in uv__io_poll (loop=0x1e925c0 <default_loop_struct>, timeout=0) at ../deps/uv/src/unix/linux-core.c:345
#18 0x000000000160c17b in uv_run (loop=0x1e925c0 <default_loop_struct>, mode=UV_RUN_ONCE) at ../deps/uv/src/unix/core.c:341
#19 0x000000000156bff0 in node::StartNodeInstance (arg=0x7ffedb2748c0) at ../src/node.cc:4045
#20 0x000000000156c2b4 in node::Start (argc=2, argv=0x27c5c20) at ../src/node.cc:4121
#21 0x000000000159a6c7 in main (argc=2, argv=0x7ffedb274a38) at ../src/node_main.cc:44
(gdb)

SO Information

uname -a
Linux tao 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux

I don't want to overwhelm you with immaterial information so that I'm eager to supply requested information.
@mscdex mscdex added tls Issues and PRs related to the tls subsystem. openssl Issues and PRs related to the OpenSSL dependency. labels Dec 2, 2015
@mscdex
Copy link
Contributor

mscdex commented Dec 2, 2015

/cc @nodejs/crypto

@diegosanchez
Copy link
Author

I cannot reproduce the error neither on node v0.12 xor v0.10 but on v4.2.2, v5.0.0, v5.1.0

@indutny
Copy link
Member

indutny commented Dec 2, 2015

@diegosanchez what is the platform you are running this on? What CPU features does it support? Could you please gist your /proc/cpuinfo?

@diegosanchez
Copy link
Author

  • Platform: PC Intel Compatible
  • SO: Linux ARCH
  • Kernel: 4.2.5-1
  • /proc/cpuinfo
processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 22
model       : 0
model name  : Quad-Core Processor (up to 1.4GHz)
stepping    : 1
microcode   : 0x700010f
cpu MHz     : 1000.000
cache size  : 2048 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt topoext perfctr_nb bpext perfctr_l2 arat cpb hw_pstate proc_feedback npt lbrv svm_lock nrip_save tsc_scale flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1 xsaveopt
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 1997.42
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [11]

processor   : 1
vendor_id   : AuthenticAMD
cpu family  : 22
model       : 0
model name  : Quad-Core Processor (up to 1.4GHz)
stepping    : 1
microcode   : 0x700010f
cpu MHz     : 600.000
cache size  : 2048 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 4
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt topoext perfctr_nb bpext perfctr_l2 arat cpb hw_pstate proc_feedback npt lbrv svm_lock nrip_save tsc_scale flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1 xsaveopt
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 1997.42
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [11]

processor   : 2
vendor_id   : AuthenticAMD
cpu family  : 22
model       : 0
model name  : Quad-Core Processor (up to 1.4GHz)
stepping    : 1
microcode   : 0x700010f
cpu MHz     : 1000.000
cache size  : 2048 KB
physical id : 0
siblings    : 4
core id     : 2
cpu cores   : 4
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt topoext perfctr_nb bpext perfctr_l2 arat cpb hw_pstate proc_feedback npt lbrv svm_lock nrip_save tsc_scale flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1 xsaveopt
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 1997.42
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [11]

processor   : 3
vendor_id   : AuthenticAMD
cpu family  : 22
model       : 0
model name  : Quad-Core Processor (up to 1.4GHz)
stepping    : 1
microcode   : 0x700010f
cpu MHz     : 600.000
cache size  : 2048 KB
physical id : 0
siblings    : 4
core id     : 3
cpu cores   : 4
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt topoext perfctr_nb bpext perfctr_l2 arat cpb hw_pstate proc_feedback npt lbrv svm_lock nrip_save tsc_scale flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1 xsaveopt
bugs        : fxsave_leak sysret_ss_attrs
bogomips    : 1997.42
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [11]

@bnoordhuis
Copy link
Member

Does env OPENSSL_ia32cap=0 node app.js work?

@indutny
Copy link
Member

indutny commented Dec 2, 2015

@diegosanchez are you running this in some sort of virtualized environment like VirtualBox or something like this? Or is it running on a bare metal?

@indutny
Copy link
Member

indutny commented Dec 2, 2015

@bnoordhuis I think it is BMI2 that is absent, that causes this.

@indutny
Copy link
Member

indutny commented Dec 2, 2015

It is strange that OpenSSL did not pick up AVX+SSSE3, though

@indutny
Copy link
Member

indutny commented Dec 2, 2015

Ah, it is AMD...

@diegosanchez
Copy link
Author

@indutny Not it isn't a virtualized env.

@indutny
Copy link
Member

indutny commented Dec 2, 2015

@diegosanchez so far it looks like OpenSSL assembly code is not supported by your CPU. To work it should either:

  • Have SHA support
  • XOP support
  • BMI2+AVX2+BMI1
  • Be an Intel CPU and have AVX+SSSE3

From the cpuinfo that you have posted - it looks like your CPU does not support either of this modes. (@bnoordhuis please correct me if I'm off with this statement).

The simplest fix would be to use ./configure --openssl-no-asm before building node.js. This will unfortunately make crypto slower, but it should work without crashing.

@diegosanchez
Copy link
Author

@indutny

It seems that it env OPENSSL_ia32cap=0 node index.js works. I would like to know what that means (more research is needed).

So far, it is a good workaround.

I have narrowed down the issue a bit more here Project where I reproduce the error

@bnoordhuis
Copy link
Member

I would like to know what that means (more research is needed).

Here is documentation. In a nutshell, openssl tries (and apparently fails in your case) to auto-detect the CPU's capabilities. OPENSSL_ia32cap=0 tells it to skip auto-detection and just be very conservative.

@mscdex
Copy link
Contributor

mscdex commented Dec 3, 2015

It looks like this was fixed upstream recently?

@diegosanchez
Copy link
Author

@bnoordhuis @indutny awesome thanks.
Was useful for you to have a backtrace with this level of details? I'm asking because the default level of debug information is -g and I would like to update it to -g3 it it were useful.

@bnoordhuis
Copy link
Member

Yes, it was helpful, thanks. The ud2 and the stack trace told me where to go looking.

@diegosanchez diegosanchez mentioned this issue Dec 3, 2015
Closed
@mscdex
Copy link
Contributor

mscdex commented Dec 3, 2015

@diegosanchez Can you try the master branch now and check if the upgrade to OpenSSL 1.0.2e helped you at all?

@diegosanchez
Copy link
Author

Sure
On Dec 3, 2015 6:14 PM, "Brian White" notifications@github.com wrote:

@diegosanchez https://github.com/diegosanchez Can you try the master
branch now and check if the upgrade to OpenSSL 1.0.2e helped you at all?

—
Reply to this email directly or view it on GitHub
#4121 (comment).

@diegosanchez
Copy link
Author

@mscdex Works fine!

with v6.0.0-pre

$ env |grep -i open
$ node --version
v6.0.0-pre
$ nodei
^C
$ # Not sigfault

whereas with v4.2.2

$ env|grep -i open
OPENSSL_ia32cap=0
$ node --version
v4.2.2
$ node index.js 
^C
$ # Not sigfault (as expected)
$ unset OPENSSL_ia32cap
$ env|grep -i open
$ node index.js 
Illegal instruction (core dumped)
$ # (as expected)

@mscdex
Copy link
Contributor

mscdex commented Dec 4, 2015

Awesome! 👍 If you want to stick to LTS, the OpenSSL upgrade should be in v4.2.3 now too.

@mscdex mscdex closed this as completed Dec 4, 2015
@johnhaley81 johnhaley81 mentioned this issue Jan 6, 2016
Merged
@snyk-bot snyk-bot mentioned this issue Apr 30, 2023
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
openssl Issues and PRs related to the OpenSSL dependency. tls Issues and PRs related to the tls subsystem.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mscdex @indutny @bnoordhuis @diegosanchez