Exception when using libykcs11.dll with Cygwin ssh

[dzhus]

Hello,

I'm running into the following issue with libykcs11.dll from yubico-piv-tool-2.3.1-win64.msi. When using it with ssh from Cygwin, I'm observing an exception:

> C:\cygwin64\bin\ssh.exe -I "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll" git@github.com
      0 [main] ssh 112 cygwin_exception::open_stackdumpfile: Dumping stack trace to ssh.exe.stackdump

ssh.exe.stackdump

Exception: STATUS_ACCESS_VIOLATION at rip=000100479FB1
rax=00007FFD153EE000 rbx=0000000A00036040 rcx=0000000000000000
rdx=0000000000000133 rsi=00000007FFFFB288 rdi=00000007FFFFB290
r8 =00007FFD153E58B0 r9 =0000000000000001 r10=0000000000000000
r11=0000000000000000 r12=0000000A00037010 r13=00007FFD153D0000
r14=0000000000000000 r15=0000000000000000
rbp=00007FFD153D6DC0 rsp=00000007FFFFB030
program=C:\cygwin64\bin\ssh.exe, pid 112, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B
Stack trace:
Frame         Function      Args
7FFD153D6DC0  000100479FB1 (000A00033C30, 0000000303E9, 000000000000, 000100000001) ssh.exe+0x79FB1
000A00036A80  0001004A0F22 (000A00001AB9, 0007FFFFCC20, 000000000030, 7FFCC6565760) ssh.exe+0xA0F22
0007FFFFCD30  7FFCC63780A1 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x80A1
0007FFFFFFF0  7FFCC6375C86 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x5C86
0007FFFFFFF0  7FFCC6375D34 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x5D34
End of stack trace
Loaded modules:
000100400000 ssh.exe
7FFD3DC70000 ntdll.dll
7FFD3C750000 KERNEL32.DLL
7FFD3B5C0000 KERNELBASE.dll
0003FFCE0000 cygcrypto-1.1.dll
0003FEF70000 cyggssapi_krb5-2.dll
0003FF280000 cygz.dll
0003FFC30000 cyggcc_s-seh-1.dll
0003FEE60000 cygkrb5-3.dll
0003FEF30000 cygk5crypto-3.dll
0003FEE40000 cygkrb5support-0.dll
0003FF090000 cygcom_err-2.dll
0003FFA10000 cygintl-8.dll
7FFCC6370000 cygwin1.dll
0003FFA30000 cygiconv-2.dll
7FFD3BC70000 advapi32.dll
7FFD3C220000 msvcrt.dll
7FFD3D2E0000 sechost.dll
7FFD3C820000 RPCRT4.dll
7FFD3A8D0000 CRYPTBASE.DLL
7FFD3B1D0000 bcryptPrimitives.dll
7FFD309B0000 netapi32.dll
7FFD31290000 SAMCLI.DLL
7FFD3B0B0000 ucrtbase.dll
7FFD31260000 SAMLIB.dll
7FFD39BF0000 NETUTILS.DLL
7FFD3C940000 user32.dll
7FFD3B3B0000 win32u.dll
7FFD3D720000 GDI32.dll
7FFD3B960000 gdi32full.dll
7FFD3B250000 msvcp_win.dll
7FFD3D750000 IMM32.DLL
7FFD3BBF0000 ws2_32.dll
7FFCFA300000 napinsp.dll
7FFCF9500000 pnrpnsp.dll
7FFD3A660000 mswsock.dll
7FFD39CE0000 DNSAPI.dll
7FFD39C70000 IPHLPAPI.DLL
7FFD3C150000 NSI.dll
7FFCF94E0000 winrnr.dll
7FFD34800000 wshbth.dll
7FFCF94B0000 nlansp_c.dll
7FFD31AD0000 dhcpcsvc6.DLL
7FFD31AB0000 dhcpcsvc.DLL
7FFD38430000 WINNSI.DLL
7FFD2D420000 rasadhlp.dll
7FFD31900000 fwpuclnt.dll
7FFD153D0000 libykcs11.dll
7FFD2ADE0000 VCRUNTIME140.dll
7FFCBEA20000 libcrypto-1_1-x64.dll
7FFD3B450000 CRYPT32.dll
7FFD3AA70000 bcrypt.dll
7FFD154A0000 libykpiv.dll
7FFD20930000 WinSCard.dll

The version of the ssh from Cygwin used is:

> C:\cygwin64\bin\ssh.exe -V
OpenSSH_9.2p1, OpenSSL 1.1.1s  1 Nov 2022

The exact same incantation works as expected when using ssh bundled with windows, i.e.

> C:\Windows\System32\OpenSSH\ssh.exe -I "C:\Program Files\Yubico\Yubico PIV Tool\bin\libykcs11.dll" git@github.com
...

This version is

> C:\Windows\System32\OpenSSH\ssh.exe -V
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

Any ideas whether it's an issue with how libykcs11.dll is built or with Cygwin's ssh.exe?

Thanks

[qpernil]

One thing that springs to mind is the issue Windows has with different runtime libraries in different dll:s within a single process. This could easily lead to crashes like we see here. The latest YubicoPIV Tool is linked against the VS 2019 dll runtime.

[JasonHK]

Looks like this is the reason why my git clone crashed

[dzhus]

@JasonHK yeah I also ran into this when using git originally (see git-for-windows/git#4296 in case you need a workaround).

@qpernil

The latest YubicoPIV Tool is linked against the VS 2019 dll runtime.

Thanks that's helpful info. I didn't get around to asking about this issue with cygwin yet. Would it help to verify if Cygwin git is linked against a different version of a particular dll?

[qpernil]

Im reasonably certain that it is. They wouldn't be using Visual Studio Runtime. A way to check dependensies is by using the utility called depends.exe, you will easily find it by googling. I'd point out that is pretty dated by now, so it throws some warnings when run on recent Windows version.

[qpernil]

I'd say that the proper solution for this case would be to build yubico-piv-tool under cygwin, and then use that binary. Unfortunately we don't officially support that. If you do get that to work and contribute it to the projekt it would be greatly appreciated.