math syscalls might behave incorrectly when arithmetic overflow occurs

[greenknot]

On a real device, it seems that some arithmetic syscalls such as cx_math_add truncate results silently when an arithmetic overflow occurs. The speculos implementation should have the same behavior. See

speculos/src/emu_os_bip32.c

Lines 276 to 284 in 8081103

	// cx_math_add(ZR, ZR, kRP, 32) can return invalid result when an overflow
	// occurs, so perform addition on 33 bytes and truncate the result to 32
	// bytes
	a[0] = '\x00';
	memcpy(a + 1, ZR, 32);
	b[0] = '\x00';
	memcpy(b + 1, kRP, 32);
	cx_math_add(a, a, b, 33);
	memcpy(ZR, a + 1, 32);

BTW, the return value of BN_bn2binpad isn't verified in cx_math_add:

speculos/src/emu_cx_math.c

Line 90 in 8081103

BN_bn2binpad(rr, r, len);

while it returns -1 because len is less than BN_num_bytes(rr). Return values of OpenSSL functions are almost never checked. This should be enforced.

[greenknot]

cx_math_add behavior is documented in the SDK: @return carry.