Bypass color choice for channelId==0 (floating point exceptions in check.exe for ggttg, gqttq, nobm_pp_ttW)

[valassi]

This is a followup of #733 and PR #706.

I have now added to all processes via CODEGEN the option to enable FPEs in the check.exe executable. I have rerun this for my usual 78 tests. I find 8 failures out for this, for ggttg (float/mixed) and gqttq

tput/logs_ggttg_mad/log_ggttg_mad_f_inl0_hrd0_bridge.txt:Floating Point Exception (CPU)
tput/logs_ggttg_mad/log_ggttg_mad_f_inl0_hrd0.txt:Floating Point Exception (CPU)
tput/logs_ggttg_mad/log_ggttg_mad_f_inl0_hrd1.txt:Floating Point Exception (CPU)
tput/logs_ggttg_mad/log_ggttg_mad_m_inl0_hrd0.txt:Floating Point Exception (CPU)
tput/logs_ggttg_mad/log_ggttg_mad_m_inl0_hrd1.txt:Floating Point Exception (CPU)
tput/logs_gqttq_mad/log_gqttq_mad_f_inl0_hrd0_bridge.txt:Floating Point Exception (CPU)
tput/logs_gqttq_mad/log_gqttq_mad_f_inl0_hrd0.txt:Floating Point Exception (CPU)
tput/logs_gqttq_mad/log_gqttq_mad_f_inl0_hrd1.txt:Floating Point Exception (CPU)

The fact that this happens for the mixed mode makes me suspect that this is in the color algebra, for ggttg?

On the other hand for gqttq this is maybe in the Feynman diagrams, as it only happens in f and not in m?

Note: the madevent 'tmad' tests for these processes conversely have run ok for me so far. Maybe just because I have not tried enough events? By default FPEs are normally enabled in madevent (by linking fortran and c++). Here the difference is that I have explicitly enabled FPEs in the c++-only check.exe executables.

[valassi]

I am pinning this issue as high priority. There are FPEs in check.exe for ggttg and gqttq.

I have instead closed #733 for FPEs in madevent in nobm_pp_ttW.

Note however that there are ALSO still FPEs in check.exe in nobm_pp_ttW. I will follow these up here and also in PR #706.

[valassi]

tput/logs_gqttq_mad/log_gqttq_mad_f_inl0_hrd0.txt:Floating Point Exception (CPU)

I have checked this again. Note that these FPEs only happen in very special corners now. For instance the one above gives

cd epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux> 

make cleanall; AVX=512y FPTYPE=f make -j
./check.exe  -p 1 16 1
[all OK!]

make cleanall; AVX=512z FPTYPE=f make -j
CUDACPP_RUNTIME_ENABLEFPE=1 ./check.exe  -p 1 16 1
[crashes!]

So: this one only crashes for float (not double and not mixed) and only for 512z (not for 512y)

[valassi]

Hi @roiser I added the comment above for you.

Via email I got a notification that you had sent a comment here asking if this was fixed? But I do not see it in the web interface. (You deleted it or there was a problem in email-github interface?)

Anyway. The issue is still pending at least in some cases. I guess it will need to be debugged one by one.

I will try to add this to the testsuite in my PR #792

[valassi]

I have implemented this in the extended CI, but the issue does not show because the github default nodes do not have AVX512...
https://github.com/valassi/madgraph4gpu/actions/runs/6802025406/job/18494211030

Execute build.avx2_f_inl0_hrd0/gcheck.exe -p 1 32 1
(SKIP missing build.avx2_f_inl0_hrd0/gcheck.exe)

(SKIP 512y which is not supported - no avx512vl in /proc/cpuinfo)

(SKIP 512z which is not supported - no avx512vl in /proc/cpuinfo)

We will need to run these tests on our own nodes with GPU and AVX512

[valassi]

I believe I have found the source and a fix for (at least part of?) the floating point exceptions here. The fix is committed as part of PR #706.

I have debugged this as follows

    cd nobm_pp_ttW.mad/SubProcesses/P1_gd_ttxwmu
    make -f cudacpp.mk debug -j
    CUDACPP_RUNTIME_ENABLEFPE=on gdb --args ./check.exe -p 1 8 1
    (gdb) run
    Starting program: /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/nobm_pp_ttW.mad/SubProcesses/P1_gd_ttxwmu/check.exe -p 1 8 1
    ...
    Program received signal SIGFPE, Arithmetic exception.
    mg5amcCpu::_ZN9mg5amcCpu8sigmaKinEPKdS1_S1_S1_PdjS2_S2_PiS3_i._omp_fn.0(void) () at CPPProcess.cc:1250
    1250              const bool okcol = allrndcol[ievt] < ( targetamp[icolC][ieppV] / targetamp[ncolor - 1][ieppV] );
    Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-60.el9.x86_64 libgcc-11.3.1-4.3.el9.alma.x86_64 libgomp-11.3.1-4.3.el9.alma.x86_64 libstdc++-11.3.1-4.3.el9.alma.x86_64
    (gdb) p channelIdC
    $1 = -1

Essentially the problem is that line 1250 gives a division 0/0 because it should not be executed at all to start with! This belongs to event-by-event color choice, which should be skipped when channelId==0 and it was not. The fix simpoly consists in skipping the event-by-event color choice if channelId==0. I checked that this does bypass and fix the FPEs at least for a few cases like nobm_pp_ttW and gg_ttg. I am now rerunning more complete coverage tests.

(PS I have no idea why FPEs happen only for come combinations of processes, fp precision and AVX architecture... I have no idea, I guess that those that do not fail are "lucky").

[valassi]

Note, this sounds very much related to what @nscottnichols reported in #611, There is a channelIdC that becomes -1 and this line should not be executed.

[valassi]

Note also that bypassing the color choice is needed in cases such as reweighting #607