Prepare Makefile to combine MadEvent + Cudacpp plugin

[valassi]

Hi @oliviermattelaer I am having a look at trying to combine MadEvent and the cudacpp plugin.

I am using for instance

generate g g > t t~
output madevent CODEGEN_mad_gg_tt --vector_size=16 --me_exporter=standalone_cudacpp

This is the infrastructure you created that contains BOTH the fortran code and the cudacpp code.

As a reference, I am using the madevent only

generate g g > t t~
output madevent CODEGEN_mad_gg_tt --vector_size=16

Can you please remind me already how to build this? I imagine I need to do

cd Source
make
cd ../SubProcesses/P1_gg_ttx
make

but the first step fails with

../bin/madevent treatcards run
launch in debug mode
...
  File "/data/avalassi/GPU2020/madgraph4gpuX/epochX/cudacpp/gg_tt.madonly/bin/internal/common_run_interface.py", line 3752, in set_configuration
    config_file = open(config_path)
FileNotFoundError: [Errno 2] No such file or directory: '/data/avalassi/GPU2020/MG5aMC/3.1.1_lo_vectorization/input/mg5_configuration.txt'
/cvmfs/sft.cern.ch/lcg/releases/gcc/10.2.0-c44b3/x86_64-centos7/bin/gfortran -w -fPIC  -ffixed-line-length-132  -c -o setrun.o setrun.f
setrun.f:78: Error: Can't open included file 'run_card.inc'
make: *** [setrun.o] Error 1

Thanks
Andrea

[oliviermattelaer]

Which branch are you using? (for the git plugin)
such that I can run the exact same code

[valassi]

Thanks Olivier!
I am using the 3.1.1_lo_vectorization branch.
Andrea

[oliviermattelaer]

So the weird part is why you do not have the file:
/data/avalassi/GPU2020/MG5aMC/3.1.1_lo_vectorization/input/mg5_configuration.txt
Can you do

touch /data/avalassi/GPU2020/MG5aMC/3.1.1_lo_vectorization/input/mg5_configuration.txt

This file can technically be empty so this should fix the issue. (in principle it should be created the first time you run the code such that you can edit it without having to worry about commiting the change)

(my question was on the madgraph4gpu side which branch you use)

[valassi]

Hm wait... there may be something wrong in my scripts... my bad, sorry!

[valassi]

Now it built fine... let m edebug it
Thanks for the help and sorry for the noise!

[valassi]

Hm ok, the problem is that the MG5aMC auto generated directories are not relocatable as-is.

I mean,

diff -r /data/avalassi/GPU2020/MG5aMC/3.1.1_lo_vectorization/CODEGEN_madonly_gg_tt\
          /data/avalassi/GPU2020/madgraph4gpuX/epochX/cudacpp/gg_tt.madonly

where the first is the directpry I created, and the second is a directory to which I copied the previous one. When I build in the first location (the original location), all is ok. When I build in the second location, things are not ok.

Can this be fixed maybe upstream?

Anyway. I will investigate and fix it in my scripts

[valassi]

Ok the problem was that mg5_path in a card file was pointing to the main MG5aMC directory. I had already noticed when preparing gridpacks that this must be commented out. So this is the patch
c702cca

Now Source builds. However the Subprocess/P1 does not build because smatrix1 is not found (looks like matrix1_orig.f is not built). I guess this is also eventually related to helicity recycling #279 where matrix1_optim is used?

[valassi]

Hi @oliviermattelaer I have trouble understanding this, can you explain please?

MATRIX_HEL = $(patsubst %.f,%.o,$(wildcard matrix*_orig.f))
MATRIX = $(patsubst %.f,%.o,$(wildcard matrix*_optim.f))
ifeq ($(strip $(MATRIX_HEL)),)
        MATRIX = $(patsubst %.f,%.o,$(wildcard matrix*.f))
endif

I am not even sure if "strip" there is the Makefile function (https://www.gnu.org/software/make/manual/html_node/Text-Functions.html) or if it is meant to be the linux strip command (https://linux.die.net/man/1/strip), but I imagine it is the first (also because the second woul dneed an extra shell keyword).

There is no directive here for going from the _orig.f file to the _optim.f file (indeed, helicity recycling #279), so I am not sure how this could be meant to build?

I will try "--hel_recycling=False"

[valassi]

Ok I have disabled helicity recycling
02ab07b

Now madevent builds! However I need to find some parameter files to feed to it to get some sensible results.

[valassi]

Trying to understand what madevent accepts as inputs.

The madevent in my case (is it always the case) is fror instance driver.f in
https://github.com/valassi/madgraph4gpu/blob/5cb4b6a0fcecde34f63d06bc07da9faf8c14f49b/epochX/cudacpp/gg_tt.madonly/SubProcesses/P1_gg_ttx/driver.f

The driver looks for some files, for instance using open_file
https://github.com/valassi/madgraph4gpu/blob/5cb4b6a0fcecde34f63d06bc07da9faf8c14f49b/epochX/cudacpp/gg_tt.madonly/Source/open_file.f#L1
This routine uses the Cards directory for some files.

Anyway, it reads dname.mg which is a local file with just the directory name.

open("dname.mg", O_RDWR|O_CLOEXEC)      = 3

Then it looks for results.dat but this does not exist in my directory (use strace and grep .dat)

open("results.dat", O_RDWR|O_CLOEXEC)   = -1 ENOENT (No such file or directory)
open("./G1results.dat", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory)

It also (strace) opens a tmp gfortran file. Maybe this is the 'scratch' file?

open("/tmp/avalassi/gfortrantmpHskNB3", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3

Note that it does NOT seem (strace) to read param_card.dat?

Then (use strace) it reads

open("../../lib/Pdfdata/NNPDF23_lo_as_0130_qed_mem0.grid", O_RDWR|O_CLOEXEC) = 4

Then it prompts for user input.
This is using
https://github.com/valassi/madgraph4gpu/blob/5cb4b6a0fcecde34f63d06bc07da9faf8c14f49b/epochX/cudacpp/gg_tt.madonly/SubProcesses/P1_gg_ttx/driver.f#L209
which does not seem to use cards. So probably it must received piped input.

[valassi]

Ok refine.sh and survey.sh show the use:

madgraph4gpu/epochX/gridpack/29x/gg_tt.auto/madevent/SubProcesses/refine.sh

Line 60 in 34ab2d4

%(Ppath)s/madevent 2>&1 >> $k <input_sg.txt | tee -a $k;

madgraph4gpu/epochX/gridpack/29x/gg_tt.auto/madevent/SubProcesses/refine_splitted.sh

Line 53 in 34ab2d4

../madevent 2>&1 >> $k <input_app.txt | tee -a $k;

madgraph4gpu/epochX/gridpack/29x/gg_tt.auto/madevent/SubProcesses/survey.sh

Line 51 in 34ab2d4

../madevent 2>&1 >> $k <input_app.txt | tee -a $k;

The inputs are input_app.txt and input_sg.txt

[avalassi@itscrd70 gcc10.2/cvmfs] /data/avalassi/GPU2020/madgraph4gpuX/epochX/gridpack/29x/gg_tt.auto> find . -name 'input*'
./madevent/SubProcesses/P1_gg_ttx/input_app.txt
./madevent/SubProcesses/P1_gg_ttx/G2/input_app.txt
./madevent/SubProcesses/P1_gg_ttx/Hel/input_app.txt
./madevent/SubProcesses/P1_gg_ttx/G1/input_app.txt
./madevent/Source/MODEL/input.inc

[oliviermattelaer]

So you have two executable that can be generated for madevent
./madevent
./madevent_forhel

The first is using matrixX_optim.f while the second use matrixX_orig.f

to bypass the helicity recycling you can do a cp
cp matrixX_orig.f matrixX_optim.f
(or just always use madevent_forhel exectuble).

Those executable are a "interactive" script that take input within stdin:
Those inputs are written on disk within the input_app.txt (or similar)
so typically you can run
./madevent_forhel < input.txt

where input.txt is
1000 5 3 !Number of events and max and min iterations
0.1 !Accuracy
2 !Grid Adjustment 0=none, 2=adjust
1 !Multi-channel mode 1=yes
0 !Helicity Sum/event 0=exact
1 !channel of integration

[valassi]

Thanks Olivier! Yes now I see

[avalassi@itscrd70 gcc10.2/cvmfs] /data/avalassi/GPU2020/madgraph4gpuX/epochX/gridpack/29x/gg_tt.auto> more ./madevent/SubProcesses/P1_gg_ttx/G1/input_app.txt
         2000         8           3      !Number of events and max and min iterations
  0.01    !Accuracy
  2       !Grid Adjustment 0=none, 2=adjust
  1       !Suppress Amplitude 1=yes
  0        !Helicity Sum/event 0=exact
        1
[avalassi@itscrd70 gcc10.2/cvmfs] /data/avalassi/GPU2020/madgraph4gpuX/epochX/gridpack/29x/gg_tt.auto> diff ./madevent/SubProcesses/P1_gg_ttx/G1/input_app.txt ./madevent/SubProcesses/P1_gg_ttx/G2/input_app.txt
6c6
<         1
---
>         2

[valassi]

About the executables: I think madevent_forhel did not get built. The built failed, unless I put helicity recycling false. But once I do, it builds ok

[valassi]

I think I am understdning better, the "channel of integration) is 1 in G1, 2 in G2 etc, as many as there are channels (as many as there are diagrams?)

More important: is there a way I can get Grid Adjustement = 0, ie use a fixed grid? I think for performance tests this would probably be the best?

[oliviermattelaer]

For the executable, should be the opposite, you can compile madevent only if hel_recycling is False since in that case.
madgraph does the cp (or symbolic link) directly to have the file matrix_optimX.f

For the channel of integration, yes G1 means 1, .... the number of channels is not exactly the number of diagrams but it is correlated to it (some diagram are not included -- 4 point interaction--, some diagram are integrated together due to symmetry, some diagram are split in multiple piece if the phase-space is really complex -- like h >ww --)

For the grid, yes you can put the grid adjustement to 0 to freeze the vegas layer.

[valassi]

I tried the cards above, which are the same you had given as example

more /data/avalassi/GPU2020/madgraph4gpuX/epochX/gridpack/29x/gg_tt.auto/madevent/SubProcesses/P1_gg_ttx/G1/input_app.txt
         2000         8           3      !Number of events and max and min iterations
  0.01    !Accuracy
  2       !Grid Adjustment 0=none, 2=adjust
  1       !Suppress Amplitude 1=yes
  0        !Helicity Sum/event 0=exact
        1

but it fails eventually?

 DiscreteSampler:: Error in subroutine DS_get_point, dimension 'grouped_processes' not found.

[oliviermattelaer]

so the issue with the discretesampler is because this version requires to set
mc_grouped_subproc to False in the run_card.dat when you set vector_size.
Will see if I can force that automatically (in a way or another)

[valassi]

but it fails eventually?

 DiscreteSampler:: Error in subroutine DS_get_point, dimension 'grouped_processes' not found.

Thanks @oliviermattelaer for the chat! Ok so as we found out, your suggestion to add "False = mc_grouped_subproc" fixes the issue.

In practice:

• generate the code
• append "False = mc_grouped_subproc" to the end of Cards/run_card.dat
• build the code in Source (NB the run_card influences how the code is built)
• build also SubProcesses/P1_gg_ttx
• execute madevent with an input file

[valassi]

About the cards:

more /data/avalassi/GPU2020/madgraph4gpuX/epochX/gridpack/29x/gg_tt.auto/madevent/SubProcesses/P1_gg_ttx/G1/input_app.txt
2000 8 3 !Number of events and max and min iterations
0.01 !Accuracy
2 !Grid Adjustment 0=none, 2=adjust
1 !Suppress Amplitude 1=yes
0 !Helicity Sum/event 0=exact
1

Line by line:

     2000         8           3      !Number of events and max and min iterations

0.01 !Accuracy

The number of events are the events used in the first iteration. Then every iteration doubles the number of events, until either you reach max iterations or you reach the desired accuracy. In order to have an easily reproducible test case, the easiest is to set min iterations equal to max iterations: in that case, always the same number of events will be used, while the accuracy will be ignored.

2 !Grid Adjustment 0=none, 2=adjust

Using option 2 means adaptive integration: there is one first iteration where a grid is created and written to files, then further iterations start from the file. This is one of those ftn26 files, IIUC.

To have an easily reproducible test case for functional and performance tests, the best os probably to use option 0. In this case, if there is no grid file ftn26, some internal defaults are used. Note hoewver that if ft26 does exist, then it will be used.

I think that for our functional/performance tests we do not care about having an efficent sampling algorithm (we use rambo in cudacpp!), so option 0 with no grid file sounds good.

1 !Suppress Amplitude 1=yes

This actually means using "multichannel" (ie I guess the single diagram enhanced algo). We should use 1.

0 !Helicity Sum/event 0=exact

For this one, Olivier suggests that we use 0. IIUC, when we have no helicity recycling the option 0 is used always anyway.

    1

This is the choice of a specific channel. I guess we can use 1 (or we can try once with 1 and once with 2 and do a few internal tests).

PS Note we can then get results like this, useful for tests

 -------------------------------------------------------------------------------
 Accumulated results:   Integral =  0.3984E+00
                        Std dev  =  0.8925E-03
                       Cross sec =  0.3984E+00
             Chi**2 per DoF.     =      2.6763
 -------------------------------------------------------------------------------
 Found         8022  events.
 Wrote         8018  events.
 Actual xsec   0.39836187259476075     
 Correct abs xsec   0.39836187259476075     
 Event xsec   0.39836187259470696     
 Events wgts > 1:            0
 % Cross section > 1:    0.0000000000000000        0.0000000000000000     
-------------------------------------------------
---------------------------
 Results Last   3 iters: Integral =   0.3984E+00
                     Abs integral =   0.3984E+00
                          Std dev =   0.8925E-03
                  Chi**2 per DoF. =      2.7741
-------------------------------------------------
---------------------------
 Status   1.0000000000000000E-002           4           3

[valassi]

A full dump of what we still need in the bridge interface is in #404.

In practice

• we need to modify fbridge.inc (an dthe cudacpp internals) to add a few things such as multichannel multi-channel implementation in cudacpp codegen plugin (pick it up from 311 branch cpp/gpu standalone) #342 , alphas Running of alphas (QCD coupling) #373 , color choice Randomly pick one color for each event written to file (jamp2 handling) #402 , helicity choice Randomly pick one helicity for each event written to file #403 )
• we then need to call these from auto_dsig1.f
• at least initially, we agreed with Olivier that we can simply have two copies of the auto_dsig1.f file, with two different names: one which is used for fortran MEs in madevent ie the default SMATRIX methods, and one which delegates MEs to the bridge

Coming finally to this specific #400 about makefiles:

• we now have two, one lowercase makefile for fortran and one uppercase Makefile for cudacpp (which is ok on linux but may not be ok on mac!...)
• maybe better to rename these (for mac but not only), however the idea of having two is good
• maybe keep the fortran makefile unchanged, an dadd instead the building of a new executable with the new auto_dsig1_cudacpp.f into the cudacpp plugin?

[valassi]

I have added a few comments about this issue of makefiles in #413.

For the moment, madevent and cudacpp source code is mixed, superposed in the same directories. Maybe eventually we can cleanly separate them, but for the moment I would keep them as they are.

In any case, I would keep two separate makefiles. I would probably rename the cudacpp Makefiles as plugin.mk, and add a symlink to Makefile only if it is a standalone installation. Otherwise, for madevent+cudacpp, the plugin.mk would be called by the (lowercase) madevent makefile.

[valassi]

Note that makefile and Makefil clash on case-insensitive systems #480

[valassi]

I am about to merge PR #484, which is mainly about using different names for madevent and cudacpp makefile's, fixing the clash on case-insensitive systems #480.

I renamed the cudacpp Makefile as cudacpp.mk. I then add a symlink of that as makefile (lowercase) so that in standalone mode a 'make' works without the need for '-f'. In madevent+cudacpp mode, the madevent code generation overwrites that makefile symlink (the cudacpp makefile) with its own makefile file (the madevent makefile). The latter includes also a call to cudacpp.mk, which cleans up this #400.

I am closing this issue. Of course it can be cleaned up, and its code generation in upstream MG5aMC has to be done. But all that will be a general issue for much more than just makefiles.