implement new CI and fixed the one related to the compilation issue. #980

oliviermattelaer · 2024-08-30T13:22:41Z

Hi,

This PR is basically the same as #865 but not targetting master but testsuite_only.
(and with a lot of cherry-pick/squashing from the original branch), so #865 is just the sum of this PR and the one related to testsuite_only.

Note that this PR need a change to the upstream madgraph (which should be urgently approved) and the associated PR is
mg5amcnlo/mg5amcnlo#132

… and put ee_mumua to the sde=1 cross-section

valassi · 2024-08-30T16:26:26Z

Hi @oliviermattelaer thanks so far looks good.

One 'minor' point, I would probably prefer (need to cross check) to split the clean logic in Source from

clean:
	$(RM) *.o $(LIBRARIES) $(BINARIES)
	cd PDF; make clean; cd ..
	cd PDF/gammaUPC; make clean; cd ../../
	cd DHELAS; make clean; cd ..
	cd CERNLIB; make clean; cd ..
	cd MODEL; make clean; cd ..
	if [ -d RUNNING ]; then cd RUNNING; make clean; cd ..; fi
	cd BIAS/dummy; make clean; cd ../..
	cd BIAS/ptj_bias; make clean; cd ../..
	if [ -d $(CUTTOOLSDIR) ]; then cd $(CUTTOOLSDIR); make clean; cd ..; fi
	if [ -d $(IREGIDIR) ]; then cd $(IREGIDIR); make clean; cd ..; fi
	for i in `ls -d ../SubProcesses/P*`; do cd $$i; make clean; cd -; done;

to the following (what I have now in master through patches)

cleanSource:
	$(RM) *.o $(LIBRARIES) $(BINARIES)
	cd PDF; make clean; cd ..
	cd PDF/gammaUPC; make clean; cd ../../
	cd DHELAS; make clean; cd ..
	cd CERNLIB; make clean; cd ..
	cd MODEL; make clean; cd ..
	if [ -d RUNNING ]; then cd RUNNING; make clean; cd ..; fi
	cd BIAS/dummy; make clean; cd ../..
	cd BIAS/ptj_bias; make clean; cd ../..
	if [ -d $(CUTTOOLSDIR) ]; then cd $(CUTTOOLSDIR); make clean; cd ..; fi
	if [ -d $(IREGIDIR) ]; then cd $(IREGIDIR); make clean; cd ..; fi

clean: cleanSource
	for i in `ls -d ../SubProcesses/P*`; do cd $$i; make clean; cd -; done;

The motivation is that this allows faster make cleanall of cudacpp: by calling cleanSource I ONLY clean Source, then the more complex cleaning looping through P* I do elsewhere (I need to delete the build.* directories).

I can do that no problem, I propose

I would change the default Source makefile making the split above (might look useless, but it is harmless, make clean does the same)
I would add some comments to explain what the different clean commands do
I would then fix the rest using your nice mechanism
By the way, I would then prefer to actually add the clean at the end rather than in the middle, so I can add one dictionary entry 'append_to_makefile' or something instead of using `additional_dependencies'

I will let you know when done

valassi · 2024-08-31T06:51:52Z

The motivation is that this allows faster make cleanall of cudacpp: by calling cleanSource I ONLY clean Source, then the more complex cleaning looping through P* I do elsewhere (I need to delete the build.* directories).

I did a simple test to confirm this, on gq_ttq which is the simplest process with more than one P* in the repo.

Using upstream/master (i.e. what I would lik to go back to)

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> make cleanall | \grep Entering 
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/DHELAS'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/CERNLIB'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/MODEL'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/dummy'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/ptj_bias'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'

Using instead what is currently in this PR

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> make cleanall | \grep Entering 
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/DHELAS'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/CERNLIB'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/MODEL'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/dummy'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/ptj_bias'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'

The difference is

11,12d10
< make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
< make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'

In a project with many P*, where looping through P* is slow, this is an issue. I will go back to the upstream/master logic (with the new codegen however)

…ll' instead of 'make cleanavx' to clean the build (these are identical)

… up P* cleanup; remove cleanavx which is identical to cleanall (launch_plugin.py has been changed accordingly)

…ource+clean and add an %(additional_clean) section This is needed to speed up the cleanup of P* subprocesses in cudacpp See madgraph5/madgraph4gpu#980

…efile, split clean into cleanSource+clean and move cleanavxs at the end via %(additional_clean)

…k cleanSource to speed up P* cleanup Also remove cleanavx which is identical to cleanall (launch_plugin.py has been changed accordingly) Add some comments in Olivier's write_source_makefile

…l commit message later (regenerating the patch changes nothing)

…g changes: I just want to mark that Source/makefile is no longer there) The only files that still need to be patched are - 2 in patch.common: Source/genps.inc, SubProcesses/makefile - 3 in patch.P1: auto_dsig1.f, driver.f, matrix1.f ./CODEGEN/generateAndCompare.sh gg_tt --mad --nopatch git diff --no-ext-diff -R gg_tt.mad/Source/genps.inc gg_tt.mad/SubProcesses/makefile > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/SubProcesses/P1_gg_ttx/auto_dsig1.f gg_tt.mad/SubProcesses/P1_gg_ttx/driver.f gg_tt.mad/SubProcesses/P1_gg_ttx/matrix1.f > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.P1 git checkout gg_tt.mad

…979 and #980

valassi

Approve, but we need also #984

…ource+clean and add an %(additional_clean) section This is needed to speed up the cleanup of P* subprocesses in cudacpp See madgraph5/madgraph4gpu#980

…_only_fixed

valassi · 2024-09-01T07:34:47Z

Hi @oliviermattelaer some of your confusion in #984 (comment) probably came from the fact that this testsuite_only_fixed did not include the latest upstream/master.

I have now merged master into testsuite_only, and I have also merged testsuite_only into this testsuite_only_fixed.

…_fixci): fix my bug, define additional_clean in the second place where it is needed in export_v4.py

…'make cleanavxs' target as suggested by Olivier

Improvements to complete PRs 979 and 980

valassi · 2024-09-01T19:57:43Z

I have now merged #984 into this PR as discussed with Olivier.

After the CI runs I will merge 979 and this 980 into master.

valassi · 2024-09-01T20:05:28Z

Ok the CI is as expected, only the three pptt012j tests fail.

I will merge #979 and then this 980.

valassi · 2024-09-01T20:07:10Z

I have merged #979 into master.

Now merging into master this #980 which also includes #984.

…, madgraph5#980, madgraph5#984 patches for the new CI and Source/makefile) into june24 Fix conflicts: - MG5aMC/mg5amcnlo (keep the current june24 version 4ef15cab1 i.e. current valassi_gpucpp_june24) - epochX/cudacpp/gg_tt.mad/bin/internal/banner.py (keep a debug printout)

…Source/makefile madgraph5#980) into helas

…Source/makefile madgraph5#980) into prof (Checked that regenerating gg_tt.mad is all ok)

…Source/makefile madgraph5#980) into grid

…adgraph5#980) into cmsdy Fix conflicts: - epochX/cudacpp/CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common (remove Source/makefile) - epochX/cudacpp/CODEGEN/allGenerateAndCompare.sh (add processes from both branches) (Checked that regenerating gg_tt.mad is ok)

…rce/makefile madgraph5#980) into cmsdyps Fix conflicts: - epochX/cudacpp/CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common (remove Source/makefile) (NB regenerating gg_tt.mad is not ok: the newranmar.o is now missing)

oliviermattelaer added 4 commits August 30, 2024 10:36

PR where I only add CI but not trying to fix them

b3123d0

fix wrong name/cross-section for ggtt (the check was pptt originally)…

5a0527d

… and put ee_mumua to the sde=1 cross-section

fix issue when changing the vector_size

cb8416a

set the mg5 branch to be fpucpp_fixci

4c6d84e

oliviermattelaer changed the base branch from testsuite_only to master August 30, 2024 15:09

oliviermattelaer requested a review from a team as a code owner August 30, 2024 15:09

valassi added 2 commits August 31, 2024 08:43

Merge remote-tracking branch 'upstream/testsuite_only_fixed' into gpucpp

a8ff342

[gpucpp] regenerate gq_ttq.mad using the current proposal for PR #980

d28f85e

valassi added 2 commits August 31, 2024 09:11

[gpucpp] in gq_ttq.mad and CODEGEN launch_plugin.py, use 'make cleana…

6da7115

…ll' instead of 'make cleanavx' to clean the build (these are identical)

[gpucpp] in gq_ttq.mad Source/makefile, add back cleanSource to speed…

aad68d5

… up P* cleanup; remove cleanavx which is identical to cleanall (launch_plugin.py has been changed accordingly)

valassi mentioned this pull request Aug 31, 2024

Fixes for Source makefile (on top of Olivier's CI fixes) mg5amcnlo/mg5amcnlo#135

Merged

valassi added 7 commits August 31, 2024 20:29

[gpucpp] upgrade mg5amcnlo from a3ccfd2bf to 185cc4c9c: in Source mak…

44e4f7d

…efile, split clean into cleanSource+clean and move cleanavxs at the end via %(additional_clean)

[gpucpp] in CODEGEN backport Source/makefile from gq_ttq.mad: add bac…

6ff1878

…k cleanSource to speed up P* cleanup Also remove cleanavx which is identical to cleanall (launch_plugin.py has been changed accordingly) Add some comments in Olivier's write_source_makefile

[gpucpp] regenerate gq_ttq.mad - heck that all is ok

326ce7e

[gpucpp] regenerate gg_tt.mad - so that I can regenerate patches from it

5aa2b44

[gpucpp] in CODEGEN dummy change to patch.common to allow a meaningfu…

c79843a

…l commit message later (regenerating the patch changes nothing)

[gpucpp] regenerate all processes - this completes the add-ons to PR #…

807f481

…979 and #980

valassi mentioned this pull request Aug 31, 2024

Improvements to complete PRs 979 and 980 #984

Merged

valassi approved these changes Aug 31, 2024

View reviewed changes

valassi mentioned this pull request Aug 31, 2024

PR where I only add CI but not trying to fix them #979

Merged

valassi added 2 commits September 1, 2024 09:28

Merge remote-tracking branch 'upstream/master' into testsuite_only

4cf1721

Merge remote-tracking branch 'upstream/testsuite_only' into testsuite…

dedbb15

…_only_fixed

valassi added 2 commits September 1, 2024 09:35

Merge remote-tracking branch 'upstream/testsuite_only_fixed' into gpucpp

57c72a4

[gpucpp] upgrade mg5amcnlo from 185cc4c9c to fa706c696 (latest gpucpp…

93ef4ba

…_fixci): fix my bug, define additional_clean in the second place where it is needed in export_v4.py

valassi and others added 4 commits September 1, 2024 18:32

[gpucpp] in gq_ttq.mad and CODEGEN for Source/makefile, add back the …

bc12fdc

…'make cleanavxs' target as suggested by Olivier

[gpucpp] regenerate gq_ttq.mad, check that all is ok

f1db0b7

[gpucpp] ** COMPLETE GPUCPP ** regenerate all processes

4776d29

Merge pull request #984 from valassi/gpucpp

c153482

Improvements to complete PRs 979 and 980

valassi merged commit c4c6e13 into master Sep 1, 2024
170 of 173 checks passed

valassi mentioned this pull request Sep 2, 2024

add file for the test-suite of MG5aMC and a ci related to it #865

Closed

valassi added a commit to valassi/madgraph4gpu that referenced this pull request Sep 2, 2024

Merge remote-tracking branch 'upstream/master' (including new CI and …

062527c

…Source/makefile madgraph5#980) into helas

valassi added a commit to valassi/madgraph4gpu that referenced this pull request Sep 2, 2024

Merge remote-tracking branch 'upstream/master' (including new CI and …

eb7e826

…Source/makefile madgraph5#980) into prof (Checked that regenerating gg_tt.mad is all ok)

valassi added a commit to valassi/madgraph4gpu that referenced this pull request Sep 2, 2024

Merge remote-tracking branch 'upstream/master' (including new CI and …

271afaf

…Source/makefile madgraph5#980) into grid

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement new CI and fixed the one related to the compilation issue. #980

implement new CI and fixed the one related to the compilation issue. #980

oliviermattelaer commented Aug 30, 2024

valassi commented Aug 30, 2024 •

edited

Loading

valassi commented Aug 31, 2024

valassi left a comment

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

implement new CI and fixed the one related to the compilation issue. #980

implement new CI and fixed the one related to the compilation issue. #980

Conversation

oliviermattelaer commented Aug 30, 2024

valassi commented Aug 30, 2024 • edited Loading

valassi commented Aug 31, 2024

valassi left a comment

Choose a reason for hiding this comment

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

valassi commented Sep 1, 2024

valassi commented Aug 30, 2024 •

edited

Loading