CMS generator cards for 2015 files #97

katilp · 2021-05-19T15:15:53Z

(edited 11.10.2021)

The generator "gridpacks" are stored in /cvmfs/cms.cern.ch/phys_generator/gridpacks/
However, note that not all the LHE cards 2015 are stored yet there.

Take an example dataset from https://github.com/cernopendata/data-curation/blob/master/cms-YYYY-simulated-datasets/inputs/CMS-2015-mc-datasets.txt

Find the generator cards "by-hand" with:

Case no LHE:

Three options:

Through "fragments" stored in McM

Advantage: gets directly the relevant information

search the dataset in McM (request -> output dataset) (example query MiniAODSIM)
find the parent name and redo 1.
- example query AODSIM
- example query GEN-SIM
if GEN-SIM, find the generator parameters in "Name of fragment"
- example: https://raw.githubusercontent.com/cms-sw/genproductions/f63476b2fdbd3b05dffe4ac3cd8edfefefbe0d88/python/ThirteenTeV/sherpa_ADDGravitonToDiPhoton_MS3000_NED2_KK4_M-2000To3000_13TeV-sherpa_MASTER_cff.py
- in the dictionary: "name_of_fragment": "Configuration/GenProduction/python/ThirteenTeV/sherpa_ADDGravitonToDiPhoton_MS3000_NED2_KK4_M-2000To3000_13TeV-sherpa_MASTER_cff.py" and also "fragment_tag" which is "f6347..." and the URL can be constructed from both

As the metadata script reads the full dictionary we should have this information already

GEN-SIM access https://github.com/cernopendata/data-curation/blob/master/cms-YYYY-simulated-datasets/code/mcm_store.py#L50
gen_fragment_url may require updating https://github.com/cernopendata/data-curation/blob/master/cms-YYYY-simulated-datasets/code/mcm_store.py#L189 if all gen fragments available in "name of fragment"
NB: name of fragment gives Configuration/GenProduction/python/ and link has /cms-sw/genproductions/f63476b2fdbd3b05dffe4ac3cd8edfefefbe0d88/python/ i.e refers to a specific commit
at the time of 2015, these files were in cmssw and not in cms-sw https://github.com/cms-sw/cmssw/tree/CMSSW_7_1_X/Configuration/Generator/python

Config files

Advantage: already available as config for GEN-SIM step

Disadvantage: shows the full config file, not only the cards

The steps 1,2 as above, then

through "config id" to https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/c4d03e96b97e6067c54800f67781f8a7/configFile
hash is also in the dictionary "config_id": ["c4d03e96b97e6067c54800f67781f8a7"]

From edmProvDump

Advantage: get the information directly from the file

Disadvantage: to be done in a CMSSW release area, formatting not the best for the display

Find a single root file name $file
In a CMSSW release area (e.g. .../CMSSW_7_6_7/src) and after cmsenv:
edmProvDump -f "generator SIM" root://eospublic.cern.ch/$file | grep -A9999 "generator SIM"

Case LHE:

Case no gridpack

MINIAODSIM has mcdb_id > 1 in McM
in use before gridpack was adopted

Example McM query for a MINIAODSIM
Take "Mcdb id", in the dictionary: "mcdb_id": 15839
Find the lhe file in /eos/cms/store/lhe/$mcdb_id
Extract file and read the header with

xz -d -c /eos/cms/store/lhe/$mcdb_id/* > lhe.lhe
  awk '/<header>/,/<\/header>/' lhe.lhe > lhe_header

Case gridpack

mcdb_1 = 0
Find the gridpack address, two options

From McM dictionary

At the LHE level example query
find the gridpack address
- in https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_fragment/B2G-RunIIWinter15wmLHE-00372/0
  - with /cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/madgraph/V5_2.2.2/BBbarDM/DMPseudo_bbbar01j_mphi_10000_mchi_10_gSM_1p0_gDM_1p0_tarball.tar.xz
- or in the dictionary as "fragment": "import FWCore.ParameterSet.Config as cms\n\nexternalLHEProducer = cms.EDProducer(\"ExternalLHEProducer\",\n args = cms.vstring('/cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/madgraph/V5_2.2.2/BBbarDM/DMPseudo_bbbar01j_mphi_10000_mchi_10_gSM_1p0_gDM_1p0_tarball.tar.xz'),\n nEvents = cms.untracked.uint32(5000),\n numberOfParameters = cms.uint32(1),\n outputFile = cms.string('cmsgrid_final.lhe'),\n scriptName = cms.FileInPath('GeneratorInterface/LHEInterface/data/run_generic_tarball_cvmfs.sh')\n)\n\n#Link to datacards:\n#https://github.com/cms-sw/genproductions/tree/427b801b62574119ef1b48d8c48153f167c3c9b3/bin/MadGraph5_aMCatNLO/cards/production/13TeV/DarkMatter/DMPseudo_bbar/DMPseudo_bbbar01j_mphi_10000_mchi_10_gSM_1p0_gDM_1p0",
- NB the link to datacards in the comment see CMS generator cards as "notes" in MC 2015 metadata #96 https://github.com/cms-sw/genproductions/tree/427b801b62574119ef1b48d8c48153f167c3c9b3/bin/MadGraph5_aMCatNLO/cards/production/13TeV/DarkMatter/DMPseudo_bbar/DMPseudo_bbbar01j_mphi_10000_mchi_10_gSM_1p0_gDM_1p0

From edmProvDump

Find a single root file name $file
In a CMSSW release area (e.g. .../CMSSW_7_6_7/src) and after cmsenv:

edmProvDump -f "externalLHEProducer LHE" root://eospublic.cern.ch/$file | grep gridpacks > line
gp=$(sed "s/'/ /g" line | awk '{print $6}')

Extract cards once $gp address is know

if [[ $file == *"madgraph"* ]]; then
      tar -xf $gp ./process/madevent/Cards/run_card.dat
      tar -xf $gp ./process/madevent/Cards/proc_card*.dat
      tar -xf $gp ./process/madevent/Cards/param_card.dat
      mv ./process/madevent/Cards/*.dat $dir/
 elif [[ $file == *"powheg"* ]]; then
      tar -xf $gp *.input
      mv *.input $dir
 elif [[ $file == *"amcatnlo"* ]]; then
      tar -xf $gp process/Cards/run_card.dat
      tar -xf $gp process/Cards/proc_card*.dat
      tar -xf $gp process/Cards/param_card.dat
      mv process/Cards/*.dat $dir/
fi

The text was updated successfully, but these errors were encountered:

katilp · 2021-10-25T09:22:11Z

@OsamaMomani as we discussed, this could be done separated from the other script, and in this case, the pseudocode is

input: list of datasets

find the datasets which have the LHE step (the ones without it should already have generator parameters)
if mcdb_id > 0:
- get the generator parameters from the "LHE header" and name it according to the recid with
```
xz -d -c /eos/cms/store/lhe/$mcdb_id/* > lhe.lhe
awk '/<header>/,/<\/header>/' lhe.lhe > $recid_lhe_header
```
or with similar
- write the header to a generator parameter store
else:
- get the gridpack address (it is a string starting with /cvmfs/cms.cern.ch/phys_generator/gridpacks/) from the fragment or from the dictionary
- write the whole gridpack in a gridpack store (add the recid to the name, or have them as folders)
- get the generator parameters from the gridpack tar file with the codesnippet at the end of the previous comment ($file can be the dataset name)
- save the generator parameter files to files starting with recid and record them the generator parameter store

katilp · 2021-11-21T21:10:42Z

Exceptions/additions:

mcdb_id, but no `<header>`

eg: /GluGluToContinToZZTo2e2nu_13TeV_MCFM701_pythia8/RunIIWinter15pLHE-MCRUN2_71_V1-v1/LHE -> mcdb_id = 14301

$ ls /eos/cms/store/lhe/15401/
gg_ZZ_2El2Nu_13TeV_NNPDF30_lo_as_0130_MCFM70.lhe
-bash-4.2$ head /eos/cms/store/lhe/15401/gg_ZZ_2El2Nu_13TeV_NNPDF30_lo_as_0130_MCFM70.lhe
<LesHouchesEvents version="1.0">
<!--
file generated with MCFM version 7.0
Input file input.DAT contained:
#  Cross-section is:             14.9316     +/-            0.136621E-01)

 #  Contribution from parton sub-processes:
#         GG     |        0.0000        0.00%
#         GQ     |        0.0000        0.00%
#         GQB    |        0.0000        0.00%

take what is in the comment, i.e. between 
keyword for identification "generators": ["MCFM701"]

swp, swo files

/GluGluWWTo2L2Nu_MCFM_13TeV/RunIIWinter15pLHE-MCRUN2_71_V1-v1/LHE

$ ls /eos/cms/store/lhe/15275
ggWWbx_lord_NNPDF30_proc127_ll_500kevents.lhe  ggWWbx_l.swo  ggWWbx_l.swp

to be investigated

no files in `./process/madevent/Cards` for madgraph

/BulkGravTohhTohtatahbb_narrow_M-1000_13TeV-madgraph/RunIIWinter15wmLHE-MCRUN2_71_V1-v1/LHE

/cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/madgraph/V5_2.2.2/exo_diboson/Spin-2/BulkGraviton_hh_htatahbb/narrow/v3/BulkGraviton_hh_htatahbb_narrow_M1000_tarball.tar.xz

./process/madevent/Cards/run_card.dat: Not found in archive

Remove ./ and use tar -xf $gp process/madevent/Cards/run_card.dat etc

no powheg.input for powheg

/GluGluHToBB_M125_13TeV_powheg_pythia8/RunIIWinter15wmLHE-MCRUN2_71_V1-v1/LHE

/cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/powheg/V2/gg_H_quark-mass-effects_NNPDF30_13TeV_M125/v2/gg_H_quark-mass-effects_NNPDF30_13TeV_M125_tarball.tar.gz

$ tar -tf $gp | grep powheg.input
./powheg.input

Take ./powheg.input instead of powheg.input

corrupt compress?/timeout?

/RSGravToWW_width0p2_M-3000_13TeV-madgraph/RunIIWinter15wmLHE-MCRUN2_71_V1-v1/LHE

/cvmfs/cms.cern.ch/phys_generator/gridpacks/slc6_amd64_gcc481/13TeV/madgraph/V5_2.3.3/exo_diboson/Spin-2/RSGraviton_WW_inclu/wide/v1/RSGraviton_WW_inclu_width0.2_M3000_tarball.tar.xz

Files are there and get extracted but tar fails in exit. If I kill the process before it fails:

$ tar -xf $gp ./process/madevent/Cards/run_card.dat
^C
$ ls process/madevent/Cards/run_card.dat
process/madevent/Cards/run_card.dat
$ head process/madevent/Cards/run_card.dat
#*********************************************************************
#                       MadGraph5_aMC@NLO                            *
#                                                                    *
#                     run_card.dat MadEvent                          *
#                                                                    *
#  This file is used to set the parameters of the run.               *
#                                                                    *
#  Some notation/conventions:                                        *
#                                                                    *
#   Lines starting with a '# ' are info or comments                  *

Multiple lhe files

e.g.

$ ls /eos/cms/store/lhe/15453/
Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_0.lhe   Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_2.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_6.lhe
Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_10.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_3.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_7.lhe
Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_11.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_4.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_8.lhe
Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_1.lhe   Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_5.lhe  Higgs0L1ToWWTo2L2Nu_M-125_13TeV-powheg2-JHUgenV6_9.lhe

take ..._0.lhe

OsamaMomani · 2021-11-25T05:38:19Z

folder structure

lhe_generators
│
├── mcdb
│      ├── {mcdb_id}_header (files)
│        .....
├── gridpacks
        ├── {recid} (folders)
               ├── (one or more .dat|.input files)
         .....

~3478 mcdb files (tens of MB)
~3636 gridpacks folders (few hundreds of MB)

in opendata portal

check if record has lhe then check mcdb or gridpacks
- if mcdb provide a link to /eos/..../lhe_generators/mcdb/{mcdb_id}_header
- if gridpacks provide links to files inside /eos/..../lhe_generators/gridpacks/{recid}

katilp mentioned this issue Oct 11, 2021

CMS: generator parameters #106

Closed

katilp mentioned this issue Oct 20, 2021

records: Add cms-simulated-datasets-2015 cernopendata/opendata.cern.ch#3161

Merged

This was referenced Oct 31, 2021

CMS generator cards as "notes" in MC 2015 metadata #96

Closed

CMS: 2015 simulated datasets with a common LHE-GEN step #118

Closed

katilp mentioned this issue Dec 7, 2021

CMS: 2015 data release checklist #65

Closed

17 tasks

katilp closed this as completed Dec 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMS generator cards for 2015 files #97

CMS generator cards for 2015 files #97

katilp commented May 19, 2021 •

edited

Loading

katilp commented Oct 25, 2021 •

edited

Loading

katilp commented Nov 21, 2021 •

edited

Loading

OsamaMomani commented Nov 25, 2021

CMS generator cards for 2015 files #97

CMS generator cards for 2015 files #97

Comments

katilp commented May 19, 2021 • edited Loading

Case no LHE:

Through "fragments" stored in McM

Config files

From edmProvDump

Case LHE:

Case no gridpack

Case gridpack

From McM dictionary

From edmProvDump

Extract cards once $gp address is know

katilp commented Oct 25, 2021 • edited Loading

katilp commented Nov 21, 2021 • edited Loading

mcdb_id, but no <header>

swp, swo files

no files in ./process/madevent/Cards for madgraph

no powheg.input for powheg

corrupt compress?/timeout?

Multiple lhe files

OsamaMomani commented Nov 25, 2021

folder structure

in opendata portal

katilp commented May 19, 2021 •

edited

Loading

katilp commented Oct 25, 2021 •

edited

Loading

katilp commented Nov 21, 2021 •

edited

Loading

mcdb_id, but no `<header>`

no files in `./process/madevent/Cards` for madgraph