Scouting Offline DQM for 2025 data-taking (EGM+MUO+autoDQM)

[elfontan]

PR description:

A skeleton for a Scouting Offline DQM for the 2025 pp data-taking is here implemented.

The goal is to monitor the distributions of the scouting objects (allowing an easier comparison with offline objects) and the efficiency of the scouting triggers. The targeted data set for the monitoring is ScoutingPFMonitor, which includes both scouting and offline objects. This PR handles the code developed for EGM and MUO (given the similar structure newly implemented) while the developments for JME are currently discussed separately in #47116 (given the strong dependence on the central JME DQM existing code).

Credits to:

• @tihsu99 for the EGM code (Merge MUO and EGM scouting DQM (solve conflicts) elfontan/cmssw#5 + Tihsu dqm cmssw1502 for pr elfontan/cmssw#7)
• @JavierGarciadeCastro and @CeliaFernandez for the MUO code (Offline Scouting Muon DQM elfontan/cmssw#6)

Updates to the autoDQM workflows to be able to centrally run these new analysers only for the ScoutingPFMonitor data set have also been included (DQMOffline/HLTScouting). The goal is to update the T0 configuration (https://github.com/dmwm/T0/blob/master/etc/ProdOfflineConfiguration.py) to run a dedicated @commonWithScouting sequence for ScoutingPFMonitor as this PR will be integrated. [cc: @rseidita]

Description of the current changes:

• MUO DQM: The current code features two plugins in HLTriggerOffline/Scouting/plugins: one to perform a tag and probe (T&P) method to monitor scouting muon and vertex distributions and efficiencies, one to monitor distributions and efficiencies of muon L1 seeds. Corresponding configuration files (called from the central @hltScouting workflow) are placed in HLTriggerOffline/Scouting/python. Configuration files to run the MUO DQM workflow standalone for testing purposes are available in HLTriggerOffline/Scouting/test. Output histograms in the DQM files are currently placed in HLT/ScoutingOffline/Muons.
• EGM DQM: The current code features two plugins in HLTriggerOffline/Scouting/plugins to perform a tag and probe (T&P) method on electrons and to monitor both scouting and offline Gamma objects, allowing for direct comparison. Corresponding configuration files (also used in the central @hltScouting workflow) are placed in HLTriggerOffline/Scouting/python. Output histograms in the DQM files are currently placed in ``HLT/ScoutingOffline/EGamma`.

Relevant presentations about the development and validation of the current code can be found in:

• Scouting meeting on October 3, 2024
• Scouting meeting on January 16, 2024

PR validation:

The PR has been prepared starting from CMSSW_15_0_0_pre2:

cmsrel CMSSW_15_0_0_pre2
cd CMSSW_15_0_0_pre2/src
cmsenv
git cms-init
git cms-addpkg DQMOffline
git cms-addpkg HLTriggerOffline
scram b && scram b code-checks && scram b code-format && scram b

The usual runTheMatrix test workflows were run by the developers. However, none of the central workflows is currently running the new plugins. A dedicated testing workflow has been designed taking inspiration from the central workflows:

[1]: input: RAW file from ScoutingPFMonitor dataset (Run2024F, [Fill9852](https://cmsoms.cern.ch/cms/fills/report?cms_fill=9852), run382686)

[2]: cmsDriver.py step2  --process reHLT -s L1REPACK:Full,HLT:@relval2025 --conditions auto:run3_hlt_relval --data  --eventcontent FEVTDEBUGHLT --datatier FEVTDEBUGHLT --era Run3_2024 -n 500 --filein  file:/eos/cms/store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/Scouting/Run3/ScoutingPFMonitor/300684ed-1a51-474f-8c4f-b3bf1e1f5044_skimmed.root

[3]: cmsDriver.py step3  --conditions auto:run3_data_prompt_relval -s RAW2DIGI,L1Reco,RECO,PAT,NANO,DQM:@hltScouting --datatier RECO,MINIAOD,NANOAOD,DQMIO --eventcontent RECO,MINIAOD,NANOEDMAOD,DQM --data  --process reRECO --scenario pp --era Run3_2024 --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run3_2024 -n 500 --filein file:step2_L1REPACK_HLT.root

[4]: cmsDriver.py step4 -s HARVESTING:@hltScouting --conditions auto:run3_data --data  --filetype DQM --scenario pp --era Run3_2024 -n 500 --filein file:step3_RAW2DIGI_L1Reco_RECO_PAT_NANO_DQM_inDQM.root

NOTE that all tests have been performed with a private file from ScoutingPFMonitor skimmed to remove all events without scouting, also because most of the RAW data from ScoutingPFMonitor are on tape. If a central dataset-file is needed for implementing the central tests, we found some accessible files from 2024G (site T1_RU_JINR_Disk) e.g. /store/data/Run2024G/ScoutingPFMonitor/RAW/v1/000/385/178/00000/b7e5f884-95f6-4369-a040-48643ed65cde.root

Please, let us know how to proceed for the integration of this (or a similar) testing workflow in the list of central ones.

Additional notes:

A PR to cms-bot to assign the content of the new repositories to the dqm area for review has been prepared: cms-sw/cms-bot#2432.

[cc: @silviodonato @missirol @mtosi]

[cmsbuild]

cms-bot internal usage

[cmsbuild]

Pull request #47235 was updated. @antoniovagnerini, @cmsbuild, @rseidita can you please check and sign again.

[elfontan]

@elfontan

We are encountering an error in the runTheMatrix flows (but it looks related to some access issue)

If you are referring to #47235 (comment) I think the suggestion at #47235 (comment) should solve (I don't understand the changes proposed in the last push)

Sorry Marco, I updated the comment and answered above: I noted the issue while repeating the full validation and in principle it should be fixed with Implementation of suggested corrections in PR47235 (config files fix) (meaning that locally step4 of the chain works successfully).

[silviodonato]

@cmsbuild please test

[mmusich]

allow @elfontan test rights

[cmsbuild]

+1

Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3abf85/44194/summary.html
COMMIT: 37cd45c
CMSSW: CMSSW_15_0_X_2025-02-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47235/44194/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 3 lines from the logs
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4016938
• DQMHistoTests: Total failures: 87
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4016831
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 218 log files, 189 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[silviodonato]

@cms-sw/dqm-l2 do you have any comments? We would like to have this merged before pre3 is cut. Thanks

[antoniovagnerini]

+1

[mmusich]

the "new package pending" flag should be removed once cms-sw/cms-bot#2432 gets merged.

[mandrenguyen]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.