Concurrently run beginStreams

[Dr15Jones]

PR description:

Have each stream's beginStream transition run in its own TBB task.

PR validation:

Code compiles. Framework unit tests pass.

NOTE: It could very well be that CMSSW modules are doing thread unsafe activities in beginStream. This change needs to be tests on a large sample of multi-threaded jobs.

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35230/25189

• This PR adds an extra 32KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35230/25189/code-format.patch
  e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35230/25189/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35230/25191

• This PR adds an extra 32KB to repository

[cmsbuild]

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

• FWCore/Framework (core)

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[makortel]

NOTE: It could very well be that CMSSW modules are doing thread unsafe activities in beginStream. This change needs to be tests on a large sample of multi-threaded jobs.

@smuzaffar Should we put this in for DEVEL first?

[Dr15Jones]

Should we put this in for DEVEL first?

that is what I was thinking

[Dr15Jones]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18504/summary.html
COMMIT: 93daef6
CMSSW: CMSSW_12_1_X_2021-09-10-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35230/18504/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @makortel Modernize BetaBoostEvtVtxGenerator #35057
• @jeongeun Migrate module configuration in RecoVertex to use default cfipython v2 #35195
• @makortel Make GenHIEventProducer edm::global #35052
• @smorovic DAQ optimized usage of FU file locking #35218

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18504/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18504/git-merge-result

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3001001
• DQMHistoTests: Total failures: 5
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3000973
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[smuzaffar]

Should we put this in for DEVEL first?

that is what I was thinking

yes sounds reasonable.

[smuzaffar]

enable threading

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18515/summary.html
COMMIT: 93daef6
CMSSW: CMSSW_12_1_X_2021-09-10-1100/slc7_amd64_gcc900
Additional Tests: THREADING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35230/18515/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @makortel Modernize BetaBoostEvtVtxGenerator #35057
• @jeongeun Migrate module configuration in RecoVertex to use default cfipython v2 #35195
• @makortel Make GenHIEventProducer edm::global #35052
• @smorovic DAQ optimized usage of FU file locking #35218

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18515/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7c23c/18515/git-merge-result

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3001001
• DQMHistoTests: Total failures: 5
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3000973
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[makortel]

The PR looks good to me. I'm a bit hesitant to give +1 though when the target branch is still master.

[smuzaffar]

+core
I have changed the base branch to 12.1.DEVEL now. @qliphy , @perrotta can you please merge it before 11am today so that i can we can test this change in 11h00 DEVEL IBs?

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_1_DEVEL_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1

• Here it is, for DEVEL IBs by now