Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider · 2025-01-22T14:31:27Z

Additional changes :

PR Unit test job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests) - add GPU_FLAVOR parameter, select node based on it.
PR Relval job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/) - select node based on TEST_FLAVOR
Monitoring job (https://cmssdt.cern.ch/jenkins/job/jenkins-monitor-queue/) needs to be changed - kill pr test jobs that are stuck due to lumi node being offline
If ROCm job is killed due to node being offline, mark commit status!

cmsbuild · 2025-01-22T14:31:53Z

cms-bot internal usage

iarspider · 2025-01-22T14:55:00Z

please test with cms-sw/cmssw#46579

to check that cpu tests are not broken

cmsbuild · 2025-01-22T15:00:14Z

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43916/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43916/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 1 CMS deprecated warnings found, see summary page for details.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

iarspider · 2025-01-22T15:07:25Z

please test with cms-sw/cmssw#47163

cmsbuild · 2025-01-22T17:13:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43917/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 1664 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3819085
DQMHistoTests: Total failures: 149
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3818916
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

iarspider · 2025-01-23T08:39:49Z

test parameters:

addpkg = HeterogeneousTest, HeterogeneousCore
enable = gpu
workflows_gpu = 141.044406,141.044408,141.044412,141.044414,141.044422,141.044424,160.03502

iarspider · 2025-01-23T08:39:53Z

please test

cmsbuild · 2025-01-23T17:05:19Z

Pull request #2418 was updated.

iarspider · 2025-01-23T17:19:57Z

please test

iarspider · 2025-03-03T12:40:25Z

please abort

iarspider · 2025-03-03T12:45:27Z

please test

cmsbuild · 2025-03-03T12:45:41Z

Pull request #2418 was updated.

cmsbuild · 2025-03-03T15:14:07Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44767/summary.html
COMMIT: 1d0c56c
CMSSW: CMSSW_15_1_X_2025-03-03-1100/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/44767/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 14 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 98
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920182
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 10
DQMHistoTests: Total histograms compared: 57865
DQMHistoTests: Total failures: 1239
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 56626
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 9 files compared)
Checked 50 log files, 48 edm output root files, 10 DQM output files
TriggerResults: no differences found

iarspider · 2025-03-03T15:17:13Z

All links are fixed now. Reminder: 1d0c56c should be reverted before merging.

iarspider · 2025-03-10T10:03:44Z

please test

cmsbuild · 2025-03-10T10:24:03Z

Pull request #2418 was updated.

cmsbuild · 2025-03-10T17:01:36Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44884/summary.html
COMMIT: e217f79
CMSSW: CMSSW_15_1_X_2025-03-09-2300/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2418/44884/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 24
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920256
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 48 differences found in the comparisons
DQMHistoTests: Total files compared: 10
DQMHistoTests: Total histograms compared: 57865
DQMHistoTests: Total failures: 1928
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 55937
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 9 files compared)
Checked 50 log files, 48 edm output root files, 10 DQM output files
TriggerResults: no differences found

smuzaffar · 2025-03-11T14:50:58Z

please test

lets run the tests with few packages ( e.g. HeterogeneousCore, HeterogeneousTest) checked out

iarspider · 2025-03-11T14:55:05Z

@smuzaffar I have started the test, if you plan to do more tests - rebuild this job, since automatic tests won't work (and the PR is ignored by bot)

cmsbuild · 2025-03-12T07:50:09Z

-1

Failed Tests: cudaUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44917/summary.html
COMMIT: e217f79
CMSSW: CMSSW_15_1_X_2025-03-11-1100/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2418/44917/install.sh to create a dev area with all the needed externals and cmssw changes.

CUDA Unit Tests

I found 1 errors in the following unit tests:

---> test cudaTimeMeasurement had ERRORS

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920277
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 47 differences found in the comparisons
DQMHistoTests: Total files compared: 10
DQMHistoTests: Total histograms compared: 57865
DQMHistoTests: Total failures: 3400
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 54465
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 9 files compared)
Checked 50 log files, 48 edm output root files, 10 DQM output files
TriggerResults: no differences found

iarspider · 2025-03-12T10:43:47Z

@smuzaffar ping

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jan 22, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 22, 2025

cmsbuild added tests-rejected and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started and removed tests-rejected labels Jan 22, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started tests-pending and removed requires-external tests-approved tests-started labels Jan 23, 2025

cmsbuild added tests-started and removed tests-pending tests-started labels Jan 23, 2025

cmsbuild added tests-pending and removed tests-started labels Mar 3, 2025

iarspider force-pushed the pr-rocm-tests branch from 17fc727 to 1d0c56c Compare March 3, 2025 12:44

cmsbuild added tests-started and removed tests-pending labels Mar 3, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 3, 2025

cmsbuild added tests-started and removed tests-approved labels Mar 10, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 10, 2025

iarspider requested a review from smuzaffar March 11, 2025 14:01

cmsbuild added tests-rejected and removed tests-approved labels Mar 12, 2025

Run PR gpu utests/relvals on both CUDA and ROCm GPUs

cc617e3

iarspider force-pushed the pr-rocm-tests branch from 7589a09 to cc617e3 Compare March 12, 2025 10:52

iarspider merged commit fcf6a5c into master Mar 12, 2025
7 checks passed

iarspider deleted the pr-rocm-tests branch March 12, 2025 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 23, 2025 •

edited by smuzaffar

Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

smuzaffar commented Mar 11, 2025

iarspider commented Mar 11, 2025 •

edited

Loading

cmsbuild commented Mar 12, 2025

iarspider commented Mar 12, 2025

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Conversation

iarspider commented Jan 22, 2025 • edited Loading

cmsbuild commented Jan 22, 2025 • edited Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Clang Build

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Comparison Summary

iarspider commented Jan 23, 2025 • edited by smuzaffar Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

Comparison Summary

CUDA Comparison Summary

iarspider commented Mar 3, 2025

iarspider commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

Comparison Summary

CUDA Comparison Summary

smuzaffar commented Mar 11, 2025

iarspider commented Mar 11, 2025 • edited Loading

cmsbuild commented Mar 12, 2025

CUDA Unit Tests

Comparison Summary

CUDA Comparison Summary

iarspider commented Mar 12, 2025

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 23, 2025 •

edited by smuzaffar

Loading

iarspider commented Mar 11, 2025 •

edited

Loading