Canary tw implementation

[aspiringmind-code]

Implementation of canary TW for #8806
Also see corresponding changes in https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein/-/merge_requests/292

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2290/artifact/artifacts/PullRequestReport.html

[belforte]

Logic looks good. But as usual I'd like to have a few one-line comments which explain the logic to the unaware reader.
Please test on preprod first.

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2291/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2294/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2295/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2296/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2297/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2298/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 60 comments to review
• Pycodestyle check: succeeded
  • 66 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2299/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 60 comments to review
• Pycodestyle check: succeeded
  • 66 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2303/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 60 comments to review
• Pycodestyle check: succeeded
  • 68 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2304/artifact/artifacts/PullRequestReport.html

[aspiringmind-code]

@belforte Tested on preprod. See task
The task was submitted on crab-preprod-tw01

crab3@crab-preprod-tw01:/data/srv/TaskManager/logs$ grep -C 5 "vchakrav" twlog.txt
2024-11-28 11:21:58,387:INFO:MasterWorker,255:Starting external scheduling.
2024-11-28 11:21:58,471:INFO:TaskUtils,25:Retrieved a total of 1 WAITING tasks
2024-11-28 11:21:58,471:INFO:MasterWorker,314:
Username | Waiting | Selected
---------|---|--
vchakrav | 1 | 1
2024-11-28 11:21:58,472:INFO:TaskUtils,55:Will set to NEW task 241128_102130:vchakrav_crab_20241128_112126
2024-11-28 11:21:58,502:INFO:MasterWorker,352:Task 241128_102130:vchakrav_crab_20241128_112126 status updated to 'NEW'.
2024-11-28 11:21:58,502:INFO:MasterWorker,331:Pruning the queue if required...logic tbd
2024-11-28 11:21:58,502:INFO:MasterWorker,334:Report Queue status... logic tbd
2024-11-28 11:21:58,502:INFO:MasterWorker,574:Work selected successfully.
2024-11-28 11:21:58,622:INFO:MasterWorker,428:TW changed from crab-preprod-tw01 to crab-preprod-tw02 during runCanary
2024-11-28 11:21:58,709:DEBUG:Worker,253:Ready to inject 0 items

And the task ran on canary crab-preprod-tw02

2024-11-28 11:23:27,505:DEBUG:Worker,284:Completed work 14 on 241128_102130:vchakrav_crab_20241128_112126
2024-11-28 11:23:27,505:INFO:MasterWorker,579:This is canary TW crab-preprod-tw02 running.

For the test we had set canary_fraction as 1 so that it certainly goes to canary. It will be set to a small number like 0.05 by default in master TW config.

[belforte]

I suggest to put in preprod with at 30% fraction and on monday enable at low rate in prod

[aspiringmind-code]

Preprod set to 0.3 in here Ok to merge?

[belforte]

I am afraid that polling cycle is disrupted (at least happens w/o sleep) in canary. Please look at it. And in any case check what is happening.
Then I made a few "style" comments about making it more clean and readable, please change in case you agree.

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 59 comments to review
• Pycodestyle check: succeeded
  • 67 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2305/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Pylint check: succeeded
  • 58 comments to review
• Pycodestyle check: succeeded
  • 65 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2306/artifact/artifacts/PullRequestReport.html