Upgrade CDash

[jwnimmer-tri]

From #19099 (comment):

I spoke to one of the maintainers of CDash. A new release of CDash is going to be released soon. He suggested that we wait until that is available and do an upgrade at the same time as moving to a different instance type.

[BetsyMcPhail]

The new CDash server is setup on a smaller instance and ready to test! There has been a few successful submissions from Jenkins:

drake-ci PR: RobotLocomotion/drake-ci#244
drake PR: #20231

Test build: https://drake-jenkins.csail.mit.edu/view/Experimental/job/linux-focal-gcc-cmake-experimental-release/9312/

The plan is to create a few more test cases with failing builds, tests, etc. If that goes well, do the switch next week, after Jenkins has been updated.

[BetsyMcPhail]

@jwnimmer-tri would Thursday morning 10/5 be okay to do the update? CDash will be down for a bit but the rest of CI should be unaffected.

[jwnimmer-tri]

Sounds fine. CDash is not on the critical path for anything, so it's fine to down it whenever.

[BetsyMcPhail]

We did some more testing and ran into some issues. Upgrade is on hold for now while we work through those.

[BetsyMcPhail]

Moving this issue out of "In Progress", we are waiting for a CDash developer to have time to look at the issues.

[BetsyMcPhail]

https://open.cdash.org encountered a similar issue with a CDash background process stopping and not restarting itself. After a fix has been implemented and tested out for a few days on that instance, we will try the TRI update again.

[BetsyMcPhail]

The test CDash server has been updated and test submissions are looking good: http://ec2-34-203-13-209.compute-1.amazonaws.com/cdash/index.php?project=Drake&date=2023-11-07

The next step is to try to switch the test instance into production.

[BetsyMcPhail]

The CDash server has been upgraded.

Still outstanding:

• Ensure new instance is auto-backed up (@svenevs)
• Delete old instance
• Set up certbot auto-renewal (@BetsyMcPhail)
• Ensure CDash logs are visible on CloudWatch (AWS)

Note, this did not fix the Sonoma CDash issues #20718

[svenevs]

https://us-east-1.console.aws.amazon.com/backup/home?region=us-east-1#/backupplan/details/83a5c41b-0f95-47f3-bac0-af87484543be/selection/details/3b288525-de5e-49a0-a19e-d0ce24dd97d8

New backup plan created, old one deleted. I am ready to check that box off after 5am tomorrow (aka lets confirm the backups exist next week first, they kick off at 5am every day).

The old instance has already been Stopped, when we are happy with our backups for drake-cdash-3 we can also then terminate it.

[BetsyMcPhail]

The certbot auto-renewal and cloudwatch logging have been setup on the new CDash server. @svenevs when you're happy with the backups, please go ahead and delete the old instance and close this issue.

[svenevs]

drake-cdash-3 is backing up as expected. drake-cdash and its associated volume have been terminated.