Adjust Scheduler to Validate Existence of Server Before Start and Update Task If Needed

[mitchelsellers]

Description of the problem

Currently within DNN Platform if you schedule a task to execute on a single server, and that server no longer exists the job will simply not run. In the case of scaled solutions, this can result in tasks that no longer run.

Description of solution

DNN Platform currently tracks a WebServers table that has a "last activity" date. The proposed solution would be to do the following when starting a task

• If no specific servers are set, continue with the normal process, no impact
• If a single server is set
  • Validate that the server is still listed in the WebServer table and that its last activity is within 10 minutes
  • If it is not, find another server from the list that IS active an update the task
  • Ensure that the task runs
• If more than one server is set (This is not an issue after Adjust Scheduled Tasks to Support Single "Run on Server" #5576)
  • Follow a similar process to the above, but if a server is invalid remove it. ONLY add another server if it would be left with no servers defined

Description of alternatives considered

We discussed possibly updating the health checks or otherwise, but it would put more load on the environment rather than targeting the task start process

Additional context

This change is important for Azure Auto Scale, other cloud, and other multi-server situations