Update Slurm shredder to ignore non-ended job states

[jtpalmer]

Description

Instead of shredding only jobs that are expected to have ended the shredder will now be able to shred all jobs and ignore those that are known to not have not ended. Also changes the Slurm helper script to query for jobs in all states.

Motivation and Context

Slurm added a new job state and I didn't notice.

See #1107

and https://app.asana.com/0/342819846538629/1144930465270465/f

Tests performed

Added more shredder tests.

Needs manual testing.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project as found in the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[jpwhite4]

Looks good. Perhaps add a sentence or two in the upgrade notes to explain that the --state filter is no longer needed in the sacct command line when running sacct manually.

Also is it worth adding an entry to the FAQ to explain what to do when they see a warning message like 'Found job with unknown state'. Presumably they are expected to go into a blind panic and delete random tables from the database until the error goes away.

[jtpalmer]

@jpwhite4 Thank you for reminding me about that. I was planning to do something like that, but forgot.