Log sacct failures (#331)

log warnings when sacct returns error messages during job step status checks

[ committed by @ankona ]
[ reviewed by @ashao ]

doc/changelog.rst

@@ -19,14 +19,18 @@ To be released at some future point in time
 
 Description
 
+- Log ignored error messages from `sacct`
 - Fix malformed logging format strings
 - Update linting support and apply to existing errors
 
 Detailed Notes
 
+- Log errors reported from slurm WLM when attempts to retrieve status fail (PR331_, PR332_)
 - Fix incorrectly formatted positional arguments in log format strings (PR330_)
 - Update pylint dependency, update .pylintrc, mitigate non-breaking issues, suppress api breaks (PR311_)
 
+.. _PR332: https://github.com/CrayLabs/SmartSim/pull/332
+.. _PR331: https://github.com/CrayLabs/SmartSim/pull/331
 .. _PR330: https://github.com/CrayLabs/SmartSim/pull/330
 .. _PR311: https://github.com/CrayLabs/SmartSim/pull/311
 

smartsim/_core/launcher/slurm/slurmLauncher.py

@@ -224,7 +224,10 @@ def _get_slurm_step_id(step: Step, interval: int = 2) -> str:
         step_id: t.Optional[str] = None
         trials = CONFIG.wlm_trials
         while trials > 0:
-            output, _ = sacct(["--noheader", "-p", "--format=jobname,jobid"])
+            output, err = sacct(["--noheader", "-p", "--format=jobname,jobid"])
+            if err:
+                logger.warning(f"An error occurred while calling sacct: {err}")
+
             step_id = parse_step_id_from_sacct(output, step.name)
             if step_id:
                 break