Dashboard issues

[tasmith4]

Issues that need to be resolved to make the dashboard green:

• Tests that require hypre are enabled in a no-hypre build #976
• BoussinesqNonIso test uses a Python 3.7 feature #977
• OpenFAST segfault in CPU builds #979
• Runtime error on Summit: cudaStreamSynchronize(stream) error( cudaErrorIllegalAddress): an illegal memory access was encountered #973
• Belos residual norm is NaN #982
• Cuda out of memory errors #983
• CUBLAS error #984
• CUSPARSE error #985
• GPU failures on SNL nightly testing #994
• AMSKernelHex8Mesh.NGP_ams_forcing GPU unit test failure #995
• Many SNL GPU tests failing due to gold file mismatch across consecutive runs #996
• edgeHybridFluids test diffing #1026
• Cuda configuration error #1027
• Clang warnings about overridden functions #1028
• Update for Kokkos 3.7 #1041

The list is complete as of this edit. If new failures appear they can be added here.

[jhux2]

@tasmith4 Just a hunch, but I wonder if many of these test failures could have a similar root cause. The current failure mode for #982 is

what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered

It appears there's a bad memory access upstream, and how it manifests changes (NAN, illegal mem access, out of memory, etc.).