Many JDBCConnectionExceptions are thrown when muse is under heavy load

[jaserud]

Details:

• It looks like muse is unsuccessfully attempting to get JDBC connections for transactionals (see logs observed during heavy submission load bellow)
• JDBC connections is used for flyway but the data operation should be via R2DBC connections
• Most likely there is a mismatch between db drivers injection some where (or could be something else)
• the overload of exceptions eventually makes muse unhealthy and it gets restarted (which is bad if we have inflight submissions as they are lost)

Logs while muse was running during some submissions:

Expected Behavior:

• muse should be consistent with its db connections

[ghost]

@joneubank can you please take a look and advise if it still occurs and needs action?

[justincorrigible]

This issue sprung up again just now as we were testing the Maestro + Song upgrades (4.2.0 and 5.0.1, respectively).

[ghost]

And I wont term this situations as "under heavy load" . it was 100 samples 3MB data. need to fix this for successfully uploading samples to the portal,