Reduce the test matrix

[sagikazarmark]

The current test matrix tests the operator on all supported Kubernetes versions with all supported Vault versions. However, for the operator we really should be testing Kubernetes compatibility, not Vault compatibility.

Although the Bank-Vaults CLI has way less integration tests (something we should probably change), failing an integration test due to a bug in Bank-Vaults itself does not mean anything from the operator's perspective.

I suggest reducing the test matrix in PRs to all supported Kubernetes versions and the latest Vault version.

We can keep running tests for all supported Vault versions on merges and in nightly builds, to make sure we catch integration errors, but I think we should reduce the size of the matrix in PRs.

WDYT?

@akijakya @ramizpolic

[akijakya]

While I agree that a bug in another repo should be caught there, I don't see how reducing the matrix should solve this issue, the time for the checks wouldn't really decrease as they are executed in parallel. Maybe we could put the same acceptance test in the bank-vaults repo as well, and use a pinned version of the operator in bank-vaults, a pinned version of bank-vaults in the operator and only update them when it is known for that version that it passes its checks?

[sagikazarmark]

checks wouldn't really decrease as they are executed in parallel.

There is a limit on the amount of concurrent jobs we can run, so they don't all run parallel. We often have to wait for a set of jobs to complete before the next ones can launch.

[akijakya]

There is a limit on the amount of concurrent jobs we can run, so they don't all run parallel. We often have to wait for a set of jobs to complete before the next ones can launch.

Oh I didn't notice that, just assumed that a rolling yellow circle means it is in progress!

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.