Make the Vault Connect CA Provider tests a separate CI line item test

[rboyer]

#6491 refactored the tests for the Vault Connect CA Provider to shell out to an actual copy of vault rather than running a copy in-process.

We should trigger those tests as a separate line item in CI so it is very obvious when they are skipped due to a lack of vault binary. This would also let us more easily run the suite against a series of vault binary versions, like the envoy integration tests.

[mkcp]

Looking at picking this up!

[mkcp]

Discussed w/ @rboyer, @alvin-huang, and @mikemorris for recommendations on an approach, porting everyone’s comments together and adding some of my own to scope this ticket out. Going to be break this down into subcomponents.

First, from discussing with Alvin, we want to add a new make target to run the tests, this will give a good interface for running the tests both locally and in CI. I expect the make target will look something like: (though it’s possible we’ll require more context, and have to write a shell script for it)

test-vault-provider:
	@echo "Running Vault Provider Tests"
	@go test $(CURDIR)/agent/connect/ca/provider_vault_test.go -run runTestVault

Specifically which tests to run is an open question — I will need to explore more here once I get the tests running locally w/ a vault binary. We will also need to add a hook somewhere in the CI workflows for it — Alvin offered some time to pair and I’ll be taking him up on that. (Especially important since I don’t have access to the Consul CI infra yet 😅)

For the future: I had a chance to explore the envoy tests and they ran great the first time on my machine. With some more abstractions in place for the Vault CA provider tests, it’ll definitely be easier to make testing against N different vault binaries easier in the future. Once this is done, I can do some more work to scope that out.

[mkcp]

Added a make target in #6949 that exposes a command for running the vault-ca-provider tests locally and in CI. Just need to verify that the circleCI config is working correctly and we should be able to complete this issue. After that, we ought to create a new issue for running against a collection of vault versions.

[mkcp]

We now have a separate CI job for vault-ca-provider under the test-integrations workflow using a new make target: test-vault-ca-provider. This target runs the set of vault-ca-provider tests polymorphically, rendering different test output whether it's in CI or run on a local dev machine.

Testing additional versions of vault is out of scope of this issue, but I added #6963 to document @rboyer's ideas there and the rest of the context I have at hand.

s/o's to @rboyer for guiding initial implementation, review, and future steps, and @alvin-huang for reviewing and working w/ me to polish the CI and makefile impls.

[ghost]

Hey there,

This issue has been automatically locked because it is closed and there hasn't been any activity for at least 30 days.

If you are still experiencing problems, or still have questions, feel free to open a new one 👍.