Follow up testing work from the addition of installation flavors

[cmacknz]

Follow up testing work from #6542

Upgrades for agents with an installed flavor

• Start with an agent with the default, basic flavor and upgrade it to another version that supports flavors. After the upgrade none of the binaries from the servers flavor should be on disk.
• Start with an agent with the servers flavor and upgrade to another version that supports flavors. After the upgrade all of the binaries from the servers flavor should be on disk.

Integration test for components with additional file dependencies

• Ensure the component_files lists of osquerybeat, apm-server, endpoint-security, and cloudbeat are correctly preserved for the appropriate flavors. Ideally, confirm the integration can be installed and remain healthy as a basic sanity check.
  • endpoint-security is already covered by an integration test but the others are not.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[michalpristas]

note: checking for files defined in spec file is not an option, component files is a filter and does not require files to be present. e.g some files present only on linux, exe paths only on windows. this information is not part of the filter definition

[jlind23]

@pkoutsovasilis after chatting with @michalpristas we both believe that these tests should rather be owned by components team and not by us. We can not decently tests all the combination. Wdyt?

[pierrehilbert]

My two cents: you shouldn't test all combinations but should ensure the most commons are working (in the same way we are running some tests to ensure we can upgrade with Endpoint enabled).

[pkoutsovasilis]

@jlind23 I definitely agree in the sense that all combinations of all flavours in the shape of integrations tests is not realistic. I had a brief discussion about my thinking with @michalpristas . So, I was thinking of a divide and conquer-style approach and having components team doing extra testing on those is only helpful, so definitely this can happen. That said, I do believe that within our responsibilities are:

• flavours should contain the expected components and specs; I am thinking that more in the sense of packaging tests rather than integration ones which run locally and do check files existence and file structure right after packaging
• with the above in place, and if we wanna be extra-cautious, I believe that if we run only 1 integration test that installs a flavour with all components in it (I think it is called unflavoured 😆 ) and test them all (or at least the ones that we wanna test, e.g. endpoint) we should be covered. Don't get me wrong we most probably have this integration test already. In terms of code footprint the same code that unpacks/installs/upgrades/invokes elastic-agent will be the same for all flavours thus no need to test for each one individually

my 2 cents, please @michalpristas do tell me if I missed anything above 🙂

[cmacknz]

flavours should contain the expected components and specs; I am thinking that more in the sense of packaging tests rather than integration ones which run locally and do check files existence and file structure right after packaging

What I am most worried about covering is all the different combinations of flavor/no flavor in the unpacking step in

elastic-agent/internal/pkg/agent/application/upgrade/step_unpack.go

Line 39 in 1eefbe0

func (u *Upgrader) unpack(version, archivePath, dataDir string, flavor string) (UnpackResult, error) {

We have tests for this step but with hypothetic components, I want us to cover the situations we'll actually be subject to in real life and not hypothetical ones. This should just be the following but with validating the structure of an actual agent with the relevant component lists:

1. No flavor file -> No flavor file
2. Basic flavor file -> Basic flavor file
3. Servers flavor file -> Servers flavor file

Agree that this does not need to be an integration test specifically. I'll edit the description to be a bit less prescriptive. The ingredients to do this as more of a unit or packaging test are already there.

Ensure the component_files lists of osquerybeat, apm-server, endpoint-security, and cloudbeat are correctly preserved for the appropriate flavors. Ideally, confirm the integration can be installed and remain healthy as a basic sanity check.

The way I view the responsibilities is:

1. It is the job of the component owners to know what files they depend on at runtime and with what permissions.
2. It is our job to make sure those files are listed correctly in the spec files and preserved at installation time and across upgrades as expected.

We don't need to go as far as actually installing the complete set of affected integrations in our automated tests, though we should probably do this manually. We do need some kind of automated test that the spec files cause agent to have the correct behavior during installation and across upgrades, though again they don't need to be integration tests necessarily.

[cmacknz]

Don't assume any of the component owners know the nuances of spec files, what flavors are, or anything about how agent upgrades work.

I would generally not expect component owners to be thinking about or testing what happens to their binaries during an agent upgrade as that is an agent platform concern. The same with the new flavor permutations. Downstream teams will switch to using the variant they need for their own development but they aren't going to stress test anything or test multiple combinations on our behalf.

My view here is generally that if we don't test this, or specify exactly what teams outside of agent need to test, nobody is going to test it at all. Lots of teams do their development of their binaries separate from Elastic Agent and then just sanity check that agent runs it via the tests in https://github.com/elastic/integrations which use the container and therefore is totally unaffected by the flavors file.