[META] Elastic Agent V2 Architecture

[jlind23]

Meta issue to publicly track Elastic Agent V2 progress.

Actual V2 architecture design.

Related repository: https://github.com/elastic/elastic-agent-shipper

Goals

• Reduce the number of connections required by Elastic Agent to Elasticsearch or other remote services.
• Reduce the complexity of the inputs
• Reduce the memory usage
• Easier debugging
• Single logging from all the components.
• Better introspection of the running processes.

Development Plan

Elastic Agent preparation phase:

• [Meta] Remove Elastic Agent V1 control protocol elastic-agent-client#32

• Phase 1:

  • Beta: Shipper is available for Filebeat and Metricbeat by turning a feature flag.
  • GA: Shipped is used by default with Filebeat and Metricbeat under Agent.

• Phase 2: Integrate with the security solution:

  • Endpoint security.
  • Security beats:
    • packetbeat
    • auditbeat
    • winlogbeat
    • osquerybeat

• Phase 3: Migrate remaining beats and agent inputs

  • APM server
  • Heartbeat
  • Cloudbeat

• Phase 4

  • Run a process per input type.

Phase 1: Separate the shipper from Filebeat and Metricbeat under Agent. (See elastic/elastic-agent-shipper#3)

The goal is to reduce the number of connections for Filebeat and Metricbeat instances to remote services.

• [Design] Elastic Agent Control Protocol v2 gRPC. #215
• [Meta][Feature] Integrate the shipper into the elastic agent packaging step #216
• Define a feature flag for enabling the shipper in the agent policy, defaulting to false. #217
• Migrate from using port to use named pipes or socketfile #218
• [Meta][Feature] Start the shipper process(es) and configure inputs to write to them when the shipper feature flag is enabled. #219
• Elastic-agent should not swallows stderr and panics #88
• Elastic Agent should aggregate the log from his subprocesses #221
• [Meta][Feature] Refactor how Elastic Agent know how to run a binary. #222

Phase 2: Integrate with Endpoint security & the Security beats.

The goal is to move a uniform output strategy.

• [Meta][Feature] The agent policy's input processor configurations must be provided to the shipper. #220

Phase 3:

• Run a process per input type: https://github.com/elastic/obs-dc-team/issues/699

QA / Testing Needs

• New end to end tests:
  • TBD.
  • Add a test where the agent is configured to use the shipper from startup.
  • Add a test where the agent is configured not to use the shipper at startup, then switched over to it.
• Performance testing:
• New manual QA tests:
  • TBD. At minimum the agent must be regression tested on each supported OS with the shipper enabled.

Documentation

The following documentation tasks should be accomplished at a minimum:

• Document the change in the logging
• Document the usage for the feature flag (environment or via elastic agent policy)
• Document the new global processing options.

Target

• Definition / Design: 8.2
• Beta: 8.4/8.5
• GA: TBD

[pierrehilbert]

Closing this one as v2 architecture has been released in 8.6