Test_Agent_Remote_ES_Output flaky due to race

[michel-laterman]

Flaky Test

• Test Name: Test_Agent_Remote_ES_Output
• Link: https://github.com/elastic/fleet-server/blob/main/internal/pkg/bulk/engine.go.
• Branch: main.
• Notes: remote bulker map can encounter a race between policy.Prepare output when doing a checkin or ack, and the policy self monitor (that checks for output health).

https://buildkite.com/elastic/fleet-server/builds/5802#019388a2-40b1-42bf-8b6e-f1c2042fd80f

Stack Trace

=== RUN   Test_Agent_Remote_ES_Output
--
  | remote_es_output_integration_test.go:147: Test fleet server port=37795
  | remote_es_output_integration_test.go:150: Create policy with remote ES output
  | remote_es_output_integration_test.go:181: Create API key and enrollment key for new policy
  | remote_es_output_integration_test.go:214: Enroll agent
  | fleet_integration_test.go:727: map[action:created item:map[access_api_key:X0lPemlKTUJqdThBeUFmVF9iSlU6TW5vQW8tN1VRQ0NjcGxVN3FJVXd6dw== access_api_key_id:_IOziJMBju8AyAfT_bJU actions:<nil> active:true enrolled_at:2024-12-02T18:48:37Z id:84c90f2e-67c0-4179-acfc-32cb048abdbf local_metadata:map[] policy_id:337d887c-628a-42cd-b99b-0c0e14384e68 status:online tags:[] type:PERMANENT user_provided_metadata:<nil>]]
  | remote_es_output_integration_test.go:40: Fake a checkin for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c0003f4360 by goroutine 3151:
  | runtime.mapassign_faststr()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/runtime/map_faststr.go:223 +0x0
  | github.com/elastic/fleet-server/v7/internal/pkg/bulk.(*Bulker).updateBulkerMap()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/bulk/engine.go:148 +0xef
  | github.com/elastic/fleet-server/v7/internal/pkg/bulk.(*Bulker).CreateAndGetBulker()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/bulk/engine.go:177 +0x64c
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*Output).Prepare()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/policy_output.go:64 +0xaa6
  | github.com/elastic/fleet-server/v7/internal/pkg/api.processPolicy()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:833 +0x1027
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*CheckinT).ProcessRequest()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:364 +0x2024
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*CheckinT).handleCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:145 +0x8de
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*apiServer).AgentCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/api.go:77 +0x4c4
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/openapi.gen.go:2179 +0x10a
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*apiVersion).middleware-fm.(*apiVersion).middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/apiVersion.go:80 +0x418
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/openapi.gen.go:2186 +0x1341
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin-fm()
  | <autogenerated>:1 +0x51
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.12/chain.go:31 +0x63
  | github.com/go-chi/chi/v5.(*Mux).routeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.12/mux.go:459 +0x594
  | github.com/go-chi/chi/v5.(*Mux).routeHTTP-fm()
  | <autogenerated>:1 +0x51
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/limit.(*Limiter).Wrap.func1.1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/limit/limiter.go:94 +0x373
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*limiter).middleware-fm.(*limiter).middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/router.go:124 +0x2d5
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5/middleware.Recoverer.func1()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.12/middleware/recoverer.go:45 +0xf0
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/logger.Middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/logger/http.go:254 +0xfe7
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.12/mux.go:90 +0x69e
  | net/http.serverHandler.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3210 +0x2a1
  | net/http.(*conn).serve()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2092 +0x12a4
  | net/http.(*Server).Serve.gowrap3()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3360 +0x4f
  |  
  | Previous read at 0x00c0003f4360 by goroutine 3088:
  | runtime.mapiterinit()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/runtime/map.go:877 +0x0
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.reportOutputHealth()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:265 +0xa9
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).updateState()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:216 +0x5bc
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).processPolicies()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:183 +0x694
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).Run()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:122 +0x912
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.SelfMonitor.Run-fm()
  | <autogenerated>:1 +0x61
  | github.com/elastic/fleet-server/v7/internal/pkg/server.loggedRunFunc.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:303 +0xb8
  | golang.org/x/sync/errgroup.(*Group).Go.func1()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78 +0x91
  |  
  | Goroutine 3151 (running) created at:
  | net/http.(*Server).Serve()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3360 +0x8ec
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*server).Run.func3()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/server.go:122 +0x110
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*server).Run.gowrap1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/server.go:125 +0x82
  |  
  | Goroutine 3088 (running) created at:
  | golang.org/x/sync/errgroup.(*Group).Go()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:75 +0x124
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).runSubsystems()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:504 +0x1898
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).runServer()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:429 +0x6ae
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).Run.func4()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:199 +0x48
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).Run.func2.1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:128 +0x64
  | golang.org/x/sync/errgroup.(*Group).Go.func1()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78 +0x91
  | ==================
  | remote_es_output_integration_test.go:50: Checkin successful, verify body
  | remote_es_output_integration_test.go:110: Fake an ack for action policy:337d887c-628a-42cd-b99b-0c0e14384e68:1 for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:129: Ack successful, verify body
  | remote_es_output_integration_test.go:234: Update policy to remove remote ES output
  | remote_es_output_integration_test.go:256: Checkin so that agent gets new policy revision
  | remote_es_output_integration_test.go:40: Fake a checkin for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:50: Checkin successful, verify body
  | remote_es_output_integration_test.go:259: Ack so that fleet triggers remote api key invalidate
  | remote_es_output_integration_test.go:110: Fake an ack for action policy:337d887c-628a-42cd-b99b-0c0e14384e68:2 for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:129: Ack successful, verify body
  | testing.go:1399: race detected during execution of test