[Fleet] Admin user is invalid: security_exception (file-based authentication)

[mtojek]

This issue is to investigate whether file-based authentication works with Fleet at all.

---

Hi,

this is a follow-up on the investigation of the failing setup of fleet in elastic/elastic-package#240 .

To fix the issue reported in elastic/elastic-package#234 , we tried to use xpack.security.authc.realms.file. Elasticsearch and Kibana started correctly, but the Elastic-Agent fails the fleet setup:

➜  elastic-package git:(234-file-realm) ✗ docker logs 1f184ed75383 -f
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    22  100    22    0     0      1      0  0:00:22  0:00:11  0:00:11     5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   100  100   100    0     0     93      0  0:00:01  0:00:01 --:--:--    93
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    43  100    43    0     0   1516      0 --:--:-- --:--:-- --:--:--  1535
{"isInitialized":true}{"statusCode":400,"error":"Bad Request","message":"Fleet Admin user is invalid: security_exception"}{"list":[],"total":0,"page":1,"perPage":20}
null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    82  100    82    0     0   2437      0 --:--:-- --:--:-- --:--:--  2484
null
The Elastic Agent is currently in BETA and should not be used in production

Error: fail to enroll: fail to execute request to Kibana: Status code: 401, Kibana returned an error: Unauthorized, message: [security_exception] missing authentication credentials for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="Basic realm=\"security\" charset=\"UTF-8\"" & 1="ApiKey" } } }

Does it mean that Elastic stack with enabled file-based authentication can't use the fleet? We wanted to enable this feature to prevent unavailable_shards_exception) for .security shard.

Another observation:

I think the entrypoint should fail as soon as possible if the setup fails, not try to enroll, etc.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

@kobelb You might be able to help here? Is our fleet user not created / accepted because of the file realm? Taking from the ES docs:

You should also be aware that you cannot add or manage users in the file realm via the user APIs and you cannot add or manage them in Kibana on the Management / Security / Users page

[kobelb]

@ruflin having the file realm enabled won't prevent you from creating the fleet_user in the native realm.

However, configuring the file-realm won't prevent unavailable_shards_exception exceptions from being thrown when Fleet does try to create the fleet_user in the native realm.

[mtojek]

However, configuring the file-realm won't prevent unavailable_shards_exception exceptions from being thrown when Fleet does try to create the fleet_user in the native realm.

@kobelb Do you have any recommendation on how to overcome this issue? It happens from time to time on fresh single node clusters (Docker image).

[mtojek]

Unfortunately it started appearing again: https://beats-ci.elastic.co/job/Ingest-manager/job/package-storage/job/snapshot/201/artifact/src/github.com/elastic/package-storage/build/elastic-stack-logs/kibana.log

@ph is it possible to raise a priority for this one?

[jen-huang]

@mtojek Is this still a valid issue with the move to Fleet Server and the Fleet service account in ES?

[mtojek]

I'm not sure about it. Most likely somebody from the Fleet team would have to verify if it's possible to use fleet with "file-based auth". Last time it wasn't possible at all (failures).