[Stack Monitoring] [R&D] Document how we store, query, and display shard information

[jasonrhodes]

There is an ongoing effort to reduce the amount of inefficient queries against Elasticsearch, related to retrieving information about ES shards. Stack Monitoring is one of the biggest callers of these endpoints* and we are aware of the idea that for a long time, we've been retrieving and storing shard information that we may or may not need to power the monitoring UI(s).

We need to document which endpoints we call, which documents/fields we store (related to ES shards), and whether or not we use those in the UI. If we don't, we should log an issue to remove this retrieval and storage. If we do use them, or if we determine that we should keep the retrieval in place for some reason, we should work with the ES team to determine a new API for querying this information.

*Which ES endpoints are the problematic ones we need to look into here?

Acceptance Criteria

• A write-up exists documenting
  • the shard-related ES APIs that the elasticsearch MB module calls,
  • the actual fields we store in our monitoring indices from these calls, and
  • which of these fields we display in the monitoring UI
• New issue(s) is/are filed to either remove some of these calls, adjust the UI, and/or switch to a new ES API that can be more performant/efficient for what we really need

Outstanding Questions:

• Does ES internal collection have the same problem? cc: @jakelandis
• Which endpoints are the problem ones from the ES "many shards" perspective?

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

I think there's some overlap with elastic/beats#29980 (Establish first class monitoring procedure for elasticsearch) here.

Today we just query cluster state which isn't an explicitly stable API and it's also heavy, as outlined in this issue.

[jasonrhodes]

We can probably replace this with the wider effort to document this same information for each module -- elasticsearch module here #127209