Bulk API return document level 500 when new shards are being allocated

[esatterwhite]

Describe the bug
Under certain conditions using the elasticsearch _bulk api, quickwit, particularly during spikes in ingestion traffic, or when an index is initially created, quickwit will reject documents with a document level status of 500, internal_exception with the reason no shards available. This tends to indicate that something has gone wrong on the server and the document cannot be retried

{
  status: 500
, error: {
    type: 'internal_exception'
  , reason: 'no shards available'
  }
}

This can cause problems when using existing elasticsearch client libraries. Many of them have logic implemented for handing retires and document level errors from the bulk api. However, the rate limiting generally only kicks in when the document level status is a 429. This can be problematic for existing applications where the retry logic is leveraged. In the current quickwit behavior, documents will generally be dropped assuming the error is terminal when its really a transient warmup problem.

Expected behavior
The bulk api document errors should be a 429 when there are no shards available. It may also be helpful to return a error code that is more indicative of the problem rather than an `internal_exception

{
  status: 429
, error: {
    type: 'no_shard_available_action_exception' // elasticsearch has this error code, but it may mean something else in that context.
  , reason: 'no shards available'
  }
}

[fulmicoton]

One trouble is that we actually don't want you to retry right away in that case. Maybe we should set an informative retry_after header? (500ms maybe)

[esatterwhite]

One trouble is that we actually don't want you to retry right away in that case. Maybe we should set an informative retry_after header? (500ms maybe)

Es clients don't always return response headers. But it's not a bad idea. Maybe in the metadata returned in the response, and header.

Retry-After: 500 is the standard header for rate limiting
We do have delays on the retries though

[esatterwhite]

I will try to trace down how the error metadata is constructed to see if it can be influenced.

[esatterwhite]

Yeah, I don't see anything that relays response headers through the elasticsearch client libs. Thats not to say it isn't a good idea. If that kind of info can be transmitting back, its should.

I think in the case of the bulk api, it would probably need to be included in the document level responses as well though. That is expected pattern for that one as its kind of a multi-response situation.