Cap the number of concurrent requests

[iszulcdeepsense]

Added

Maximum number of concurrent requests can be limited by jobtype_extra.max_concurrency field:
By default, concurrent requests are unlimited. Setting max_concurrency to 1 will make the job
process requests one by one. Overdue requests will be queued and processed in order.

Having such concurrency limits may cause some requests to wait in a queue.
If a throughput is higher than the job can handle, the queue will grow indefinitely.
To prevent that, you can also set jobtype_extra.max_concurrency_queue to limit the queue size.
When the queue is full, the job will return 429 Too Many Requests status code.

Example (1 request at a time, with up to 10 requests waiting in a queue):

jobtype_extra:
  max_concurrency: 1
  max_concurrency_queue: 10

Without this setting, FastAPI uses a threadpool of 40 threads internally to handle requests using non-async endpoints.
FastAPI uses Startlette's concurrency support to run requests in a threadpool. Starlette uses anyio, which in turn has 40 as the default