Rework the pipeline: batch allocation + deterministic backoff

[wyfo]

See more details in #1769 (comment)

[wyfo]

The performance regression on Linux has been fixed.
The random backoff has been replaced by a deterministic batching mode: when there is more than one batches used at the same time (for a given priority), it means that the tx task is not delivering messages fast enough, so the pipeline enters in batching mode where it doesn't notify the tx task when messages are written on the current batch.
On the tx side, when batching mode is detected, it backs off until a deadline starting from the latest successful pull (or until a batch is pushed ofc); if deadline is reached, batching mode is manually stopped (next notification will be triggered on the pipeline side). If there is no backoff (or deadline was reached), the current buffer is retrieved — or a small backoff is applied if it is not available, but a notification should arrive soon in this case.

This PR gives the same throughput as main branch on Linux, around 2.7Mmsg/s on my Linux setup. On my Mac, both branches are around 6Mmsg/s. Latency of low throughput is untouched. The CPU consumed by the tx task seems to have been improved, which was expected because batching is done more deterministically and systematically.
Regarding memory consumption, batches are allocated with a size depending on the current workflow: if the last pushed batch was small enough, then the small size (an arbitrary constant of 2kiB for now) is used for the next one, otherwise, it will use the max size.

The current PR version embeds a batch reusing mechanism: one single batch is kept in memory, and if it is not available (meaning it is currently owned by the tx task), a new one is allocated on demand, respecting the limit fixed by the configuration. The batch will be refilled by the tx task after use, the next one being dropped if the slot is already refilled.
However, the impact of the allocations on performance is quite negligible, as reported on the graph below: when refilling is commented out, malloc in get_batch counts for only 0.01% of the CPU, and free in refill for 0.1%. As expected with these numbers, not using refilling doesn't have impact on performance.

Here are the flamegraphs of z_pub_thr

• this PR
  
• this PR (without using batch refill, so only allocations)
  
• main
  

[wyfo]

This constant is of course arbitrary. It can be exposed in configuration if needed.