Make DefaultCacheBufferExpiration configurable to prevent OOM in small system at high pub and sub rates

[roeschter]

Proposed change

Make DefaultCacheBufferExpiration filestore.go configurable at server start

Use case

DefaultCacheBufferExpiration defaults to 10s - in certain situations (most commonly WorkQueues) all blocks reads in a stream for 10s will be cached. This can easily amount to several GB of rapid memory growth, resulting in an unexpected OOM.

We are aware that reducing DefaultCacheBufferExpiration can impact throughput.

It is a valid use case for small systems to consume or send data in bursts. The memory consumption should be bounded though as the data is streaming through the system. Reducing the DefaultCacheBufferExpiration will also throttle the system, which is a desirable side effect.

Also see:
#5673
#5739

Contribution

No response