New component: FIFO diskbuffer processor

[CatherineF-dev]

The purpose and use-cases of the new component

Purpose: buffer metrics data in disk when network disconnected.

The buffer is 1. FIFO 2. support retention size 3. support retention time

Why FIFO: some metrics database don't support writing old data points.

Example configuration for the component

processors:
  disk_buffer:
    buffer_dir: "testNsq"
    retention_size_mib: 5000
    retention_time: 6h
    probe_interval: 2s
    backend_endpoint: "http://monitoring.backend.com"

Telemetry data types supported

metrics, logs, traces

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@CatherineF-dev, @leonzz

Sponsor (optional)

No response

Additional context

No response

[CatherineF-dev]

cc @dashpole

[dashpole]

cc @pmm-sumo , who worked on the persistent queue exporter helper.

@CatherineF-dev some initial questions:

• Is there a reason this only supports metrics?
• Can/should this use the storage extension? This is a common way of generically referencing "storage" resources, like a directory, or a local database.
• I assume you want to introduce a new component because the features you are looking for are not included in the persistent queue exporterhelper. IIRC, the persistent queue is FIFO, and retry can be disabled. Please correct me if i'm mistaken. Can you clarify what parts of the persistent queue don't work for your use case?
• Can you expand on the meaning and behavior of backend_endpoint and probe_interval? Are those used to determine when the network is disconnected?
• You mention "collector high memory issue[s]". Can you expand on what happened that caused high memory, and explain how this component solves that?

Thanks in advance!

[CatherineF-dev]

Is there a reason this only supports metrics?

Updated: it can support metrics, logs and traces. Didn't add logs and traces because they don't need FIFO.

Can/should this use the storage extension? This is a common way of generically referencing "storage" resources, like a directory, or a local database.

It doesn't support this storage extension, because it's a FIFO buffer. So available operations are: 1. push(item) 2. item = pop()

Can you expand on the meaning and behavior of backend_endpoint and probe_interval? Are those used to determine when the network is disconnected?

Yes, they are used to determine whether network is disconnected.

You mention "collector high memory issue[s]". Can you expand on what happened that caused high memory, and explain how this component solves that?

Deleted this part, since retry is related to exporter instead of processor.

[dashpole]

It doesn't support this storage extension, because it's a FIFO buffer. So available operations are: 1. push(item) 2. item = pop()

I believe you could implement a FIFO queue on top of the storage extension. It keeps track of a "read index" and a "write index", and uses storage.Get/SetOperation to keep items in order. Might be worth exploring.

IIRC, the persistent queue is FIFO, and retry can be disabled.

I think I am incorrect on this, actually. For send failures, the items are requeued at the back, rather than kept at the front.

[dashpole]

There is also a WAL in the remote write exporter, which is FIFO, if i'm reading it correctly: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/wal.go. Would making something like that more generically available to components work for your use-cases?

[CatherineF-dev]

Thanks! Seems it's FIFO, will check whether it supports retention size and retention time.

Not sure whether buffer_size is storage size, or the number of metrics.

      buffer_size: 100 # Optional count of elements to be read from the WAL before truncating; default of 300

[dashpole]

I think it is the number of metrics.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.