Use OTEL exporterhelper to implement a 2nd queue in v1 collector for OTLP data #6487
Labels
Comments
yurishkuro
added a commit
that referenced
this issue
Jan 6, 2025
## Which problem is this PR solving? - Part of #6487 ## Description of the changes - Creation of exporterhelper may result in an error, so extend the processor construction signature to return the error. ## How was this change tested? - CI Signed-off-by: Yuri Shkuro <github@ysh.us>
3 tasks
adityachopra29
pushed a commit
to adityachopra29/jaeger
that referenced
this issue
Jan 9, 2025
## Which problem is this PR solving? - Part of jaegertracing#6487 ## Description of the changes - Creation of exporterhelper may result in an error, so extend the processor construction signature to return the error. ## How was this change tested? - CI Signed-off-by: Yuri Shkuro <github@ysh.us> Signed-off-by: adityachopra29 <adityachopra2912@gmail.com>
yurishkuro
added a commit
that referenced
this issue
Jan 12, 2025
## Which problem is this PR solving? - Part of #6487 - Part of #6474 ## Description of the changes - Swap v1 spanWriter for v2 traceWriter in collector pipeline - Currently the traceWriter is provided via v1 adapter, so it's always v1 writer underneath - And since only v1 spans entry point is currently implemented, there is no performance impact from additional data transformations - However, as soon as OTLP entry point is utilized (e.g. via OTLP receiver), the `ptrace.Traces` batch will be handled via exporterhelp queue as a single item (not broken into individual spans) and then passed directly to the writer as a batch. Since the writer is implemented via adapter the batch will be converted to spans and written one span at a time. There will be no additional data transformations on this path either. ## How was this change tested? - CI ## Outstanding - [x] Invoking proper preprocessing, like sanitizers and collector tags, on the OTLP path - [x] Adequate metrics parity, ideally same as v1 collector - [ ] Test coverage, including passing a v2-like (mock) writer that cannot be downgraded to v1 - Idea: parameterize some tests (ideally those that also validate pre-processing) to execute both v1 and v2 write paths ## Follow-up PRs * Enable v2 write path from OTLP and Zipkin receivers (they currently explicitly downgrade to v1). This will also allow adding better unit tests. --------- Signed-off-by: Yuri Shkuro <github@ysh.us> Signed-off-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
ekefan
pushed a commit
to ekefan/jaeger
that referenced
this issue
Jan 14, 2025
## Which problem is this PR solving? - Part of jaegertracing#6487 - Part of jaegertracing#6474 ## Description of the changes - Swap v1 spanWriter for v2 traceWriter in collector pipeline - Currently the traceWriter is provided via v1 adapter, so it's always v1 writer underneath - And since only v1 spans entry point is currently implemented, there is no performance impact from additional data transformations - However, as soon as OTLP entry point is utilized (e.g. via OTLP receiver), the `ptrace.Traces` batch will be handled via exporterhelp queue as a single item (not broken into individual spans) and then passed directly to the writer as a batch. Since the writer is implemented via adapter the batch will be converted to spans and written one span at a time. There will be no additional data transformations on this path either. ## How was this change tested? - CI ## Outstanding - [x] Invoking proper preprocessing, like sanitizers and collector tags, on the OTLP path - [x] Adequate metrics parity, ideally same as v1 collector - [ ] Test coverage, including passing a v2-like (mock) writer that cannot be downgraded to v1 - Idea: parameterize some tests (ideally those that also validate pre-processing) to execute both v1 and v2 write paths ## Follow-up PRs * Enable v2 write path from OTLP and Zipkin receivers (they currently explicitly downgrade to v1). This will also allow adding better unit tests. --------- Signed-off-by: Yuri Shkuro <github@ysh.us> Signed-off-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
The original collector's queue is designed to pass through one span at a time. While we can extend it to pass OTLP batches as an alternative (per the parent issue), the sizing of the queue will become misleading, counting spans vs. batches depending on the data type. An easier option is to introduce another queue with its own settings to use it for OTLP data payloads.
This can be done using OTEL's exporterhelper that implements the queue and concurrent processors.
The text was updated successfully, but these errors were encountered: