Skip to content

Add OTLP Benchmark to measure push performance #6977

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

SungJin1212
Copy link
Member

@SungJin1212 SungJin1212 commented Aug 17, 2025

This PR adds a benchmark to measure OTLP push performance.

The Benchmark results are:

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/util/push
cpu: Apple M1 Max
BenchmarkOTLPWriteHandlerPush
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1-10         	   26020	     46032 ns/op	   86703 B/op	     534 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1-10        	    6900	    179708 ns/op	  232596 B/op	    2638 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1-10       	     855	   1391819 ns/op	 1707477 B/op	   23364 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10-10         	    7507	    154604 ns/op	  265258 B/op	    2129 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100-10        	     871	   1377683 ns/op	 2248653 B/op	   20196 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000-10       	      73	  14949837 ns/op	22246728 B/op	  200325 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1-10         	    7959	    153782 ns/op	  231220 B/op	    2070 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1-10        	     906	   1321136 ns/op	 1718853 B/op	   19181 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1-10       	      86	  13179998 ns/op	17437157 B/op	  189383 allocs/op

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@dosubot dosubot bot added the type/chore Something that needs to be done; not a bug or a feature label Aug 17, 2025
@SungJin1212
Copy link
Member Author

SungJin1212 commented Aug 17, 2025

Currently, our OTLPHandler uses a Prometheus converter, which appears to cause significant memory allocation during the conversion to the Cortex format.

I was thinking we could manage a dedicated Cortex converter, which can directly convert OTLP to Cortex format. This would allow us to skip the intermediate conversion from Prometheus format to Cortex. If the case, the memory allocation will be reduced, but this is expected to come at the cost of tracking overhead.

@yeya24 @friedrichg
WDYT?

@yeya24
Copy link
Contributor

yeya24 commented Aug 17, 2025

Hey @SungJin1212, I think that makes sense. But at the same time, do we have a benchmark to understand OTLP performance? If not let's create one so that we can compare the performance impact and its baseline

@SungJin1212
Copy link
Member Author

@yeya24
Yeah, let me add a Benchmark to show current memory allocations.

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212
Copy link
Member Author

@yeya24
I added a Benchmark to evaluate push performance.

@SungJin1212 SungJin1212 changed the title Chore: add small optimization to otlp Add OTLP Benchmark to measure push performance Aug 18, 2025
@SungJin1212 SungJin1212 requested a review from friedrichg August 20, 2025 01:04
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
size/L type/chore Something that needs to be done; not a bug or a feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants