[DRAFT]: feat: use pooled http transport for grafana client

[eyazici90]

Context

Given reusing the same http client for he same host significantly reduces the overhead for dns lookup and handshakes for the same host.

Also leveraging sync.Pool to reduce allocations for geenrating httppool and transportconfig to prevent allocating same structs over again, instead reusing.

This becomes quite important when there are many resources(dashboards, folders, datasources) at thousands level being reconciled concurrently. Reusing the connection for the same host improves a lot.

Benchmark shows an improvement both in speed and mem. usage.

Benchmark_GenGrafanaClient
Benchmark_GenGrafanaClient-12        	  368535	      3266 ns/op	    6130 B/op	      94 allocs/op
Benchmark_GenGrafanaClient_Old
Benchmark_GenGrafanaClient_Old-12    	  307849	      3852 ns/op	    7176 B/op	      97 allocs/op

Related https://github.com/hashicorp/go-cleanhttp/blob/master/cleanhttp.go

[Baarsgaard]

Since you pivoted a bit, would you update the description or leave a comment with some of the considerations?
We squash commits on merge and the detailed commit descriptions are unfortunately lost outside of the PR history.

[eyazici90]

Since you pivoted a bit, would you update the description or leave a comment with some of the considerations? We squash commits on merge and the detailed commit descriptions are unfortunately lost outside of the PR history.

Sorry, I didn't understand fully. Do you want me to amend the commit messages 😅 The title remains the same. I updated the PR desc a bit. Lmk, what exactly u want me to update. Happy to follow ;)

[Baarsgaard]

Ah, we do not get notified when the description changes 😅
The Commit messages are fine, I would have liked a some of the considerations in the description or as a comment.
Commit messages are not the easiest thing to find at a glance in the Github UI and is often overlooked.
I will do a review of this quite soon. I'm having to read up on the sync.Pool some more to understand how others use it and what is idiomatic.

[Baarsgaard]

This is coming from a limited understanding of sync.Pool, so take it with some salt.
But this defer looks "off".

Every example I see for using sync.Pool always finishes the work before returning the object to the pool.
But here the object is returned to the pool before the client and the containing transports are used.
Could another Goroutine Get() the httpTransport and cfg during a preemption outside of NewGeneratedGrafanaClient and overwrite both?

We have multiple controllers making more than one request during a single reconcile resulting in a lot of preemption opportunities.

[eyazici90]

Very good point. Thanks for noticing that. I reckon, I didn't completely check the control flows and solely focused on my use case. Which was maxConcurrentReconcilation:1 for datasources that was simply serialized execution 🤦

I totally forgot about we can set maxConcurrentReconcilation to more or even different control flows from diff resources here.

I will go for a long weekend vacation and will work on this pooling(sync.Pool) more seriously(not the part for http transport that is pretty much necessary) when I back. Will ping u later on.

Thank u so much for spending ur time on the review, appreciated 🙏