Serious performance issue with redis connection pooling due to CaseInsensitiveDict

[aldem]

I was doing performance tests in my app, and noticed that if I serve a large number of requests simultaneously, doing something like this:

async with redis.asyncio.Redis(connection_pool=redis_blocking) as conn:
    await conn.set(...)

... the number of requests per second is significantly lower (around 45-50%) compared to the case where I take a connection only once.

Since every request (as the app provides an API) requires taking a connection from the pool, I cannot do it once and then use multiple times - it will be even slower due to serialization (and will actually defeat the purpose of the pool).

Profiling revealed that the slowdown is primarily caused by the use of CaseInsensitiveDict in the Redis.__init__() method:

redis-py/redis/client.py

Lines 382 to 387 in a4df6b2

	self.response_callbacks = CaseInsensitiveDict(_RedisCallbacks)
	if self.connection_pool.connection_kwargs.get("protocol") in ["3", 3]:
	self.response_callbacks.update(_RedisCallbacksRESP3)
	else:
	self.response_callbacks.update(_RedisCallbacksRESP2)

Especially, update() is expensive, since there are many keys to update; the profiler shows that __setitem__() alone consumes approximately 50% of the time spent, most of which is attributed to the str.upper() method.

Replacing the dictionary with a simple dict reduced the time needed to acquire the client (not a connection) by a factor of 12, resulting in a performance boost of approximately 45% in the aforementioned use case, and its performance became nearly comparable to that of acquiring a connection only once.

Now the question - is there a reason to use CaseInsensitiveDict? I could not spot anything obvious in the code that requires case-insensitive keys, while the performance gain is significant.

Additionally, moving the initialization of self.response_callbacks to the pool (as they are constants anyway) could potentially lead to further performance improvements.

Thank you!

[petyaslavova]

Hi @aldem , thank you for pointing this out! We'll have a look at this.