contrib/redis asyncio.CancelledError being generated

[chamini2]

Summary of problem

It seems when we get a cancellation on the redis operation being done, even though the redis asyncio client has a asyncio.shield call

https://github.com/redis/redis-py/blob/7fc4c76c778163c21d396f99dcc710d99942895f/redis/asyncio/client.py#L531

        return await asyncio.shield(
            self._try_send_command_parse_response(conn, *args, **options)
        )

We are seing a cancellation errors on our Datadog traces.

 asyncio.exceptions.CancelledError
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/ddtrace/contrib/redis/util.py", line 60, in _trace_redis_cmd
    yield span
  File "/usr/local/lib/python3.9/site-packages/ddtrace/contrib/redis/asyncio_patch.py", line 18, in traced_async_execute_command
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/client.py", line 531, in execute_command
    return await asyncio.shield(
 asyncio.exceptions.CancelledError

We are running Python 3.9, dd-trace 1.10.2, redis 4.5.4.

I first thought it was about this issue: redis/redis-py#2633

But since we are using 4.5.4, which has a fix for this. It means our redis is using async_timeout instead of asyncio? (https://github.com/redis/redis-py/pull/2659/files)

https://github.com/bellini666/redis-py/blob/63a7864ec7af4fc58b2717514519e7f1d0722de1/redis/asyncio/connection.py#L28-L33

# the functionality is available in 3.11.x but has a major issue before
# 3.11.3. See https://github.com/redis/redis-py/issues/2633
if sys.version_info >= (3, 11, 3):
    from asyncio import timeout as async_timeout
else:
    from async_timeout import timeout as async_timeout

NOTE: asyncio version is only used in >=3.11.3 and we use 3.9.

I don't really know how to reproduce since this happend during a moment he had downtime in our system.

Maybe something to do with the work done in #3706?

---

Lots of guessing on my part, just wanted to give as much of the context I have.

[ZStriker19]

Hi @chamini2 could you start by giving version 1.12.4 a try? I'm wondering if this could be related to the loading issues we've seen here recently, and released a fix for in 1.12.3.

[chamini2]

Ok, added it @ZStriker19. the problem is that I have not been seeing this error because it happened during a specific condition where we were cancelling redis scans.

I think we can close for now and if it does reappear I will open this issue again.

Thanks for the response.