ETIMEDOUT connecting to AWS Elasticache (non-cluster) #1042

chrisfinch · 2020-01-10T22:06:36Z

Trying to connect to an AWS Elasticache instance and running in to this problem.

Verifying that Redis connection works from EC2 instance;

/usr/src/app/packages/api # redis-cli -h <AWS_URL> -p 6379
<AWS_URL>:6379> SET a b
OK
<AWS_URL>:6379> GET a
"b"

Connecting like this:

const getRedisConfig = () => {
  const cfg = {
    host: process.env.REDIS_HOST
  };
  if (process.env.NODE_ENV === 'production') {
    cfg['tls'] = {};
  }
  return cfg;
};
// ...
this.connection = new Redis(getRedisConfig());
// ...

Recieving ETIMEDOUT error when connection through ioredis;

/usr/src/app/packages/api # DEBUG=ioredis:* yarn start
yarn run v1.21.1
warning package.json: No license field
$ node dist/server/index.js
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +0ms
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +2ms
ioredis:redis queue command[<AWS_URL>:6379]: 0 -> set([ 'a', 'b' ]) +1ms
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +2ms
ioredis:redis queue command[<AWS_URL>:6379]: 0 -> subscribe([ 'ASSESSMENTS' ]) +0ms
ioredis:redis status[pad-app-api-redis.dj3r40.0001.euw1.cache.amazonaws.com:6379]: [empty] -> connecting +1ms
ioredis:redis status[10.0.0.33:6379]: connecting -> connect +546ms
ioredis:redis write command[10.0.0.33:6379]: 0 -> info([]) +1ms
ioredis:redis status[10.0.0.33:6379]: connect -> ready +4ms
(node:843) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
    ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +2ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +10s
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +1ms
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +1ms
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +48ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +2ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +10s
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +1ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +10s
ioredis:connection reconnect in 100ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +1ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +0ms
ioredis:connection reconnect in 100ms +0ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +0ms
ioredis:connection reconnect in 100ms +0ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +101ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms

Timeouts continue indefinitely like above.

Out of ideas with this one so if anyone could help I would really appreciate it.

The text was updated successfully, but these errors were encountered:

chrisfinch · 2020-01-10T22:57:33Z

Checked timeout setting on Elasticache parameters - is set to 0 so doesn't look like that's the problem.

simondutertre · 2020-01-13T15:50:34Z

+1 Some of the time, having this error in AWS lambdas connecting to elasticache

ERROR	[ioredis] Unhandled error event: Error: connect ETIMEDOUT
    at Socket.<anonymous> (/var/task/src/SQS-consumer/node_modules/ioredis/built/redis/index.js:282:31)
    at Object.onceWrapper (events.js:286:20)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at Socket._onTimeout (net.js:443:8)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)

wjt382063576 · 2020-01-14T11:51:59Z

I have encountered the same problem. Do you have a solution?

chrisfinch · 2020-01-14T12:08:22Z

@simondutertre @wjt382063576 I'm still trying to find a solution - if I find anything I will post it here - hopefully the maintainers of ioredis will be able to help soon.

chrisfinch · 2020-01-14T15:34:05Z

I managed to find a working stopgap solution as follows;

In AWS Elasticache Redis:

Turn OFF Encryption in-transit
Turn OFF Encryption at-rest

In code - connect like this;

const connection = new Redis({
    host: "my_aws_redis_instance_url"
});

This is only a temporary solution so I can continue development - encryption will be required for production deployments of my code so a solution for this timeout problem will still need to be found.

vcalmic · 2020-01-17T14:14:02Z

@chrisfinch
I have an Elasticache with encryption in-transit and at-rest, with TLS. Here is my implementation:

this._client = new IORedis({
  tls: {},
  port: '6379',
  host: process.env.REDIS_HOST,
  password: process.env.REDIS_AUTH,
  keyPrefix: `${process.env.LAMBDA_ENV}-`,
  connectTimeout: 17000,
  maxRetriesPerRequest: 4,
  retryStrategy: (times) => Math.min(times * 30, 1000),
  reconnectOnError: (error) => {
    const targetErrors = [/READONLY/, /ETIMEDOUT/];

    targetErrors.forEach((targetError) => {
      if (targetError.test(error.message)) {
        return true;
      }
    });
  }
});

You can try it out and check if it works ;)

stale · 2020-02-16T15:13:59Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

aalimovs · 2020-03-31T09:08:09Z

FYI, you have to run your Lambdas in your own VPC to be able to connect to Elasticache. Double check subnets and security groups.

demian85 · 2020-05-19T22:17:10Z

I've already double-checked everything you mentioned in this thread but I keep getting timeout errors and I don't know where to look at!

aalimovs · 2020-05-20T10:13:46Z

@demian85 is your Lambda in your own VPC with the right subnet group?

demian85 · 2020-05-20T12:54:29Z

Which should be the right subnet group?
The Redis instance is in 2 subnets, the same for the lambda function. The same security group also, allowing inbound and outbound connections for everyone in the same group.

ShraddhaMantri · 2020-08-05T21:18:02Z

Did anyone find a solution for this? I am seeing the same issue. @luin

ColeTownsend · 2020-10-16T16:52:11Z

Same issue as well.

demian85 · 2020-10-22T19:56:58Z

I'm still running into the same issue and I don't know which is the proper solution to Lambda aggressively killing the socket and making the connection not reusable. Is that the problem, right?
So:

If I use lazyConnect and disconnect at the end of the lambda execution, will this work when reusing the lambda instance later?
Can I use the client that was previously closed and connect implicitly when sending a command? Should I connect explicitly always?

Thanks.

CostaFernando · 2020-11-22T12:59:42Z

Hi guys,

I'm still having this problem on AWS Elasticache with encryption in-transit and at-rest, with TLS. Did you find any solution? If I turn encryption off on AWS, it works, but I need encryption for my production environment.

Aung-Myint-Thein · 2021-01-30T04:51:20Z

hello.. I am on ioredis 4.19.4 and still having this issue. i tried @vcalmic 's configs but it still keep trying to reconnect. I tried to call disconnect() manually but look like it doesn't have affect.

willm · 2021-07-16T15:19:23Z

I'm also having this issue, I've tried using very similar config to @vcalmic but I still get ETIMEDOUT events to my client. I'm wondering if this is just a logging issue whether these errors can be ignored when you have reconnectOnError set up. If it helps I'm running on aws lambda and sharing the connection between executions.

juli0mendes · 2022-02-08T19:57:26Z

I'm also facing this problem

vaibhavp2710 · 2023-08-14T05:57:45Z

+1 facing performance issue due to this problem

ayeshasiddiqs · 2023-11-02T15:13:45Z

+1 facing same issue

vaibhavphutane · 2023-11-02T15:22:10Z

Encountered the identical problem and successfully resolved it by

const redisClient = new Redis({
        host: 'url',
        port: 6379,
        connectTimeout: 17000,
        maxRetriesPerRequest: 4,
        retryStrategy: (times) => Math.min(times * 30, 1000),
        reconnectOnError: (error)  => {
                const targetErrors = [/READONLY/, /ETIMEDOUT/];
               return targetErrors.some((targetError) => targetError.test(error.message));
       }
      });

nrathi · 2023-11-15T15:18:36Z

@vcalmic @vaibhavphutane question for you:

What does your error handling look like for this issue? I create the Redis client outside of my handler, so that it may be re-used.

Right now, my lambda is timing out at 45 seconds (because the connection is never made). Do you handle the connection error itself, or just let your lambda timeout? What is your lambda timeout?

stale bot added the wontfix label Feb 16, 2020

stale bot closed this as completed Feb 23, 2020

alexbielen mentioned this issue Dec 10, 2020

Bugfix: Redis connections hanging GSA/site-scanning-engine#51

Merged

BatuhanW mentioned this issue Jan 3, 2023

Replace ioredis with redis package OptimalBits/bull#1773

Closed

ygpark80 mentioned this issue Apr 23, 2024

Override reconnectOnError in tsed ioredis module tsedio/tsed#2668

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ETIMEDOUT connecting to AWS Elasticache (non-cluster) #1042

ETIMEDOUT connecting to AWS Elasticache (non-cluster) #1042

chrisfinch commented Jan 10, 2020 •

edited

Loading

chrisfinch commented Jan 10, 2020

simondutertre commented Jan 13, 2020

wjt382063576 commented Jan 14, 2020

chrisfinch commented Jan 14, 2020

chrisfinch commented Jan 14, 2020

vcalmic commented Jan 17, 2020 •

edited

Loading

stale bot commented Feb 16, 2020

aalimovs commented Mar 31, 2020

demian85 commented May 19, 2020

aalimovs commented May 20, 2020

demian85 commented May 20, 2020

ShraddhaMantri commented Aug 5, 2020

ColeTownsend commented Oct 16, 2020

demian85 commented Oct 22, 2020

CostaFernando commented Nov 22, 2020

Aung-Myint-Thein commented Jan 30, 2021

willm commented Jul 16, 2021

juli0mendes commented Feb 8, 2022

vaibhavp2710 commented Aug 14, 2023

ayeshasiddiqs commented Nov 2, 2023

vaibhavphutane commented Nov 2, 2023 •

edited

Loading

nrathi commented Nov 15, 2023 •

edited

Loading

ETIMEDOUT connecting to AWS Elasticache (non-cluster) #1042

ETIMEDOUT connecting to AWS Elasticache (non-cluster) #1042

Comments

chrisfinch commented Jan 10, 2020 • edited Loading

chrisfinch commented Jan 10, 2020

simondutertre commented Jan 13, 2020

wjt382063576 commented Jan 14, 2020

chrisfinch commented Jan 14, 2020

chrisfinch commented Jan 14, 2020

vcalmic commented Jan 17, 2020 • edited Loading

stale bot commented Feb 16, 2020

aalimovs commented Mar 31, 2020

demian85 commented May 19, 2020

aalimovs commented May 20, 2020

demian85 commented May 20, 2020

ShraddhaMantri commented Aug 5, 2020

ColeTownsend commented Oct 16, 2020

demian85 commented Oct 22, 2020

CostaFernando commented Nov 22, 2020

Aung-Myint-Thein commented Jan 30, 2021

willm commented Jul 16, 2021

juli0mendes commented Feb 8, 2022

vaibhavp2710 commented Aug 14, 2023

ayeshasiddiqs commented Nov 2, 2023

vaibhavphutane commented Nov 2, 2023 • edited Loading

nrathi commented Nov 15, 2023 • edited Loading

chrisfinch commented Jan 10, 2020 •

edited

Loading

vcalmic commented Jan 17, 2020 •

edited

Loading

vaibhavphutane commented Nov 2, 2023 •

edited

Loading

nrathi commented Nov 15, 2023 •

edited

Loading