Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Redis ENOTFOUND over private network (ipv6) on Railway #13117

Closed
antonlvovych opened this issue Feb 7, 2025 · 6 comments · Fixed by #13118
Closed

Redis ENOTFOUND over private network (ipv6) on Railway #13117

antonlvovych opened this issue Feb 7, 2025 · 6 comments · Fixed by #13118
Labels
in linear Issue or PR has been created in Linear for internal review

Comments

@antonlvovych
Copy link

antonlvovych commented Feb 7, 2025

Bug Description

When deploying n8n in a queue mode on Railway without publicly exposed Redis, the n8n instances crash with the following error:

[Redis client] getaddrinfo ENOTFOUND redis.railway.internal

This issue occurs when n8n attempts to connect to Redis using Railway’s private network (redis.railway.internal).
The root cause appears to be ioredis not supporting IPv6, which is used for private networking in Railway.

Additional Context

Potential Fix:
According to a Railway employee response, n8n does not support connecting to Redis via the private network due to upstream ioredis misconfigurations (it doesn't resolve IPv6, which is used by Railway in a private network).

If that's the case, it could be fixed by adding an option for ioredis dual stack lookup, which will try to resolve the redis.railway.internal hostname using both IPv4 and IPv6 addresses (A and AAAA records).

Current workaround:
Update Redis connection settings in n8n to use the public Redis URL instead of private (redis.railway.internal).

⚠️ Security Risk: Using the public network exposes Redis unnecessarily, which is not a good practice and should be avoided unless necessary.

To Reproduce

  1. Deploy this template to Railway:

Deploy on Railway

  1. Check deploy logs of Primary or Worker n8n instance.

Both instances fail with the following error:

[Redis client] getaddrinfo ENOTFOUND redis.railway.internal

Expected behavior

n8n should successfully connect to Redis without encountering ENOTFOUND redis.railway.internal, resolving the hostname via both IPv4 and IPv6.

Operating System

N/A

n8n Version

latest

Node.js Version

N/A

Database

PostgreSQL

Execution mode

queue

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2025

Hey @antonlvovych,

We have created an internal ticket to look into this which we will be tracking as "CAT-627"

@Joffcom Joffcom added the in linear Issue or PR has been created in Linear for internal review label Feb 7, 2025
@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2025

Hey @antonlvovych

Thanks for the docs I will pass this on internally for the team to look into, it makes sense to do this however it is more of an enhancement request than a bug so while I should close this and directly it to the support forum it feels like something that should remain here for now.

I just wanted to note that the person that told you this does not work for n8n but railway, I am sure it is just a typo but this should save time for anyone else who looks at this and wonders who it might have been 🙂

@antonlvovych
Copy link
Author

antonlvovych commented Feb 7, 2025

@Joffcom yeah, that was a typo — it’s a Railway employee. I just faced this issue today and found this discussion. It looks valid, so it would be great if you could add support for IPv6 networks. PostgreSQL works just fine using an IPv6 private network on Railway.

I see there is already a PR with a fix by @netroy ❤️‍🔥🚀 Can we keep this issue open for now?

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2025

Hey @antonlvovych ,

Of course, I raised this with Netroy after posting. I just need to review it then we can merge and release it as part of the next release.

@antonlvovych
Copy link
Author

@Joffcom @netroy thanks a ton for the super quick upgrade!

@netroy
Copy link
Member

netroy commented Feb 7, 2025

Starting next release, you can set the env variable QUEUE_BULL_REDIS_DUALSTACK to true to enable IPv6 support.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
in linear Issue or PR has been created in Linear for internal review
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants