Docker Swarm Mode? #927

ghost · 2017-09-14T12:30:01Z

Is there any support for Docker Swarm Mode yet?

marcwaz · 2017-10-09T09:25:02Z

I have the following errors in swarm mode on one node only:

upstream server temporarily disabled while connecting to upstream

connect() failed (110: Connection timed out) while connecting to upstream

Have you planned to support swarm mode ?

kennethklee · 2018-01-18T02:17:17Z

works for me on swarm mode, but only with the latest and alpine images since those are built from master.

it's possible the connection timeout is from the ingress network it tries to connect on. this was fixed back in August, hence, only available in the latest and alpine images

biels · 2018-02-21T20:15:17Z

@kennethklee Did you try it with the proxy and applications in different stacks (deployed using docker stack deploy -c docker-compose.yml). This would be the ideal use case because you can deploy and remove individual stacks without messing with the proxy stack.

arefaslani · 2018-03-07T11:23:38Z

@kennethklee Can you provide an example with compose file for a docker stack with multiple nodes?

arefaslani · 2018-03-10T07:51:01Z

@biels Could you run nginx-proxy in swarm mode with docker stack?

loic294 · 2018-03-10T21:04:46Z

@arefaslani Did you figure out how to make it work on multiple nodes in Swarm mode. It only seems to connect to services on the same node.

biels · 2018-03-10T21:17:43Z

I found a service called traefik that does exactly the same and seems to work a little bit better. However I'm seriously considering switching over to kubernetes.

…

On Mar 10, 2018 8:51 AM, "Aref Aslani" ***@***.***> wrote: @biels <https://github.com/biels> Could you run nginx-proxy in swarm mode with docker stack? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#927 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE64MCU0p8JWCc6n4BcRttH6QM6wr6rJks5tc4XrgaJpZM4PXgPe> .

arefaslani · 2018-03-11T11:54:19Z

@loic294 All I did was forking the project and manually change this line to server {{ .Network.IP }}:3000; and it works. But the problem is that my API runs on port 3000 and my frontend app server runs on 3001 and in this solution I can't have both.
After that I thought It's a good idea to use variables instead of hardcoding ports. I don't know much about go and it's templating language, but I thought $CurrentContainer in this line is a variable and change this line to server {{ .Network.IP }}:{{$CurrentContainer.Env.VIRTUAL_PORT}}. But docker-gen complains that $CurrentContainer is not a variable. If you found a better solution, notify us :)

loic294 · 2018-03-12T02:58:43Z

@arefaslani I almost changed everything for Kubernetes but got it to work with normal Nginx, but without this library. It's far from optimal but for my use case, it works great. In my docker_stack.yml file, I created a nginx service:

nginx:
    image: nginx:latest
    volumes:
      - ~/nginx/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
    networks:
      - proxy
    depends_on:
        - metabase
        - api
    deploy:
      placement:
        constraints: [node.role == manager]

The placement is important because I manually uploaded the nginx.conf file. There is probably a way to sync it between node but it was getting to complex. Now my services in the stack file look something like this (here is the example for metabase):

 metabase:
    environment:
      - ...
    image: 'metabase/metabase:latest'
    networks:
      - proxy
      - ...
    depends_on:
      - ...
    ports:
      - 3000:3000 <-- not sure this is required, but I was using it to test
    deploy:
      replicas: 1
      update_config:
        parallelism: 3
        delay: 10s
      restart_policy:
        condition: on-failure

Finally the nginx.conf look something like this:

user  www-data;

events {
    worker_connections  1024;
}

http {

	upstream api-app {
			server api:8500;
	}

	server {
			listen 80;
			server_name api.example.com;

			location /  {
					proxy_pass http://api-app;
					proxy_http_version 1.1;   <-- These lines allow websocket to pass.
					proxy_set_header Upgrade $http_upgrade;
					proxy_set_header Connection "upgrade";
					proxy_read_timeout 86400;
			}
	}
	upstream metabase-app {
			server metabase:3000;
	}

	server {
			listen 80;
			server_name metabase.example.com;

			location / {
					proxy_pass http://metabase-app;
			}
	}

}

The cool thing about this is that the upstream server uses the service name to resolve the host. It then uses the internal load balancer to redirect the request to the best container.

The super important part is that you need to expose the port of your app in the Dockerfile. For example, my api image has an EXPOSE 8500. This allows Nginx to know which port to listen to.

I really hope that nginx-proxy solves the issue because their solution is way cleaner.

arefaslani · 2018-04-23T13:23:44Z

Checkout my image. It's written in javascript and will be developed easily. Pull requests are welcome 🙂

arefaslani · 2018-05-23T12:50:00Z

docker-gen Network structure doesn't have Port property.

type Network struct {
    IP                  string
    Name                string
    Gateway             string
    EndpointID          string
    IPv6Gateway         string
    GlobalIPv6Address   string
    MacAddress          string
    GlobalIPv6PrefixLen int
    IPPrefixLen         int
}

So one solution is to change these lines in nginx-proxy template to support port through VIRTUAL_PORT env variable:

{{ else if .Network }}
    # {{ .Container.Name }}
    {{ if .Network.IP }}
        {{ if .Container.Env.VIRTUAL_PORT }}
            server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }};
        {{ else }}
-          server {{ .Network.IP }} down;
+         server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }}; 
        {{ end }}
    {{ else }}
        server 127.0.0.1 down;
    {{ end }}
{{ end }}

Then rebuild the image and use VIRTUAL_PORT env variable for setting upstreams in swarm mode, correctly. It worked well for me.

bfwg · 2018-07-06T18:55:38Z

Any updates?

zodern · 2018-07-06T19:30:32Z

I am working on a solution for Meteor Up: nginx-proxy-swarm-upstream.

arefaslani · 2018-07-11T06:05:56Z

@zodern This image works well with swarm mode. Formerly I created the nginx-autoconf to work with swarm mode. But then I realized that just by changing the original template and rebuilding, this image works well in swarm mode.

zodern · 2018-07-12T02:49:02Z

Thanks @arefaslani. That should work for most apps.

There are two main problems that nginx-proxy-swarm-upstream solves:

Adds support for sticky sessions, which requires each task's ip address to be listed as an upstream server in the nginx config.
Allow nginx-proxy and the service's tasks to be on different servers

arefaslani · 2018-07-12T21:59:01Z

@zodern I don't know what sticky session is, but for problem #2 I use nginx proxy and service tasks in different servers (tested in 2 different machines created by docker machine in a swarm)

bfwg · 2018-07-12T23:13:20Z

I end up wrote a customize nginx.conf file alone with nginx:latest container. It works perfectly for swarm mode.

radyz · 2018-08-18T23:32:10Z

@arefaslani , care to share examples of your service's configurations to make it work in swarm mode accross different nodes :)

arefaslani · 2018-08-19T11:40:29Z

Configuration

Build your custom Nginx Proxy

Create a Dockerfile like this:

FROM jwilder/nginx-proxy
COPY nginx.tmpl nginx.tmpl

Copy nginx-proxy's nginx.tmpl alongside your docker file and change it as this comment. Then name and build your new proxy with the new template file.

Docker Compose file

version: "3.1"

volumes:
  nginx-certs:
  nginx-vhosts:
  nginx-letsencrypt-challenge:

services:
  app:
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -b 'ssl://0.0.0.0:3000?key=/etc/nginx/certs/beta.chanka.ir.key&cert=/etc/nginx/certs/beta.api.chanka.ir.crt'"
    volumes:
      - nginx-certs:/etc/nginx/certs
    ports:
      - 3000:3000
    environment:
      - RAILS_ENV=production
      - VIRTUAL_HOST=example.com
      - VIRTUAL_PORT=3000
      - VIRTUAL_PROTO=https
      - LETSENCRYPT_HOST=example.com
      - LETSENCRYPT_EMAIL=your@email.com

  nginx-proxy:
    image: your/nginx-proxy:latest
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - nginx-certs:/etc/nginx/certs:ro
      - nginx-vhosts:/etc/nginx/vhost.d
      - nginx-letsencrypt-challenge:/usr/share/nginx/html
    labels:
      - com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy

  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion:v1.8.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - nginx-certs:/etc/nginx/certs
      - nginx-vhosts:/etc/nginx/vhost.d
      - nginx-letsencrypt-challenge:/usr/share/nginx/html

Like the example above add VIRTUAL_PORT to your service. Note that I added Letsencrypt conf just because everybody wants to use it. the next thing is that I've added nginx-certs volume to the Rails app because Puma needs certs to start with HTTPS.

How does it work

NOTE: All the things I write next is just to clarify what does these configs if you don't understand some parts. If you understand it, go and config yours.

Volumes

There are 3 volumes in this compose file:

nginx-certs: Let's Encrypt certificates
nginx-vhosts: configuration of vhosts
nginx-letsencrypt-challenge: Letsencrypt challenge files.

Services

There are 3 services in this compose file:

app: The application. In this case a Rails application with Puma web server.
- Environment Variables:
  - RAILS_ENV: set to production to force Puma to run in production mode.
  - VIRTUAL_HOST: Name of the virtual host (your domain name)
  - VIRTUAL_PORT: The port in which your application runs
  - VIRTUAL_PROTP: Set to https to force your app to serve https traffic.
  - LETSENCRYPT_HOST: Must be equal to VIRTUAL_HOST. Letsencrypt uses this variable to generate certs for the specified domain.
  - LETSENCRYPT_EMAIL: Letsencrypt uses this email to send notifications about your certs
- Volumes:
  - nginx-certs: certificates generated by letsencrypt. Puma needs these certs to start in ssl mode. If you don't use Puma or Rails, This volume is useless.
nginx-proxy: The reverse proxy.
- Volumes:
  - nginx-certs: To use certs generated by Letsencrypt
  - nginx-vhosts: Configuration of vhosts
  - nginx-letsencrypt-challenge: To serve chanllenge files to letsencrypt
letsencrypt:
- Volumes:
  - nginx-certs: To create/renew Let's Encrypt certificates
  - nginx-vhosts: To change the configuration of vhosts (needed by Let's Encrypt)
  - nginx-letsencrypt-challenge: To write challenge files.

jamiejackson · 2018-09-06T15:42:34Z

@arefaslani , thanks for sharing your work.

However, your patch has drifted from the current state of the repo (which changed in this commit):

Your patch:

{{ else if .Network }}
    # {{ .Container.Name }}
    {{ if .Network.IP }}
        {{ if .Container.Env.VIRTUAL_PORT }}
            server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }};
        {{ else }}
-          server {{ .Network.IP }} down;
+         server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }}; 
        {{ end }}
    {{ else }}
        server 127.0.0.1 down;
    {{ end }}
{{ end }}

Current (link pinned to a revision, for posterity):

	{{ else if .Network }}
		# {{ .Container.Name }}
		{{ if .Network.IP }}
			server {{ .Network.IP }} down;
		{{ else }}
			server 127.0.0.1 down;
		{{ end }}
	{{ end }}

How does this change figure in?

jamiejackson · 2018-09-07T01:01:47Z

This, based on @arefaslani's solution, seems to work so far: https://gist.github.com/jamiejackson/bac3a440975d1cd98c2d19cd96da1b56

It uses a patch in an attempt to be flexible with regard to updates of the base image.

ghost · 2018-10-10T10:59:36Z

Has someone successfully implemented it with multiple node? For me when nginx-proxy needs to contact another node, it ends in a 503 Service Temporarily Unavailable.
Anyone reading this should also that a look there: #520

RazaGR · 2018-12-01T04:16:23Z

@jamiejackson @arefaslani does it also works with Kubernetes?

arefaslani · 2018-12-01T05:52:26Z

@Divxtaman Unfortunately I didn't test it with Kubernetes. I've tested it with Docker Swarm.

arefaslani · 2018-12-01T05:56:32Z

@jamiejackson Sorry for the late response. Yes I created this patch for myself almost a year ago. Thank you for updating it. But I think it's a better solution to create a pull request to solve this issue once for all. How about that?

orangelynx · 2019-05-10T18:29:59Z

I seem to have the same issue on a single node. I have a pydio container connected with an ssl backend on port 443 (set the VIRTUAL_PORT and VIRTUAL_PROTO env vars accordingly). Works fine in docker compose but with docker stack deploy it fails to find a proper upstream (config shows the proper upstream but with a "down" flag).

Likely has to do with "expose" being deprecated in docker swarm (ignored by docker stack deploy) and nginx failing to fall back on VIRTUAL_PORT.

FabianSchurig · 2019-07-19T19:11:06Z

Got stuck with the same issues, finally used traefik.

93Kamuran · 2019-08-18T06:50:26Z

I was allready using Traefik as a reverse proxy traefik was perfect. But I occurred an issue with using signalr with traefik than I decided to go back to Nginx however I tried either solutions of @jamiejackson and @arefaslani. I could not achieve to run the proxy in my cluster (swarm mode).

tunailgaz · 2019-10-09T08:49:05Z

docker-gen Network structure doesn't have Port property.

type Network struct {
    IP                  string
    Name                string
    Gateway             string
    EndpointID          string
    IPv6Gateway         string
    GlobalIPv6Address   string
    MacAddress          string
    GlobalIPv6PrefixLen int
    IPPrefixLen         int
}

So one solution is to change these lines in nginx-proxy template to support port through VIRTUAL_PORT env variable:

{{ else if .Network }}
    # {{ .Container.Name }}
    {{ if .Network.IP }}
        {{ if .Container.Env.VIRTUAL_PORT }}
            server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }};
        {{ else }}
-          server {{ .Network.IP }} down;
+         server {{ .Network.IP }}:{{ .Container.Env.VIRTUAL_PORT }}; 
        {{ end }}
    {{ else }}
        server 127.0.0.1 down;
    {{ end }}
{{ end }}

Then rebuild the image and use VIRTUAL_PORT env variable for setting upstreams in swarm mode, correctly. It worked well for me.

you saved my time that template looking monkaS to debug EZClap

this is my custom fork https://github.com/tunailgaz/nginx-proxy repo working fine

from evertramos/docker-compose-letsencrypt-nginx-proxy-companion

bkraul · 2019-10-09T09:37:05Z

Merge into master for this would be great!

daveteu · 2021-03-12T07:24:48Z

Apologise if I revive this here but many post referenced this post. The current version of nginx.tmpl does look different from this.

Anyone have a working version? With the current version, the proxy is still showing only the local connected container, instead of all the replica containers in the swarm.

bkraul · 2021-03-12T12:54:24Z

@daveteu this PR has never been merged into the project. As the Japanese would say, akiramete (give up) ...Seems the project maintainer would rather this project fall into obscurity than to accept contributions from the community. I switched to Traefik for swarmed environments. While it is a beast to understand at first, it works perfectly under swarms.

daveteu · 2021-03-12T12:59:30Z

thanks @bkraul , didn't know this project is outdated. I wanted nginx to dynamically add upstreams so I can easily scale without touching and changing the "upstream" bit + caching. Traefik fills over complicated and i still need to add nginx nevertheless to do the caching bit (where i have custom lua to do purging).

I wonder if this will work for kubernetes. i just haven't moved to kubernetes because of the VPS requirements.

orangelynx · 2021-03-12T13:04:08Z

@daveteu Not to do any unfair advertising, but +1 for traefik. I too switched to that project after finding no workarounds for problems with this project and traefik is powerful but lightweight, so don't worry about "overkill". I literally just use it for reverse proxying ~10 containers on my personal VPS and it hasn't let me down in the slightest so far.

daveteu · 2021-03-12T18:39:45Z

@orangelynx @bkraul thank u, after struggling for a bit, managed to get traefik up and running! May I ask, if I want to put up a nginx cache infront of traefik, my nginx upstream proxy_pass can be any of my 3 servers IP that are behind traefik am i right? The swarm will auto load balance so long the host matches service-name.loadbalancer ?

bkraul · 2021-03-12T18:54:55Z

@daveteu, I am not sure what kind of cache you need, but traefik supports a rather large number of what's called Middlewares to achieve a number of purposes. Have you looked into this?

That said what you described sounds feasible.

jamiejackson · 2021-06-01T20:05:05Z

Does this fix the issue? 97a5dec?branch=97a5dec57a6e33a3d4dd93ab22a52c0fffcc0872&diff=split

It definitely breaks my old patch so I'm hoping I can go without the patch now.

tkw1536 · 2022-04-10T17:01:07Z

Duplicate of #97

arefaslani mentioned this issue Mar 10, 2018

Swarm mode support feasibility #1093

Closed

This was referenced May 23, 2018

Setup with Swarm Using Docker 1.12.x #738

Closed

[Swarm Mode 1.12] Improvements #520

Closed

blackandred added a commit to riotkit-org/riotkit-harbor that referenced this issue Jun 4, 2020

[nginx-proxy/nginx-proxy#927 (comment)] Swarm mode fix

3269807

bugficks pushed a commit to bugficks/nginx-proxy that referenced this issue Jul 16, 2020

Docker Swarm Mode? nginx-proxy#927 (comment)

eefb05e

tkw1536 marked this as a duplicate of #97 Apr 10, 2022

tkw1536 closed this as completed Apr 10, 2022

tkw1536 added the status/duplicate This issue / PR is a duplicate of another one label Apr 10, 2022

Docker Swarm Mode? #927

Docker Swarm Mode? #927

Comments

ghost commented Sep 14, 2017

marcwaz commented Oct 9, 2017

kennethklee commented Jan 18, 2018 • edited Loading

biels commented Feb 21, 2018

arefaslani commented Mar 7, 2018

arefaslani commented Mar 10, 2018

loic294 commented Mar 10, 2018

biels commented Mar 10, 2018 via email

arefaslani commented Mar 11, 2018 • edited Loading

loic294 commented Mar 12, 2018 • edited Loading

arefaslani commented Apr 23, 2018

arefaslani commented May 23, 2018

bfwg commented Jul 6, 2018

zodern commented Jul 6, 2018

arefaslani commented Jul 11, 2018

zodern commented Jul 12, 2018 • edited Loading

arefaslani commented Jul 12, 2018

bfwg commented Jul 12, 2018

radyz commented Aug 18, 2018

arefaslani commented Aug 19, 2018

Configuration

Build your custom Nginx Proxy

Docker Compose file

How does it work

Volumes

Services

jamiejackson commented Sep 6, 2018

jamiejackson commented Sep 7, 2018 • edited Loading

ghost commented Oct 10, 2018 • edited by ghost Loading

RazaGR commented Dec 1, 2018

arefaslani commented Dec 1, 2018

arefaslani commented Dec 1, 2018

orangelynx commented May 10, 2019 • edited Loading

FabianSchurig commented Jul 19, 2019

93Kamuran commented Aug 18, 2019

tunailgaz commented Oct 9, 2019 • edited Loading

bkraul commented Oct 9, 2019

daveteu commented Mar 12, 2021 • edited Loading

bkraul commented Mar 12, 2021

daveteu commented Mar 12, 2021 • edited Loading

orangelynx commented Mar 12, 2021

daveteu commented Mar 12, 2021

bkraul commented Mar 12, 2021 • edited Loading

jamiejackson commented Jun 1, 2021

tkw1536 commented Apr 10, 2022

kennethklee commented Jan 18, 2018 •

edited

Loading

arefaslani commented Mar 11, 2018 •

edited

Loading

loic294 commented Mar 12, 2018 •

edited

Loading

zodern commented Jul 12, 2018 •

edited

Loading

jamiejackson commented Sep 7, 2018 •

edited

Loading

ghost commented Oct 10, 2018 •

edited by ghost

Loading

orangelynx commented May 10, 2019 •

edited

Loading

tunailgaz commented Oct 9, 2019 •

edited

Loading

daveteu commented Mar 12, 2021 •

edited

Loading

daveteu commented Mar 12, 2021 •

edited

Loading

bkraul commented Mar 12, 2021 •

edited

Loading