Docker, Swarm mode, LXC, and you

I currently run a lot of containers on very specific machines and manage all of it with compose. What if I didn't have to care about where something was running? Let's try.

To start, I spun up three new instances (I use debian 12 lxcs running on Proxmox, but you can do this however you'd like). I like to use these helper scripts these days to make my life easier.

Then on the one I like the best, I ran:

$ docker swarm init
Swarm initialized: current node (my-node) is now a manager.

To add a worker to this swarm, run the following command:

$ docker swarm join \
--token a-long-token \

To add a manager to this swarm, run 'docker swarm join-token manager'
 and follow the instructions.

On the other two, I simply copied and pasted the command that gave me:

$ docker swarm join \
    --token a-long-token \

Cool, okay, they're all magically connected. Let's try to run something like Portainer. Following their guide, on the manager node, I ran:

$ curl -L https://downloads.portainer.io/ee2-19/portainer-agent-stack.yml -o portainer-agent-stack.yml
$ docker stack deploy -c portainer-agent-stack.yml portainer

This completed successfully and I should be able to access it on the manager node...right?

$ curl
curl: (7) Failed to connect to port 9000 after 0 ms: Couldn't connect to server

Hmm. (If you've done this before, maybe you noticed an issue with the IP addresses in play?)

Subnet overlap

From the docs:

By default Docker Swarm uses a default address pool for global scope (overlay) networks. Every network that does not have a subnet specified will have a subnet sequentially allocated from this pool. In some circumstances it may be desirable to use a different default IP address pool for networks.

That's important because so does my personal network. I need to make sure there isn't overlap between them or things won't work:

# First make sure nothing is running from the manager node:
$ docker stack rm portainer

# Then on each worker node
$ docker swarm leave
# And on the manager node
$ docker swarm leave --force
# For good measure I also removed the docker gateway bridge network
$ docker network rm docker_gwbridge
# Don't worry, it gets recreated when doing swarm init

Then set things back up again on the manager node:

$ docker swarm init --default-addr-pool

And then use the generated join command on the worker nodes.

To test, we'll run something simple, like nginx, on all three nodes:

$ docker service create \
    --mode global \
    --publish mode=host,target=80,published=8080 \
    --name=nginx \

And then test that it's working from any of our nodes:

$ curl localhost:8080

Or load it in our browser using the ip of any swarm node and the port 8080

Great! mode=host works. (Note the lack of ports)

$ docker service ls
ID             NAME      MODE      REPLICAS   IMAGE          PORTS
udq3pgcwv8b6   nginx     global    3/3        nginx:latest

Let's remove that service and try it not in host mode:

$ docker service rm nginx
$ docker service create --name test \
        --replicas 3 \
        --publish published=8080,target=80 \
$ docker service ls
ID             NAME      MODE         REPLICAS   IMAGE          PORTS
kaduhaktjc5h   test      replicated   3/3        nginx:latest   *:8080->80/tcp
$ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server

Let's scream into the void (aka, keep searching online)

The LXC Issue

Apparently there are enough differences in how lxcs behave that sometimes docker swarm doesn't quite work right. This is annoying not documented (yes, running docker in lxcs is not exactly the most supported scenario, but up until now it's worked pretty great)

From what I can tell, we need to do two things to make this work:

  • Ensure your containers are running as privileged
  • Allow ipv4 forwarding to the ingress network

Since I'm running Proxmox, I changed my lxcs to run as privileged by stopping the container, editing /etc/pve/lxc/container-id.conf on the host, and setting unprivileged: 0 at the bottom of that and starting it up again.

Then to enable forwarding, on each node in your swarm, run this:

$ nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

Confirm all of that is working:

    $ curl localhost:8080
    <!DOCTYPE html>
    <title>Welcome to nginx!</title>
    html { color-scheme: light dark; }
    body { width: 35em; margin: 0 auto;
    font-family: Tahoma, Verdana, Arial, sans-serif; }
    	...default nginx body

Sweet. Open that in a browser on your local machine using any node ip address and you should see the nginx page.

It works!

But if you reboot, it won't until you run the command again. @drallas has a good github gist with how to handle it so do that (sigh, on every node) and you should be good to go.