How to use nginx as a revese proxy with docker containers

Introduction

In this tutorial we will explain how to use nginx as a reverse proxy to provide a load balance solution with more than one container.

A reverse proxy is not only used for load balance, it could be used for caching, compression and many other things.

Using a reverse proxy has the advantage of no down time when doing production updates and makes the life easier when using docker containers as we will see.

Requirements

For this tutorial we require:

Step 1: Install docker-gen

Before starting we need to know who we are going to solve the problem of random IPs and ports assigned to containers. This problem could be an issue when containers are started or stopped and to solve it we need an automatic tool to help us. In particular we will use (docker-gen)[https://github.com/jwilder/docker-gen] that will generate nginx configuration automatically. Everytime the nginx configuration files changes we will reload nginx to make the update available.

First install docket-gen with:

wget https://github.com/jwilder/docker-gen/releases/download/0.7.3/docker-gen-linux-amd64-0.7.3.tar.gz
atar xf docker-gen-linux-amd64-0.7.3.tar.gz
./docker-gen

Step 2: Install nginx as a reverse

This step will be the easier of all. If you already have installed nginx as it was required you don't have to do anything else since docket-gen will the responsible for generate and reload nginx configs.

Step 3: Create a docker-gen template for nginx

Create a new file with the template for nginx reverse proxy configuration (we will use template.tmpl as filename):

{{ range $host, $containers := groupBy $ "Env.VIRTUAL_HOST" }}
upstream {{ $host }} {

{{ range $index, $value := $containers }}
    {{ with $address := index $value.Addresses 0 }}
    server {{ $address.IP }}:{{ $address.Port }};
    {{ end }}
{{ end }}

}

server {
    #ssl_certificate /etc/nginx/certs/demo.pem;
    #ssl_certificate_key /etc/nginx/certs/demo.key;

    gzip_types text/plain text/css application/json application/x-javascript
               text/xml application/xml application/xml+rss text/javascript;

    server_name {{ $host }};

    location / {
        proxy_pass http://{{ $host }};
        include /etc/nginx/proxy_params;
    }
}
{{ end }}

The template will generate a load-balance backend and it provides zero downtime deployments. In the template is assumes that all the docker containers are running web services that can be load balanced. It will iterate through all containers to obtain the IP and the port and it will generate the a load balance configuration.

Step 4: Executing docker-gen

We are ready to execute docker-gen command to watch containers en reload nginx configuration:

./docker-gen -only-exposed -watch -notify "/etc/init.d/nginx reload" template.tmpl /etc/nginx/sites-enabled/default

Note that it will override the configuration file default, change this to your's site configuration file. The parameters of docker-gen were adjusted to watch and reload nginx, the only-exposed is to filter containers that have exposed ports. Remeber that docker-gen don't know what is inside each container, so every container must be configured to be loaded balanced.

Step 5: Nginx proxy params configuration

At this step we are going to create the file /etc/nginx/proxy_params, which handles the configuration to setup a reverse proxy in our case.

Just add the following content to the /etc/nginx/proxy_params

proxyset_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;Step 5: Check everything is working

This configuration specify which information to pass from the original request to the balanced hosts. You can read more here about the reverse proxy configuration.

Step 6: Create docker containers

Load balance a "Hello world" doesn't make any sense, but we do this test just to test the load balancer. The docker for testing is a minimal flask template that will print the IP address and port of the service. We will add three of them.

git clone https://github.com/llazzaro/flask-docker-print-host-port.git
docker build -t print-host-app:latest .
docker run -d -p 9001:5000 -e VIRTUAL_HOST=yourserver.com print-host-app
docker run -d -p 9002:5000 -e VIRTUAL_HOST=yourserver.com print-host-app
docker run -d -p 9003:5000 -e VIRTUAL_HOST=yourserver.com print-host-app

Step 7: Varify and Testing of the load balancer

Test the ports 9001, 9002 and 9003 with wget or curl and verify that each one return a different APP_ID. The output should looks like this:

Running at 192.168.116.73:9003. APP_ID=533490

You can also cat your /etc/nginx/sites-enables/default to see the contents:

upstream testserver.com {
    server 172.17.0.9:5000;
    server 172.17.0.8:5000;
    server 172.17.0.7:5000;
}
server {
    #ssl_certificate /etc/nginx/certs/demo.pem;
    #ssl_certificate_key /etc/nginx/certs/demo.key;
    gzip_types text/plain text/css application/json application/x-javascript
               text/xml application/xml application/xml+rss text/javascript;
    server_name testserver.com;
    location / {
        proxy_pass http://testserver.com;
        include /etc/nginx/proxy_params;
    }
}

Use curl (or open any browser) to verify that APP_ID is different on some requests: If you are doing tests, you can add an entry in your /etc/hosts for the testserver.com.

curl testserver.com
curl testserver.com
curl testserver.com
curl testserver.com

The output should be similar to this one:

Running at testserver..com. APP_ID=482562
Running at testserver.com. APP_ID=647829
Running at testserver.com. APP_ID=717720
Running at testserver.com. APP_ID=482562
...

That's it! you have configured a nginx server with docker workers to balance the work.