All Posts

Connectivity between Nginx and Consul-Template

by gusemery
tags: shipping code

Overview

In the past couple of weeks, I have been wrestling with the connectivity between Nginx and Consul-Template.  While it’s fairly straightforward to get Consul-Template to control Nginx, having them work well within Docker isn’t the easiest thing in the world.  But that is jumping quite a bit forward, let’s start with the problem before I get to the solution.

 

Problem

The project that we are working on currently is a retro-fit of a monolith system that we are currently breaking up into many microservices. Breaking up the services allows us  to be able to control our environment, and allows easy publishing of the services in comparison to publishing a large and very integrated monolithic piece of software.

The biggest issue with a monolith is upgrading it without interfering with your production uptime SLA.  Remember, in most cases when you publish a monolith, at some point, all of your service endpoints will be unavailable for a period of time.  To do this the correct way would be to safely shut down all of your services, install the new version, and then spin up the new service. When doing this, no matter how big and awesome your servers are, you will have an OUTAGE….

 

Solution

By moving to a microservice architecture you open up the ability to institute blue-green deployment patterns. This pattern allows you to start a deployment, and just add a couple of the new revisions into your online group as a final test to make sure you aren’t seeing errors, and then you may turn off the old, and the new just take over.   All of this is done seamlessly via a load-balancer. To complete this solution we had to research and pick some tools to accomplish our architecture goals.

We chose the following tools:

This set of tools allows us to cross off all of our items that we wanted to accomplish in complete harmony. We were also able to have these components available on our development boxes with the use of Docker.  Docker allows the bundling of an ‘application’ to be combined with all of it’s dependencies within a ‘container’.  The container is similar to a virtual machine. To install a docker container you must first have a Docker-Machine on which to install your items. For further information on what the chosen tools do, please follow the links above.

I’ll explain quickly; however not too deeply what each tool above does. Consul is a service registry container: services register themselves here with their IP and port to allow other services to find them. Nginx is a web server that also allows a single port to re-direct and encompass simple load-balancing. Consul-Template polls Consul and updates the Nginx configuration to incorporate any changes within Consul’s service registry. Registrator is an automatic registration service for Docker. When any containers status changes, the containers status will be updated within Consul.  I use this so that the service that will be tested later will automatically register within Consul, saving a lot of extra busy work.

 

Base Installation

Before attempting any of the below, please make sure you have the following software installed.  And know that this demo is built upon Windows 10 and the commands will change when moving to another platform.

To install all of this via Chocolatey you can copy and paste the following command.

PS e:> choco install docker docker-machine virtualbox -y

Alternatively, you can install all this via Docker Toolbox.

 

GitHub – Local Setup

I have created a GitHub repository for this blogpost.  Please clone this link and go through the following quick steps.

Go to the cloned directory and execute the following command:

PS E:> ,.CreateContainer.ps1

The script will create your VM, install all of the containers and start Consul-Template in ‘DRY’ mode.  Within this mode you can see when changes are made when services are started and stopped.  You should see this within the PowerShell window:

upstream app {
  least_conn;
  server 127.0.0.1:65535; # force a 502    
}

server {
  listen 80 default_server;

  location / {
    proxy_pass http://app;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

At the top this shows that the server entry points to port 65535 which will force a 502 error; this means that there are no services that match what Consul-Template is looking for (see Consul-Template File below).

Now open a new PowerShell (Admin) window and execute the second script.

PS E:> ./AddService.ps1

This adds a container called ‘Service1’ to the docker-machine, upon which Registrator creates the proper registration of the service within Consul.  You should see the following appear within the first PowerShell window.

upstream app {
  least_conn;
  server 127.0.0.1:8081;    
}
server {
  listen 80 default_server;

  location / {
    proxy_pass http://app;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

You’ll notice that the server entry at the top changed.  This is pointing to the localhost or Docker Machine and to the port (8081) that will allow you to hit the service.  Using a browser view this url.

The internal (127.0.0.1) URL from your machine should be (192.168.99.100) when hitting from a browser. (see image below)

docker started!

Next execute the following command in the second PowerShell window

PS E:> docker stop service1

At this point you should again see an update within the first window. Again the line stating #force a 502 should be shown.  This means that the service is not available anymore, and you can now validate that within the browser by refreshing the window. (image below)

docker stopped!

This shows that Consul-Template can institute a new route under NginX for any update to the services registration.

Now let’s restart that service, and start a second version of the service.

PS E:> docker start Service1
PS E:> ./AddService.ps1

You should see two entries within the first window, and it should show ports 8081 and 8082 redirecting to port 80.  This setup will now do round-robin load balancing.

Now let’s do this for real!  In the window that has been displaying the conf files, hit CTRL-C..  This will abort the current execution; which was Consul-Template running in DRY or display mode. Now run this command:

PS E:> ./StartConsul-Template.ps1

Now you should see:

[ ok . Reloading nginx: nginx

This means that the actual .CONF file was written to NginX and was consequently loaded.

Now you should be able to hit the services from the single IP address and port:  http://192.168.99.100

This should get you running.   If you want to try this step by step, follow the below instructions.

 

 

Step by Step Setup – Windows 10

Let’s setup our docker-machine. I’m using PowerShell running as Administrator.

 

Setup Docker-Machine

First off let’s setup the Virtual Machine that we will run all of our containers or mini-VM’s. Execute the following statements. The second statement will setup and register the correct Certificates to allow you to run Docker commands against the Virtual Machine.

PS E: > docker-machine create --driver virtualbox default
PS E: > docker-machine env --shell powershell default | invoke-expression

We will be utilizing the machine name default as the name of our machine.  I’m currently only running this VM or Docker-Machine on my box, and the IP address that is assigned should be 192.168.99.100. If this were not the only machine running this IP address could change, and cause issues further on in the directions. To check to see what IP address your machine was assigned, utilize the following command. Replace any use of 192.168.99.100 to your IP address if it is different.

PS E:> docker-machine ip default

NOTE :  When attempting to talk to a Docker-Machine from a new instance of CMD/PS you may see An error occurred trying to connect: Get http://localhost:2375/v1.21/containers/json: dial tcp 127.0.0.1:2375: ConnectEx tcp: No connection could be made because the target machine actively refused it.  This means that you need to re-run the ENV command to re-setup your certificates.

 

Setup Consul

The next step is to get Consul running inside of a container within your Docker VM. There are many ports that are required for Consul to function. Please see the consul documentation here for more information on the use of the ports.

PS E:> docker run --name consul -d -p 8300-8302:8300-8302 -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1 progrium/consul -server -bootstrap -advertise 192.168.99.100

You should see some dialog and transfers happen, after this is installed we need to  test to make sure Consul installed properly. This is our first step to make sure that we have things running correctly.

PS E:> curl http://192.168.99.100:8500/v1/catalog/nodes
StatusCode        : 200
StatusDescription : OK
Content           : [{"Node":"freedom","Address":"10.0.0.177","CreateIndex":0,"ModifyIndex":0},{"Node":"node1","Address":"192.168.99.100","CreateIndex":0,"ModifyIndex":0}]
RawContent        : HTTP/1.1 200 OK
                   X-Consul-Index: 5
                   X-Consul-Knownleader: true
                   X-Consul-Lastcontact: 0
                   Content-Length: 151
                   Content-Type: application/json
                   Date: Thu, 10 Mar 2016 16:29:10 GMT
                   [{"Node":"freedom",
Forms             : {}
Headers           : {[X-Consul-Index, 5], [X-Consul-Knownleader, true], [X-Consul-Lastcontact, 0], [Content-Length,151]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 151

Seeing the above readout, this is coming from Consul, and telling us that the node that we just installed is functional.

 

Setup NginX

Now we need to install Nginx to to host our Consul-Template instance, and our entry-point to our test services.

PS E:> docker run -d --name Nginx -p 8080:80 -t nginx

At this point we need to test and make sure that Nginx loaded correctly.

PS E:> curl http://192.168.99.100:8080
StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                   <html>
                   <head>
                   <title>Welcome to Nginx!</title>
                   <...
RawContentLength  : 612

 

Setup Consul-Template

Now that we know that Nginx is running (Note “title” element above).  We need to setup Consul-Template in the Nginx container.

PS E:> docker cp consul-template nginx:/usr/local/bin
PS E:> docker exec -it mkdir /templates
PS E:> docker cp ping.ctmpl nginx:/templates/ping.ctmpl

 

Consul-Template File

At the top of this file the top line is specifying the collection ‘app’ will contain a range of ‘hello-world’ services that have been registered. This will allow the registration of any registered container into the following NginX template.  When a service registers or unregisters this template will be written.  IF no services matching ‘hello-world’ are found, then the route is re-directed to port 65535 which causes a 502 Error.

upstream app {
 least_conn;
 {{range service "hello-world” }}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
 {{else}}server 127.0.0.1:65535; # force a 502{{end}}
}

server {
 listen 80 default_server;

 location / {
   proxy_pass http://app;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $host;
   proxy_set_header X-Real-IP $remote_addr;
 }
}

Now start the consul-template service with the following command. First open a second PowerShell window. This will allow you to start and stop the service in the next steps, and validate the changes with Consul-Template.

PS E:> docker exec -t nginx /usr/local/bin/consul-template -consul 192.168.99.100:8500 -template "/templates/ping.ctmpl:/etc/nginx/conf.d/ping.conf:service nginx reload" -pid-file /var/run/consul-template.pid -log-level warn -dry

The output from this file will look like the prior template file; however will be slightly different. It’s written as a .conf file that Nginx will read and allow the server node in the second half to contain any registered service and route port 80 to that site in a round robin fashion.  The output looks like the following.

Service Running
upstream app {
  least_conn;
  server 127.0.0.1:8081 max_fails=3 fail_timeout=60 weight=1;
}

server {
  listen 80 default_server;

  location / {
    proxy_pass http://app;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}
Service Not Running
upstream app {
  least_conn;
  server 127.0.0.1:65535; # force a 502
}

server {
  listen 80 default_server;

  location / {
    proxy_pass http://app;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

 

Service Start

The final thing that you need to do is start a service upon the docker-machine to have Consul-Template execute and you will see the output in your powershell window.   The prior function should hold the prompt; hence Consul-Template is still running… Open a new PowerShell window and execute the following two statements.

PS E:> docker-machine env --shell powershell default | invoke-expression
PS E:> docker run -d --name service1 -p 8081:80 tutum/hello-world

You will remember the first command, again that sets up the certificates for your Docker-Machine connection, the second will download and execute the tutum/hello-world container. At the time that this comes online, you should see the other window update it’s config.  This should look like the service running file <above>.

Now let’s stop this service, and see Consul-Template change the file back to the 502

PS E:> docker stop service1

You should have seen the template show up in the other window specifying that the service has been stopped.

 

Test

Just as a test, let’s restart service1 and add a service.

PS E:> docker start service1

Now let’s add the second service.

PS E:> docker run -d --name service2 -p 8082:80 tutum/hello-world

Now let’s stop the process in the first window; where the configurations are updating; and execute the following command:

PS E:> docker exec -t nginx /usr/local/bin/consul-template -consul 192.168.99.100:8500 -template "/templates/ping.ctmpl:/etc/nginx/conf.d/ping.conf:service nginx reload" -pid-file /var/run/consul-template.pid -log-level warn

We are now running Consul-Template for real.   The updated ping.conf file should be correct (2 services) and within the directory /etc/nginx/conf.d/ping.conf.  We can test this by running the following command:

PS E:> docker exec -it nginx cat /etc/nginx/conf.d/ping.conf

This should then display the configuration file containing 2 services. By browsing the url http://192.168.99.100 you should see the Tutum hello world screen. By pressing F5 (refresh browser) the call will go back and hit the other service.  Showing the same. Notice that the HostName changes.  That hostname is the same as the container ID within Docker.

Now you can stop either service, and refresh the browser.  Nothing changes!  The service is still up!

This document should get you up and running.  If you have questions, please feel free to comment!

[amp-cta id=’8486′]

Ready to Ship Something?
tags: shipping code

You May Also Like

Intern-to-CEO: uShip Names Hoover-Salomon Chief Executive Read Article
Gain Shipping Peace of Mind With the uShip Protection Plan Read Article
4 Ways uShip Transporters Are Earning Cash Back Through Upside Read Article
View All Articles