I wrote a How to use HAProxy with Let's encrypt a few months ago http://www.whiteboardcoder.com/2016/05/lets-encrypt-haproxy.html . Now I have recently had a chance to tweak on it again and I figured out a better way to do it. Also I am going to tackle a little more advanced example.

I am going to do this all in Ubuntu 16.04.

At the end of this I will have this basic setup.

I am going to have

· A single HAProxy box

o It will load balance two URLS

o It will handle SSL certs for those two URLS

o It will update the SSL certs via cron job using Let's Encrypt

NGINX boxes

The first nginx box will be

· Listen on port 8080

· Has a check at /check used by the HAProxy (Check always returns UP unless the file /tmp/check exists)

Install nginx server

> sudo apt-get install nginx

Edit the /etc/nginx/nginx.conf

> sudo vi /etc/nginx/nginx.conf

And place the following in it

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
use epoll;
multi_accept on;
}

http {

include /etc/nginx/mime.types;
index    index.html index.htm;

default_type application/octet-stream;

sendfile     on;
tcp_nopush   on;
tcp_nodelay on;
server_names_hash_bucket_size 128;
keepalive_timeout    70;
types_hash_max_size 2048;

gzip on;
gzip_disable "msie6";

log_format main_fmt '$remote_addr - $remote_user [$time_local] $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';

proxy_buffering    off;
proxy_set_header   X-Real-IP $remote_addr;
proxy_set_header   X-Scheme $scheme;
proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header   Host $http_host;

access_log /var/log/nginx/access.log main_fmt;

server {
      listen       8080;      server_name localhost;

      location / {
          root   /usr/share/nginx/html;
          index index.html index.htm;
      }

      location /check {
        if (-f /tmp/check) {           return 404;        }         add_header Content-Type text/plain;         return 200 "UP";      }

      error_page   500 502 503 504 /50x.html;
      location = /50x.html {
          root   /usr/share/nginx/html;
      }
}

I also edited the /usr/share/nginx/html/index.html file per machine so I would know which machine I was hitting.

> sudo vi /usr/share/nginx/html/index.html

Place the following in it.

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body {

margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif;

}

span {

color: red;

}

h1 {

font-size: 75px;

}

</style>

</head>

<body>

<h2>This is the nginx server <span>NGINX-01</span></h2>

</center>

</body>

</html>

Restart nginx

> sudo service nginx restart

Quick test open http://192.168.0.10:8080/

HAProxy server

Before I get Let's Encrypt working I want to get HAProxy installed and load balancing between the two servers.

Install haproxy

> sudo apt-get -y install haproxy

Now edit /etc/haproxy/haproxy.cfg

> sudo vi /etc/haproxy/haproxy.cfg

Let me just load balance the first two servers on port 80

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:9090

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:admin

###########################################

# Front end for all

###########################################

frontend ALL

bind *:80

mode http

# Define hosts

acl host_foo hdr(host) -i foo.test.10x13.com

acl host_bar hdr(host) -i bar.test.10x13.com

# Direct hosts to backend

use_backend foo if host_foo

use_backend bar if host_bar

###########################################

# Back end for foo

###########################################

backend foo

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:8080 check

server server2 192.168.0.11:8080 check

###########################################

# Back end for bar

###########################################

backend bar

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.12:8080 check

server server2 192.168.0.13:8080 check

Reload haproxy

> sudo service haproxy reload

Open up http://192.168.0.9:9090/

Looks like it's all connected just fine

Test http://foo.test.10x13.com/

http://bar.test.10x13.com/

Certbot and nginx

When using the letsencrypt command line tool it will try and hit the URL you are requesting the ssl cert for at

<URL>:80/.well-known/acme-challenge/

With this in mind I am going to install nginx on the same box as the haproxy box. Have the haproxy route that path to the local nginx box. This will allow me to handle and update all the ssl certs on the haproxy box with ease.

If you want to avoid this error

Errors were encountered while processing:

nginx-core

nginx

E: Sub-process /usr/bin/dpkg returned an error code (1)

You first need to stop the nginx service before installing haproxy (I think because nginx wants to grab port 80)

> sudo service haproxy stop

Install nginx

> sudo apt-get -y install nginx

Edit the nginx.conf file to listen locally on port 8888

Edit the /etc/nginx/nginx.conf

> sudo vi /etc/nginx/nginx.conf

And place the following in it

user www-data;

worker_processes 4;

pid /var/run/nginx.pid;

events {

worker_connections 1024;

use epoll;

multi_accept on;

}

http {

include /etc/nginx/mime.types;

index index.html index.htm;

default_type application/octet-stream;

sendfile on;

tcp_nopush on;

tcp_nodelay on;

server_names_hash_bucket_size 128;

keepalive_timeout 70;

types_hash_max_size 2048;

gzip on;

gzip_disable "msie6";

log_format main_fmt '$remote_addr - $remote_user [$time_local] $status '

'"$request" $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

proxy_buffering off;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Scheme $scheme;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;

access_log /var/log/nginx/access.log main_fmt;

server {

listen 8888;

server_name localhost;

location /.well-known/acme-challenge/ {

root /usr/share/nginx/html/;

}

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root /usr/share/nginx/html;

}

location / {

return 404;

}

Restart it

> sudo service nginx restart

Restart haproxy it

> sudo service haproxy restart

Tweak haproxy.cfg

So that it will send /.well-known/acme-challenge/ to the local nginx box.

> sudo vi /etc/haproxy/haproxy.cfg

Add the highlighted portion

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:9090

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:admin

###########################################

# Front end for all

###########################################

frontend ALL

bind *:80

mode http

# Define path for lets encrypt

acl is_letsencrypt path_beg -i /.well-known/acme-challenge/

use_backend letsencrypt if is_letsencrypt

# Define hosts

acl host_foo hdr(host) -i foo.test.10x13.com

acl host_bar hdr(host) -i bar.test.10x13.com

# Direct hosts to backend

use_backend foo if host_foo

use_backend bar if host_bar

###########################################

# Back end letsencrypt

###########################################

backend letsencrypt

server letsencrypt 127.0.0.1:8888

###########################################

# Back end for foo

###########################################

backend foo

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:8080 check

server server2 192.168.0.11:8080 check

###########################################

# Back end for bar

###########################################

backend bar

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.12:8080 check

server server2 192.168.0.13:8080 check

Reload haproxy

> sudo service haproxy reload

Install Certbot

Certbot is up on github at https://github.com/certbot/certbot [1]

Good notes on how to install and use certbot can be found at https://certbot.eff.org/ [2]

It even helps walk you through.

I'll choose HAProxy on Ubuntu 16.04

And it shows me the install procedure.

Looks like Ubuntu 16.04 has a simpler install procedure.

> sudo apt-get install letsencrypt

Make sure your URL is accessible from the outside world then run this command to get a certificate automatically.

Now let me get a test cert (change the email and domain to your own) Set the --server to use the staging server so you don't hit the weekly cert limit when you are testing see https://letsencrypt.org/docs/staging-environment/ [4]

> sudo letsencrypt certonly \

--server https://acme-staging.api.letsencrypt.org/directory \

--webroot --webroot-path "/usr/share/nginx/html/" \

--keep-until-expiring \

--text \

-v \

--email me@example.com \

--agree-tos \

-d foo.test.10x13.com

This will create a file in /usr/share/nginx/html//.well-known/acme-challenge/ that letsencrypt uses to confirm you own the domain name

Now you have a cert at /etc/letsencrypt/live/foo.test.10x13.com

> sudo tree /etc/letsencrypt/live/

This cert was made using the staging server so it won't work. To get a legit cert run this command.

> sudo letsencrypt certonly \

--webroot --webroot-path "/usr/share/nginx/html/" \

--keep-until-expiring \

--text \

-v \

--email me@example.com \

--agree-tos \

-d foo.test.10x13.com

Now that you have a legit cert it needs to be combined (this is needed for haproxy)

Make a directory to stick the combined cert

> sudo mkdir -p /etc/haproxy/certs/

> sudo bash -c "cat /etc/letsencrypt/live/foo.test.10x13.com/fullchain.pem /etc/letsencrypt/live/foo.test.10x13.com/privkey.pem > /etc/haproxy/certs/foo.test.10x13.com.pem"

My script

Now onto the script I made (based on https://gist.github.com/thisismitch/7c91e9b2b63f837a0c4b#file-le-renew-haproxy [3]).

List the domains you want to get SSL certs for and this script will obtainer the cert if you don't have it or if you do have it will see if it needs to be renewed and renew it. At the end if a cert has been added or renewed it reloads haproxy.

> sudo vi /usr/local/sbin/le-renew-haproxy

Here is my script

#!/bin/bash

# Let's Encrypt HAProxy script

###################################

DOMAINS=(

"foo.test.10x13.com"

"bar.test.10x13.com"

)

EMAIL="me@example.com"

WEB_ROOT="/usr/share/nginx/html/"

#When cert is down to this many days

#It is allowed to renew

EXP_LIMIT=30;

#Only reload HAProxy if a cert was created/updated

RELOAD=false

for domain in "${DOMAINS[@]}"

CERT_FILE="/etc/letsencrypt/live/$domain/fullchain.pem"

KEY_FILE="/etc/letsencrypt/live/$domain/privkey.pem"

##################################

# If no ssl for domain create it

##################################

if [ ! -f $CERT_FILE ]; then

echo "Creating certificate for domain $domain."

letsencrypt certonly \

--webroot --webroot-path $WEB_ROOT \

--email $EMAIL \

--agree-tos \

-d $domain

###################################

# Combine certs for HAProxy and

# Reload HAProxy

###################################

mkdir -p /etc/haproxy/certs/ #location to place combine cert

RELOAD=true

COMBINED_FILE="/etc/haproxy/certs/${domain}.pem"

echo "Creating $COMBINED_FILE with latest certs..."

cat /etc/letsencrypt/live/$domain/fullchain.pem \

/etc/letsencrypt/live/$domain/privkey.pem > $COMBINED_FILE

RELOAD=true

else

##################################

# Check How long cert is valid

##################################

EXP=$(date -d "`openssl x509 -in $CERT_FILE -text -noout|grep "Not After"|cut -c 25-`" +%s)

DATE_NOW=$(date -d "now" +%s)

DAYS_EXP=$(echo $ $EXP - $DATE_NOW $ / 86400 |bc)

if [ "$DAYS_EXP" -gt "$EXP_LIMIT" ] ; then

echo "$domain, no need for renewal ($DAYS_EXP days left)."

else

#################################

# Renew Certifcate

#################################

echo "The certificate for $domain is about to expire soon."

echo "Starting Let's Encrypt renewal script..."

letsencrypt certonly \

--webroot --webroot-path $WEB_ROOT \

--keep-until-expiring \

--text \

-v \

--email $EMAIL \

--agree-tos \

-d $domain

###################################

# Combine certs for HAProxy and

# Reload HAProxy

###################################

mkdir -p /etc/haproxy/certs/ #location to place combine cert

RELOAD=true

COMBINED_FILE="/etc/haproxy/certs/${domain}.pem"

echo "Creating $COMBINED_FILE with latest certs..."

cat /etc/letsencrypt/live/$domain/fullchain.pem \

/etc/letsencrypt/live/$domain/privkey.pem > $COMBINED_FILE

echo "Renewal process finished for domain $domain"

done

if [ "$RELOAD" = true ]

then

echo " ========================= "

echo " = = "

echo " === Reloading HAProxy === "

echo " = = "

echo " ========================= "

service haproxy reload

Make it executable

> sudo chmod u+x /usr/local/sbin/le-renew-haproxy

To get this script to run I had to install bc

> sudo apt-get install bc

Now run it

> sudo /usr/local/sbin/le-renew-haproxy

The first time you run it if you do not already have certs it will create them.

If I run it again it checks the certs to see if they even need to be renewed.

Update HAProxy.cfg to handle SSL

> sudo vi /etc/haproxy/haproxy.cfg

Add the highlighted portion

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

tune.ssl.default-dh-param 4096

ssl-default-bind-options no-sslv3 no-tls-tickets

ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:9090

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:admin

###########################################

# Front end for all

###########################################

frontend ALL

bind *:80

bind *:443 ssl crt /etc/haproxy/certs/bar.test.10x13.com.pem crt /etc/haproxy/certs/foo.test.10x13.com.pem

mode http

# Define path for lets encrypt

acl is_letsencrypt path_beg -i /.well-known/acme-challenge/

use_backend letsencrypt if is_letsencrypt

# Define hosts

acl host_foo hdr(host) -i foo.test.10x13.com

acl host_bar hdr(host) -i bar.test.10x13.com

# Direct hosts to backend

use_backend foo if host_foo

use_backend bar if host_bar

# Redirect port 80 to 443

# But do not redirect letsencrypt since it checks port 80 and not 443

redirect scheme https code 301 if !{ ssl_fc } !is_letsencrypt

###########################################

# Back end letsencrypt

###########################################

backend letsencrypt

server letsencrypt 127.0.0.1:8888

###########################################

# Back end for foo

###########################################

backend foo

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:8080 check

server server2 192.168.0.11:8080 check

###########################################

# Back end for bar

###########################################

backend bar

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.12:8080 check

server server2 192.168.0.13:8080 check

Reload HAproxy

> sudo service haproxy reload

Now if you open http://foo.test.10x13.com/ it will redirect you to the secure version https://foo.test.10x13.com/. The same goes for bar.test.10x13.com.

But if you go to the lets encrypt path it will not. http://foo.test.10x13.com/.well-known/acme-challenge/ Does not go to https.

Cron Job

Let me add it this script as a cron job to run every day. (I am going to run it every day at 11:00 AM local time)

> sudo vi /etc/crontab

This line should work

> 10 11 * * * root /usr/local/sbin/le-renew-haproxy

Now to just wait 60 days and prove this works J

References

[1] Certbot's github page

https://github.com/certbot/certbot

Accessed 08/2016

[2] Certbot eff page

https://certbot.eff.org/

Accessed 08/2016

[3] LetsEncrypt renewal gist example

https://gist.github.com/thisismitch/7c91e9b2b63f837a0c4b#file-le-renew-haproxy

Accessed 08/2016

[4] LetsEncrypt Staging Server

https://letsencrypt.org/docs/staging-environment/

Accessed 08/2016

6 comments:

Andrius NorkaitisSeptember 21, 2016 at 5:51 PM
Nice article, but what about using letsencrypt with haproxy+Keepalived (or other fail-over solution)?
Certificates should be synchronised between two servers. After that we will have to renew certificates manually (perhaps also synchronising letsencrypt directory) or we need other solution how to implement auto-renew on fail-over server.
feldmarschallJanuary 7, 2017 at 4:21 AM
Hi Patrick :)
Your article helped me alot in using Lets Encrypt. I have one question though. What if instead of one domain, i had multiple domains on single nginx box? lets say there were "foo1", "foo2" on box 1 and they were in different root folders. How would you configure nginx box on HaProxy box then, and how would your le-renew scrypt looked like?
Thank you in advance
Marijan Kovacic

Let's Encrypt HAProxy Round 2

Posted on Saturday, September 10, 2016

NGINX boxes

HAProxy server

Certbot and nginx

Tweak haproxy.cfg

Install Certbot

My script

Update HAProxy.cfg to handle SSL

Cron Job

References

6 comments: