Let's Encrypt HAProxy

 I wrote a How To last week showing how to install HAproxy 1.6 on Ubuntu 14.04.  Now I want to cover setting it up to use an SSL certificate using Let's Encrypt. 

My last article can be found at http://www.whiteboardcoder.com/2016/05/install-and-setup-haproxy-16-on-ubuntu.html [1]

I have a newer article that automates this process and makes it a whole lot cleaner

http://www.whiteboardcoder.com/2016/09/lets-encrypt-haproxy-round-2.html 

At the end of that How To I this basic set up.

Here is my /etc/haproxy/haproxy.cfg file

And place the following in it

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

This allows an HAProxy stats page and it keeps sticky sessions (Once you connect to a server keep going to that server during the same session)

Let's Encrypt

Let's Encrypt https://letsencrypt.org/ [2] is out of Beta! 

What is Let's Encrypt?

From Wikipedia

Let's Encrypt is a certificate authority that launched on April 12, 2016 that provides free X.509 certificates for Transport Layer Security encryption (TLS) via an automated process designed to eliminate the current complex process of manual creation, validation, signing, installation and renewal of certificates for secure websites.

In other words free SSL certs J .   They have a very lofty noble goal to make the internet more secure by helping small sites encrypt their traffic.

Now I have not fiddled with Let's Encrypt at all yet so this will be my first go around for it.

Now it’s time to read through some notes https://letsencrypt.org/how-it-works/ [3]

Installing certbot

Looks like the first thing I need to do is install certbot https://github.com/certbot/certbot [4] A tool for getting your certs setup with Let's Encrypt.

Oh look they have a cool site for helping you deploy certbot https://certbot.eff.org/ [5]

Looks like it even helps walk you through.

I'll choose HAProxy on Ubuntu 14.04

And it shows me the install procedure.

Let me log into my HAProxy box and install the tool there.

> wget https://dl.eff.org/certbot-auto   > chmod a+x certbot-auto   >  sudo ./certbot-auto

It looks like, for some applications, certbot can automatically configure a certificate.  For HAProxy though it cannot, at least not yet J

Run this command to obtain the cert only

> ./certbot-auto certonly

Or if you are doing more than just an experiment you should probably move this to a more permanent home….

> sudo mv certbot-auto /usr/bin/certbot-auto   > sudo chown root:root /usr/bin/certbot-auto

Then you could just run.

> sudo certbot-auto certonly

Interactive mode activate!

From my first reading of the how to and looking at this letsencrypt needs a way to confirm who I am.  It looks like it will do this by posting information on my website that letsencrypt can read.

So there is my first problem to overcome…  This box I am using is not on the internet.  But I do have other boxes that are.  Maybe I can install certbot on those get a certificate and run it here?  (And use /etc/hosts or an internal DNS to get the same domain name)

Let me go run the same commands on my main nginx box that is on the internet.

Run this command to obtain the cert only  (from my nginx box on the internet)

> sudo ./certbot-auto certonly

Select 1 and click OK

Enter in an emergency email and click OK

Agree to the terms of service before you use them.

Enter in a domain name.   As a test I am going to use blah.whiteboardcoder.com which is currently routing to this server.

Click OK.

Click OK

And it crased….

Looks like it left some information in /etc/letsencrypt.

> sudo tree /etc/letsencrypt/

I tried to run this process a few more times and it still crashes

OK I found out why see https://github.com/certbot/certbot/issues/2787 [6] Basically my terminal window was too small.  So I made it full screen and it worked!

My root is /nginx/sites/nginx so I will enter that in at the bottom there.

Click OK.

Another Error I got a 404!

Let me see why..

Oops I was looking at the wrong base directory should have been /nginx/sites/whiteboardcoder.

Let me try this again

It worked!

I now have a .pem cert at

/etc/letsencrypt/live/blah.whiteboardcoder.com/fullchain.pem

Now to get it set up on my HAProxy box!

But wait… Can I do that all via just one simple command line command?

> sudo certbot-auto certonly --webroot \ --webroot-path "/nginx/sites/whiteboardcoder" \ --email me@example.com -d test.whiteboardcoder.com

That basic command worked!

And there are my files

HAProxy SSL setup

On my HAProxy box I made a folder to place the .pem keys in

> sudo mkdir -p  /etc/ssl/blah.whiteboardcoder.com

Then I copied the .pem keys to that folder from my main nginx server.

Now here comes the gotcha you can read up on at https://www.digitalocean.com/community/tutorials/how-to-secure-haproxy-with-let-s-encrypt-on-ubuntu-14-04 [7]

You need to combine fullchain.pem and privkey.pem into a single file.

Run this cat command to do it

> sudo bash -c "cat fullchain.pem privkey.pem > blah.pem"

Now update my haproxy.cfg

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem

For the moment I left the port 80 intact and added listening on port 443 and it is pointing to the cert we made.

> sudo service haproxy restart

Restart the HAProxy service.

I get this warning, which I will ignore for now.

As a test I opened https://192.168.0.9/

And I actually get the option to use it in advanced even though I am not using the domain I registered.

I thought that would not work, but it did.

But on to the main show!

I need to tweak my /etc/hosts file so that blah.whiteboardcoder.com resolves to 192.168.0.9.  Easy enough to do on  a linux system but I am on a Windows box.  The hosts file is located at C:\Windows\System32\drivers\etc\hosts.

To open it from Cygwin run the following command.

> vi /cygdrive/c/Windows/System32/drivers/etc/hosts

Add the following line

192.168.0.9     blah.whiteboardcoder.com

https://blah.whiteboardcoder.com/

I think I heard an angel choir!  That worked pretty smoothly.

I admit I do not install many SSL certs, but last time I did I remember it being a lot harder than doing it this way.

The 1024 warning

Here is my tweaked config

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon   tune.ssl.default-dh-param 2048 defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

> sudo service haproxy restart

And restart it

No issues.

Redirect all http traffic to https

Now that I have it working what is an easy way to have HAProxy redirect all http traffic to https?

Turns out it is a simple config setting see http://stackoverflow.com/questions/13227544/haproxy-redirecting-http-to-https-ssl [8]

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon   tune.ssl.default-dh-param 2048 defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   redirect scheme https code 301 if !{ ssl_fc }   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

That addition will redirect traffic.

So if I open http://blah.whiteboardcoder.com it goes to https://blah.whiteboardcoder.com/

How good is the cert?

Now that I have this cert how does it stand up to scrutiny?

I have been fiddling with this site https://www.ssllabs.com/ssltest/

Where you can stick in a URL and get your SSL cert graded.

Of course in my case I had to open a path on my system so the outside world could get to it, but once that was done I ran a check against https://blah.whiteboardcoder.com/

Hmm I got a C…  That is not really appealing.

Disable SSL3?  Let me check that one out.  I found this site http://disablessl3.com/ [9]

Update /etc/haproxy/haproxy.cfg

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon   tune.ssl.default-dh-param 2048 defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl no-sslv3 crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

Just add the no-sslv3 right there.

> sudo service haproxy restart

And restart it

To retest click Clear Cache

Hey up to a B

Now I have an RC4 cipher issue…
Another site https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ [10]

Which has some HAProxy settings suggestions.

Update /etc/haproxy/haproxy.cfg

global   log         127.0.0.1 syslog   maxconn     1000   user        haproxy   group       haproxy   daemon   tune.ssl.default-dh-param 2048   ssl-default-bind-options no-sslv3   ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS   ssl-default-server-options no-sslv3   ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

I removed no-sslv3 from the frontend section and moved it and other things to the global section.

> sudo service haproxy restart

And restart it

Hey I am up to an A.

Found this on how to get an A+ http://www.mattzuba.com/2015/07/hardening-haproxy-for-an-a-rating/ [11]

Update /etc/haproxy/haproxy.cfg

global   log         127.0.0.1 local1 notice   maxconn     1000   user        haproxy   group       haproxy   daemon   tune.ssl.default-dh-param 4096   ssl-default-bind-options no-sslv3 no-tls-tickets   ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH defaults   log  global   mode  http   option  httplog   option  dontlognull   option  http-server-close   option  forwardfor except 127.0.0.0/8   option  redispatch   option  contstats   retries  3   timeout  http-request 10s   timeout  queue 1m   timeout  connect 10s   timeout  client 1m   timeout  server 1m   timeout  check 10s ########################################### # # HAProxy Stats page # ########################################### listen stats   bind *:8080   mode  http   maxconn  10   stats  enable   stats  hide-version   stats  realm Haproxy\ Statistics   stats  uri /   stats  auth admin:mypassword ########################################### # # Front end for www # ########################################### frontend www   bind   *:80   bind   *:443 ssl crt /etc/ssl/blah.whiteboardcoder.com/blah.pem   http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload   http-response set-header X-Frame-Options DENY   http-response set-header X-Content-Type-Options nosniff   mode   http   default_backend  www ########################################### # # Back end for www # ########################################### backend www   balance         roundrobin   cookie          SERVERID insert indirect nocache maxidle 30m maxlife 8h   option          httpchk GET /check   http-check      expect string UP   default-server  inter 3s fall 3 rise 2   server          server1 192.168.0.10:80 maxconn 100 check cookie nginx-01   server          server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

I removed no-sslv3 from the frontend section and moved it and other things to the global section.

> sudo service haproxy restart

And restart it

A+!!

But be forwarned it will not work with IE6/8+XP or Java 6/7.  On that list the only thing that would concern me is Java 7.

Important things to note

Let's Encrypt Certs are only good for 90 days.   It looks like they want you to run a simple command twice a day via a cron that will renew it.

> ./path/to/certbot-auto renew --quiet --no-self-upgrade

Running it without the --quite got me this

Hmmm I may have to start encrypting traffic on some of my random sites I have…

References

[1]        Install and Setup HAProxy 1.6 on Ubuntu 14.04

http://www.whiteboardcoder.com/2016/05/install-and-setup-haproxy-16-on-ubuntu.html

Accessed 05/2016

[2]        Let's Encrypt

https://letsencrypt.org/

Accessed 05/2016

[3]        Let's Encrypt (How it works)

https://letsencrypt.org/how-it-works/

Accessed 05/2016

[4]        certbot github page

https://github.com/certbot/certbot

Accessed 05/2016

[5]        certbot eff

https://certbot.eff.org/

Accessed 05/2016

[6]        certbot issue

https://github.com/certbot/certbot/issues/2787

Accessed 05/2016

[7]        How To Secure HAProxy with Let's Encrypt on Ubuntu 14.04

https://www.digitalocean.com/community/tutorials/how-to-secure-haproxy-with-let-s-encrypt-on-ubuntu-14-04

Accessed 05/2016

[8]        HAProxy redirecting http to https (ssl)

http://stackoverflow.com/questions/13227544/haproxy-redirecting-http-to-https-ssl

Accessed 05/2016

[9]        Disable SSLv3

http://disablessl3.com/

Accessed 05/2016

[10]      Hardening Your Web Server’s SSL Ciphers

https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
Accessed 05/2016

[11]      Hardening HAProxy For An A+ Rating

http://www.mattzuba.com/2015/07/hardening-haproxy-for-an-a-rating/
Accessed 05/2016