How to install HAProxy 1.6 on Ubuntu 14.04. (And start playing with it)

First what is HAProxy and why would you want it?

From Wikipedia

HAProxy is free, open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers.[2] It is written in C[3] and has a reputation for being fast and efficient (in terms of processor and memory usage).

https://en.wikipedia.org/wiki/HAProxy [1]

So in short a free, open source, highly reliable load balancer.

My Test set up goal

My goal for this test set up is to have an HAProxy box sitting in front of two nginx boxes listening on port 80.

Something like this. I want to HAProxy to balance the load (traffic) between the two nginx boxes sitting behind it.

With that in mind let me first roll two nginx boxes that listen on port 80.

Nginx box 1 setup

> sudo apt-get -y install nginx

In a default install nginx listens on port 80 and the root directory is /usr/share/nginx/html

Edit the base html file so we know which server we are hitting.

> sudo vi /usr/share/nginx/html/index.html

And place the following in it

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body {

margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif;

}

span {

color: red;

}

h1 {

font-size: 75px;

}

</style>

</head>

<body>

<h1>This is the nginx server <span>01</span></h1>

</center>

</body>

</html>

Open http://192.168.0.10/

Nginx box 2 setup

> sudo apt-get -y install nginx

In a default install nginx listens on port 80 and the root directory is /usr/share/nginx/html

Edit the base html file so we know which server we are hitting.

> sudo vi /usr/share/nginx/html/index.html

And place the following in it

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body {

margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif;

}

span {

color: red;

}

h1 {

font-size: 75px;

}

</style>

</head>

<body>

<h1>This is the nginx server <span>02</span></h1>

</center>

</body>

</html>

Open http://192.168.0.11

HAProxy install and set up

Now that I have two nginx boxes set up to load balance between let me install and set up HAProxy on my third box.

Doing a quick test to see which version apt-get would install…

> apt-cache policy haproxy

Looks like it will install 1.4.24, but I want to install the latest 1.6.5

Looking around I found this site showing how to install HAProxy 1.5 https://www.vultr.com/docs/installing-and-configuring-haproxy-on-ubuntu-14-04 [2] I am going to tweak what they did to install 1.6

Let me add a repository so I can install the newer version.

> sudo add-apt-repository ppa:vbernat/haproxy-1.6

> sudo apt-get update

> sudo apt-get dist-upgrade

Now Install HAProxy

> sudo apt-get install haproxy

Now tweak the configurations.

Configuration Getting it to load balance…

Open up the HAProxy conf file at /etc/haproxy/haproxy.cfg

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

listen http_proxy

bind *:80

balance roundrobin

server server1 192.168.0.10:80 maxconn 100

server server2 192.168.0.11:80 maxconn 100

Now restart the haproxy service

> sudo service haproxy restart

Test

Open http://192.168.0.9/

Refresh it a few times and watch it go back and forth.

That was a very simple basic example. Let me see if I can do a few more advanced things.

Add HAProxy page

Edit the conf file again adding an Haproxy stats page

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

listen http_proxy

bind *:80

balance roundrobin

server server1 192.168.0.10:80 maxconn 100

server server2 192.168.0.11:80 maxconn 100

Now restart the haproxy service

> sudo service haproxy restart

Now open http://192.168.0.9:8080/

(login with admin:mypassword)

Now I can see some detailed information on my how my load balancer is doing.

Health Checking

I found this page on HAproxy health checking https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html#health-checking [3]

HAProxy is smart enough to stop routing to unhealthy boxes. For example if I kill nginx on my first nginx box.

> sudo service nginx stop

Now if I check http://192.168.0.9/

It only shows 02 box

If I look at the stats page

I get no information about it being down.

I can configure a health check and force it to check against a URL to determine if the machine should be part of the load balance.

This is a very useful feature in a live system. (You can take a machine out of rotation and work on it or replace it)

Edit the conf file again adding an Haproxy stats page

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

listen http_proxy

bind *:80

balance roundrobin

option httpchk GET /check

server server1 192.168.0.10:80 maxconn 100 check rise 2 fall 2

server server2 192.168.0.11:80 maxconn 100 check rise 2 fall 2

Now restart the haproxy service

> sudo service haproxy restart

Now open http://192.168.0.9:8080/

You should now see that they are both down and trying to open up http://192.168.0.9/

Give a 503, since all load balanced servers are offline

So what is going on in the code?

option httpchk GET /check

This part sets the check and tells it to use GET and to use the URL /check.

server server1 192.168.0.10:80 maxconn 100 check rise 2 fall 2

server server2 192.168.0.11:80 maxconn 100 check rise 2 fall 2

This part says use the check and fall 2 is the number of consecutive checks that have to fail before the server is considered down. Rise 2 is the number of consecutive checks the server has to pass before it is considered up.

So how often does it check? The default is 2 seconds. See https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html#check-interval [4]

How do I bring the servers back? I simply need a reply when I go to this URL http://192.168.0.10/check

So since these are simple nginx servers I can make a file there.

Let me just make a file on the first nginx box and bring that one back up.

> sudo touch /usr/share/nginx/html/check

And the one server is back up!

Let me run the same command on the second machine.

Health Check 2

What if you want a smarter health check that can check against a returned string?

Let me check for the string "UP"

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

listen http_proxy

bind *:80

balance roundrobin

option httpchk GET /check

http-check expect string UP

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:80 maxconn 100 check

server server2 192.168.0.11:80 maxconn 100 check

Now restart the haproxy service

> sudo service haproxy restart

http-check expect string UP

default-server inter 3s fall 3 rise 2

Now it should check for the string UP and also I pulled out the fall and rise per server into a default-server setting.

Now it's failing again. Even though we do have a file at /check it is empty.

Let me go update those files

> sudo su root -c 'echo "UP" > /usr/share/nginx/html/check'

And we are back up again.

Note:

The expect string seems to search the entire text for a match so

it is UP

wow look UP there

all pass

There is another option you could use rstring (for regular expressions)

listen http_proxy

bind *:80

balance roundrobin

option httpchk GET /check

http-check expect rstring ^UP$

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:80 maxconn 100 check

server server2 192.168.0.11:80 maxconn 100 check

That will only match UP.

Frontend Backend

Looking around at a few configurations I see listen, frontend, and backend. What is the difference between them all.

I found this document http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4 [5]

From the document

Frontend:

A "frontend" section describes a set of listening sockets accepting client connections.

Backend:

A "backend" section describes a set of servers to which the proxy will connect to forward incoming connections.

Listen:

A "listen" section defines a complete proxy with its frontend and backend parts combined in one section. It is generally useful for TCP-only traffic.

OK so for my purposes using a frontend section and a backend section just separates out the responsibilities and I thin makes it a littler clearer as to what is going on.

I am going to edit my file to use a frontend and backend

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

###########################################

# Front end for www

###########################################

frontend www

bind *:80

mode http

default_backend www

###########################################

# Back end for www

###########################################

backend www

balance roundrobin

option httpchk GET /check

http-check expect string UP

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:80 maxconn 100 check

server server2 192.168.0.11:80 maxconn 100 check

Now restart the haproxy service

> sudo service haproxy restart

Opening http://192.168.0.9:8080/

I can see I know have a two section www for frontend and www for backend www.

As a quick test I removed the check file on one nginx server.

> sudo rm /usr/share/nginx/html/check

Worked!

Sticky Sessions

What if I am in a situation that I need to keep a session attached to the same server? In other words when someone hits my URL for the first time I do not care which server they hit. But when they hit I keep track of which box they hit and keep them talking to that box. So it if hits nginx box 2 it will keep hitting that same box for all requests.

Of course I would probably want that stickiness to disappear the next day (I don’t need them to hit the same server forever just during their 'session')

How do I do this in HAProxy? I found a nice article here http://blog.haproxy.com/2012/03/29/load-balancing-affinity-persistence-sticky-sessions-what-you-need-to-know/ [6]

Here is a link to some documentation on cookies in haproxy http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4.2-cookie [7]

Here is my first go at it.

> sudo vi /etc/haproxy/haproxy.cfg

Update it to the following

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

###########################################

# Front end for www

###########################################

frontend www

bind *:80

mode http

default_backend www

###########################################

# Back end for www

###########################################

backend www

balance roundrobin

cookie SERVERID insert indirect nocache

option httpchk GET /check

http-check expect string UP

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:80 maxconn 100 check cookie nginx-01

server server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

Looking over it

cookie SERVERID insert indirect nocache

Just says insert a cookie named "SERVERID"

server server1 192.168.0.10:80 maxconn 100 check cookie nginx-01

server server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

This says if route to this server and you are putting in a cookie put the following in the text of the cookie. So for server1 it would put nginx-01

Now restart the haproxy service

> sudo service haproxy restart

If you don't have the chrome plugin editthiscookie go and download it so you can easily look at your cookies https://chrome.google.com/webstore/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg?hl=en

When you have it you should see this little cookie icon.

Open http://192.168.0.9/

Reload it several times to see that you stick to the same server.

Now click on the cookie tool

There is the cookie name SERVERID and the value nginx-02.

Now as is I think, as long as someone does not wipe their cookies, they will be routed to the same server today, tomorrow and forever…. I do not like that idea so I tweaked mine a bit.

Here is my tweaked version

global

log 127.0.0.1 syslog

maxconn 1000

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

option contstats

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

###########################################

# HAProxy Stats page

###########################################

listen stats

bind *:8080

mode http

maxconn 10

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /

stats auth admin:mypassword

###########################################

# Front end for www

###########################################

frontend www

bind *:80

mode http

default_backend www

###########################################

# Back end for www

###########################################

backend www

balance roundrobin

cookie SERVERID insert indirect nocache maxidle 30m maxlife 8h

option httpchk GET /check

http-check expect string UP

default-server inter 3s fall 3 rise 2

server server1 192.168.0.10:80 maxconn 100 check cookie nginx-01

server server2 192.168.0.11:80 maxconn 100 check cookie nginx-02

That should at least reset it every day

Looks like it stores the data here rather than expire the cookie.

It would take a while to test this out so I am going to change the times to 1m and 2m and run some test

cookie SERVERID insert indirect nocache maxidle 1m maxlife 2m

Now restart the haproxy service

> sudo service haproxy restart

I ran several tests where I waited over 1 min to reload the page and I was able to get the other server. If I hit reload before the 1 min had elapsed I stayed on the server I was on.

If I kept hitting refresh, eventually after two minutes, I switched to the other server.

Last test

If I have a sticky session and the server goes down to I get routed to the other server?

Let me bump my time outs back to 30m and 8 hours

Reload the page

Now go to this box and remove the check file

> sudo rm /usr/share/nginx/html/check

Confirm it is down.

Confirm the cookie points to nginx-01

Now reload the page

Yep made the jump no hesitiation.

And now my cookie as nginx-02

Wrap up

That gets me pretty far with what I want to do. I am sure there are a lot more neat little tweaks I will learn over the years but for now that is more than enough to get me going.

Next I am going to do another write up trying to figure out how to add an SSL certificate.