Sensu Getting Started

This guide will get you through the very first steps in setting up Sensu Code (I want to keep it simple)

I am going to install a Sensu Master and a client on the same machine, a brand new Ubuntu 14.04 install.   For this write up I will not be using SSL keys for RabbitMQ to communicate with (This is fine for testing and may be an OK thing to do in an internal network)

What is Sensu

From this site https://www.digitalocean.com/community/tutorials/how-to-configure-sensu-monitoring-rabbitmq-and-redis-on-ubuntu-14-04  [1]

Sensu is a monitoring tool written in Ruby that uses RabbitMQ as a message broker and Redis for storing data. It is well suited for monitoring cloud environments.

Setup

I am going to start with servers that have a fresh install of Ubuntu 14.04

I have a fresh install of Ubuntu 14.04 on my test servers, I am going to start from there.

Sensu Server

I am first going to set up the Sensu Server, I will set up the client later on.

Before I get too far, sense I have a fresh install of Ubuntu 14.04, I need to update and upgrade apt-get.

> sudo apt-get update     > sudo apt-get upgrade

Installing RabbitMQ

Sensu uses RabbitMQ.  I am following the guide at http://www.rabbitmq.com/install-debian.html [2]

Edit to /etc/apt/sources.list

> sudo vi /etc/apt/sources.list

Append this to the end

deb http://www.rabbitmq.com/debian/ testing main

Add an ssl key

> wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc     > sudo apt-key add rabbitmq-signing-key-public.asc     > sudo apt-get update

Install rabitmq-server

> sudo apt-get install rabbitmq-server

The RabbitMQ server should have started, but if not you can start it with this command.

> sudo service rabbitmq-server start

Create and and edit the /etc/rabbitmq/rabbitmq.config

> sudo vi /etc/rabbitmq/rabbitmq.config

Put the following in it

[     {rabbit, [        {tcp_listeners,[{"127.0.0.1",5672}]}      ]} ].

Create a virtual host and user for Sensu.  (Change mypassword to whatever password you want)

> sudo rabbitmqctl add_vhost /sensu     > sudo rabbitmqctl add_user sensu mypassword     > sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Restart rabbitMQ

> sudo service rabbitmq-server restart

Enable the web UI interface (Optional… I don't think you need to do this, but I am going to)

> sudo rabbitmq-plugins enable rabbitmq_management

Restart rabbitMQ

> sudo service rabbitmq-server restart

Make sure the web UI is running by running this command

> netstat -teapu | grep 15672

Open the web UI in my case 192.168.0.150:15672

Login

Username:  guest

Password: guest

I hate leaving a default user/password around so I am going to fix this.

Add admin to RabbitMQ UI

Click Admin

Enter a username and password.  Click on the Admin button, this will add administrator to Tags.  Then click Add user.

Click on the new name

Click Set permission

Now you are set up.

Log out and back in with your new user

To remove the quest user do the following

Click Admin

Click guest

Click Delete this user.

Click Delete

Install Redis

> sudo apt-get install redis-server

Install Sensu

Edit to /etc/apt/sources.list

> sudo vi /etc/apt/sources.list

Append this to the end

deb http://repos.sensuapp.org/apt sensu main

Add an ssl key

> wget -q http://repos.sensuapp.org/apt/pubkey.gpg     > sudo apt-key add pubkey.gpg     > sudo apt-get update

Install Sensu

> sudo apt-get install sensu

Configure the Sensu Master

Create the rabbitmq.json file.

> sudo vi /etc/sensu/conf.d/rabbitmq.json

Put the following in it

{   "rabbitmq": {     "host": "localhost",     "port": 5672,     "vhost": "/sensu",     "user": "sensu",     "password": "mypassword"   } }

Change and edit the redis.json file

> sudo vi /etc/sensu/conf.d/redis.json

And place the following in it.

{   "redis": {     "host": "localhost",     "port": 6379   } }

Create the api.json file

> sudo vi /etc/sensu/conf.d/api.json

And place the following in it

{   "api": {     "host": "localhost",     "port": 4567   } }

Install a Sensu Dashboard

Sensu recommends using the Uchiwa dashboard http://sensuapp.org/docs/latest/install_a_dashboard  [3]

Install Uchiwa

> sudo apt-get install uchiwa

edit uchiwa.json

> sudo vi /etc/sensu/uchiwa.json

The default file is fine for my purposes.  Here is what is contained in it.

{   "sensu": [     {       "name": "Sensu",       "host": "127.0.0.1",       "ssl": false,       "port": 4567,       "user": "",       "pass": "",       "path": "",       "timeout": 5000     }   ],   "uchiwa": {     "user": "",     "pass": "",     "port": 3000,     "stats": 10,     "refresh": 10000   } }

Restart the Sensu Master with the following command.

> sudo service sensu-server restart && sudo service sensu-api restart

Start up uchiwa

> sudo /etc/init.d/uchiwa restart

Make sure its running.

> netstat -teapu | grep 3000

Open up the URL  in my case 192.168.0.150:3000/

Its working, although there is not much there yet.

Enable the services to start automatically

Run the following commands

> sudo update-rc.d sensu-server defaults     > sudo update-rc.d sensu-api defaults     > sudo update-rc.d uchiwa defaults

Reboot the server

> sudo reboot now

Open the URL again to see if it's running

Setting up a Client

I have a Sensu Master server, but no client yet to actually monitor anything.

For this write-up I am going run a Sensu client on the same box as the Sensu Master. 

If you roll over the clients section on Uchiwa you will see there are currently 0 clients

Create the client.json file

> sudo vi /etc/sensu/conf.d/client.json

And place the following in it

{   "client": {     "name": "server",     "address": "192.168.0.150",     "subscriptions": [ "ALL" ]   } }

Start the client

> sudo service sensu-client start

In a few seconds it should shows up
Enable the services to start automatically

Run the following command

> sudo update-rc.d sensu-client defaults

Reboot to make sure it all comes up like it shoould

> sudo reboot now

Open the Uchiwa dashboard and confirm it all comes back up

In my case that's at http://192.168.0.150:3000/

Success!

Finally Add a Check

If you scroll over the check in the Uchiwa Dashboard

You will see a big fat 0 Check.  I have no checks!

Install Ruby

> sudo apt-get install ruby ruby-dev build-essential

Install the sensu-plugin gem

> sudo gem install sensu-plugin

Create a check

I do not like the example check I see at https://www.digitalocean.com/community/tutorials/how-to-configure-sensu-monitoring-rabbitmq-and-redis-on-ubuntu-14-04 [1]

Instead I am going to use the recommendation at

https://github.com/sensu/sensu-plugin [4] which says to subclass Sensu::Plugin::Check::CLI

Also I will use the mixlib-cli tool https://github.com/opscode/mixlib-cli [5] to give

Install the mixlib-cli gem

> sudo gem install mixlib-cli

Create the check-file.rb file

> sudo vi /etc/sensu/plugins/check-file.rb

And place the following in it

#!/usr/bin/env ruby require 'sensu-plugin/check/cli' class CheckFile < Sensu::Plugin::Check::CLI   option :file,          :description => "Path to file",          :short => '-f FILE',          :long => '--file FILE',          :required => true   def initilize     super   end   def run     if(File.exists?(config[:file]))       ok("The file '" + config[:file] + "' Exists!  :)")     else       critical("The file '" + config[:file] + "' dose not Exists! :(")     end   end end

Make it executable

> sudo chmod a+x /etc/sensu/plugins/check-file.rb

Test it (run it directly)

>  cd /etc/sensu/plugins/    > ./check-file.rb -f ~/test.txt

I get a Critical Error since the file ~/test.txt does not exist.  If I create it

> touch ~/test.txt

And run it again

> ./check-file.rb -f ~/test.txt

This time it finds the file and returns OK.

Create the check Definition

Create the check definition, this file will determine how often this check is run amongst many other things.  (I am still trying to figure this all out)

> sudo vi /etc/sensu/conf.d/check_file.json

And place the following in it.  (change the location of the file to your own location)

{     "checks": {         "check_file": {             "handlers": [                 "default"             ],             "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",             "interval": 60,             "occurrences": 3,             "subscribers": [                 "ALL"             ]         }     } }

Restart the sensu-server on the client

> sudo service sensu-client restart

And…. Nothing…

The Sensu Master Server needs to be restarted (and in the case of a remote client server the Sensu Master would need to have the check code as well)

Restart the Sensu Master with the following command.

> sudo service sensu-server restart && sudo service sensu-api restart

And now we have a check !

At the moment everything looks happy.

If I remove the test.file

> rm ~/test.txt

Now I have 1 critical event

Click the Events icon

Click anywhere along the line where the event is displayed

I can now see all the details of the event.

If I let it got for X minutes if I look at Uchiwa I can see that I have X Critical occurrences.  Since my check is set to run once every 60 seconds.

Recreate the test.txt file

> touch ~/test.txt

And the event clears out within 60 seconds.  (the next check marks it as OK).

I think I am going to like Sensu, I really think it is a great tool you almost need in the arsenal in a cloud computer era.

References

[1]        How To Configure Sensu Monitoring, RabbitMQ, and Redis on Ubuntu 14.04

                        https://www.digitalocean.com/community/tutorials/how-to-configure-sensu-monitoring-rabbitmq-and-redis-on-ubuntu-14-04 

                Accessed 10/2014

[2]        Installing on Debian / Ubuntu

                        http://www.rabbitmq.com/install-debian.html

                Accessed 10/2014

[3]        Install a Sensu Dashboard

                        http://sensuapp.org/docs/latest/install_a_dashboard

                Accessed 10/2014

[4]        github sensu-plugin page

                        https://github.com/sensu/sensu-plugin

                Accessed 10/2014

[5]        github mixlib-cli library

                        https://github.com/opscode/mixlib-cli

                Accessed 10/2014

This post is a part of and epic, the Sensu Epic.

Epic Goal:   My goal is to figure out how to use Sensu to monitor systems.