Sensu Setting up a Client

Posted on Friday, October 24, 2014





This guide will go over setting up Sensu client on a separate server from your Sensu Master Server.  The server I will be setting up Sensu client on is a freshly installed Ubuntu 14.04 server.  This guide assumes you already have a Sensu Master Server set up, if you don't you can check out my write up on setting up a Sensu Master server at
http://www.whiteboardcoder.com/2014/10/sensu-getting-started.html

Give my Sensu Master it's own subscription


When I first set up my Master Sensu server I also set up the same box to run the Sensu Client as well.  In my /etc/sensu/conf.d/client.json I put the following text.


{
  "client": {
    "name": "server",
    "address": "192.168.0.150",
    "subscriptions": [ "ALL" ]
  }
}

I am going to change its subscription form ALL to check-from-sensu-master.  I think this is more clear this way, also while I am doing this I am going to set name to sensu-master.
Edit the client.json file


    > sudo vi /etc/sensu/conf.d/client.json


{
  "client": {
    "name": "sensu-master",
    "address": "192.168.0.150",
    "subscriptions": [ "check-from-sensu-master" ],
    "safe_mode": true
  }
}

On another note I am adding safe_mode = true.  This will require the check to be on the Master and Client.  More details on this can be found at http://sensuapp.org/docs/0.11/faq  [1]
After editing this I need to edit /etc/sensu/conf.d/check_file.json to change the subscribers.

Currently I have


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
                "ALL"
            ]
        }
    }
}


Open the file to edit


    > sudo vi /etc/sensu/conf.d/check_file.json

I updated the subscribers to


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
                "check-from-sensu-master"
            ]
        }
    }
}


Restart the Sensu Master and Client on the Master Sensu box.


    > sudo service sensu-server restart && sudo service sensu-api restart && sudo service sensu-client restart



Open up the Uchiwa web UI http://192.168.0.150:3000/  (in my case its on a local server at 192.168.0.150)




Looks like its working fine.
And if I try to test the check I made in the last article (It just checks to see if a file exists, so I will remove the file and see if I get an alert)


    >  rm ~/test.txt



Looks like its all working fine!


Touch the file to remove the alert


    >  touch ~/test.txt







Setting up a second Sensu Client on another Server


I have my master Sensu server set up and now I want to set up a Sensu Client on another server.  In my case it will be a server located at 192.168.0.151.

Before I get too far, sense I have a fresh install of Ubuntu 14.04, I need to update and upgrade apt-get.


    > sudo apt-get update
    > sudo apt-get upgrade


Install Sensu


Edit to /etc/apt/sources.list


    > sudo vi /etc/apt/sources.list


Append this to the end


deb http://repos.sensuapp.org/apt sensu main




Add an ssl key


    > wget -q http://repos.sensuapp.org/apt/pubkey.gpg
    > sudo apt-key add pubkey.gpg
    > sudo apt-get update

Install Sensu


    > sudo apt-get install sensu



Create the client.json file


Create the client.json file


    > sudo vi /etc/sensu/conf.d/client.json

And place the following in it


{
  "client": {
    "name": "client-1",
    "address": "192.168.0.151",
    "subscriptions": [ "client-1" ],
    "safe_mode":true
  }
}






Create the rabbitmq.json file


Create the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json

Put the following in it, host contains the Master Sensu hostname or IP address


{
  "rabbitmq": {
    "host": "192.168.0.150",
    "port": 5672,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypassword"
  }
}



   > sudo service sensu-client start




Then look at the log real quick



   > sudo tail -f /var/log/sensu/sensu-client.log


I have this error!

"[amqp] Detected TCP connection failure"




I think this is due to the set up of rabbitMQ on the master server.  I think it's not aware of the client.
Turns out I need to edit the /etc/rabbitmq/rabbitmq.config on the Sensu Master Server

Currently it is set to


[
    {rabbit, [
       {tcp_listeners,[{"127.0.0.1",5672}]}
     ]}
].

Which basically says only listen to localhost on port 5672


Edit the file and fix it.


    > sudo vi /etc/rabbitmq/rabbitmq.config

Here is what I set mine to  I had to remove the specific listener.


[
    {rabbit, [
     ]} 
].




Then reboot rabbitMQ on the master Sensu server


    > sudo service rabbitmq-server restart





Reload Uchiwa and click on the clients button on the right.   And you can see that my client-1 sever was added!  Now I have two Sensu clients, client-1 and sensu-master.
Enable the services to start automatically

Run the following command


  > sudo update-rc.d sensu-client defaults


Reboot to make sure it all comes up like it should


    > sudo reboot now



Add a check on the new client


I am going to recreate check-file.rb that exists on the Sensu Master onto the Sensuc Client.

Install Ruby



    > sudo apt-get install ruby ruby-dev build-essential


Install the sensu-plugin gem


    > sudo gem install sensu-plugin




Next I need to install the mixlib-cli gem


    > sudo gem install mixlib-cli




Create the check-file.rb file


    > sudo vi /etc/sensu/plugins/check-file.rb

And place the following in it


#!/usr/bin/env ruby
require 'sensu-plugin/check/cli'

class CheckFile < Sensu::Plugin::Check::CLI

  option :file,
         :description => "Path to file",
         :short => '-f FILE',
         :long => '--file FILE',
         :required => true

  def initilize
    super
  end

  def run
    if(File.exists?(config[:file]))
      ok("The file '" + config[:file] + "' Exists!  :)")
    else
      critical("The file '" + config[:file] + "' dose not Exists! :(")
    end
  end
end


Make it executable


    > sudo chmod a+x /etc/sensu/plugins/check-file.rb


Test it (run it directly)


   >  cd /etc/sensu/plugins/
   > ./check-file.rb -f ~/test.txt



I get a Critical Error since the file ~/test.txt does not exist.  If I create it


    > touch ~/test.txt

And run it again


    > ./check-file.rb -f ~/test.txt



This time it finds the file and returns OK.

Create the check Definition


Create the check definition, this file will determine how often this check is run amongst many other things.  (I am still trying to figure this all out)


    > sudo vi /etc/sensu/conf.d/check_file.json

And place the following in it.  (change the location of the file to your own location)


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "check-from-sensu-master",
               "client-1"  ]          
        }
    }
}





On the Sensu Master server update the check_file.json to the same.


    > sudo vi /etc/sensu/conf.d/check_file.json

On the Sensu Master server it was missing client-1


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "check-from-sensu-master",
               "client-1"  ]          
        }
    }
}


Restart the sensu-server on the client


    > sudo service sensu-client restart







Restart the Sensu Master with the following command, and its client


    > sudo service sensu-server restart && sudo service sensu-api restart && sudo service sensu-client restart






I still only have one check, but now that check has two subscribers.



Remove the file to test the alert


    > rm ~/test.txt



The alert is triggered and we can see the source is client-1




Bring the file back


    > touch ~/test.txt




Finally I have a Sensu Client successfully running on another server!

Setting up another Check


I want to set up a second check, but one that only client-1 subscribes to.


From the Sensu Master server create the check_second_file.json check


    > sudo vi /etc/sensu/conf.d/check_second_file.json

And place the following in it.  (change the location of the file to your own location)


{
    "checks": {
        "check_file_2": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test-2.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "client-1"  ]          
        }
    }
}

This will check to see if the file /home/patman/test-2.txt exists and only for the client-1 subscribers.

Restart the Sensu Master with the following command, and its client


    > sudo service sensu-server restart && sudo service sensu-api restart && sudo service sensu-client restart




Now I have a second check



I soon see this alert show up.



Check is not locally defined (safe mode)

Since I am running in safe_mode the code must be on the client and the server.
On the client server create the check_second_file.json check


    > sudo vi /etc/sensu/conf.d/check_second_file.json

And place the following in it.  (change the location of the file to your own location)


{
    "checks": {
        "check_file_2": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test-2.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "client-1"  ]          
        }
    }
}


Restart the sensu-client service


    > sudo service sensu-client restart






The error went away, but a new one came, because I do not have the file.

Create the file



    > touch ~/test-2.txt




And the check passed!

Now I have

·         A Sensu Master
·         A client running on the Sensu master running one check
·         A client running on another machine running two checks, but under the same subscription.


That is it for this write up.




References
[1]        Sensu Documentation FAQ
                        http://sensuapp.org/docs/0.11/faq  
                Accessed 10/2014






This post is a part of and epic, the Sensu Epic.

Epic Goal:   My goal is to figure out how to use Sensu to monitor systems.

Category:

5 comments:

  1. luMay 18, 2016 at 2:23 PM

    Do you have an ansible role to do all the above configuration

    ReplyDelete
  2. CK-the-TJanuary 7, 2017 at 9:04 AM

    thanks, very easy to follow guide. helped me a lot.

    ReplyDelete
  3. UnknownJanuary 16, 2017 at 1:29 AM

    Hi quite nice and helpful tutorial. Now what I am wondering is that any methods to hook up multiple clients to one master server? Thanks

    ReplyDelete
    Replies
    1. whiteboardcoderJanuary 18, 2017 at 5:57 AM

      I wrote an article on this subject
      1st client http://www.whiteboardcoder.com/2014/10/sensu-setting-up-client.html
      2nd client on another server http://www.whiteboardcoder.com/2014/10/sensu-setting-up-second-client-with-ssl.html

      Also there is this page that chronicles all my sensu articles
      http://www.whiteboardcoder.com/p/sensu-epic.html

      Delete