Prometheus, nginx exporter install and set up

Posted on Sunday, August 27, 2017



I have a need to do some simple monitoring of an nginx server.  All I really want to do at the end of the day is monitor the number of 404, 500s, and 200 codes coming across my log files in nginx.

It would not be too difficult to write something that monitors logs and creates a /metrics page for Prometheus.   But I want to try one of the already made exporters out there to see how well they work.

The one I am going to try out is the Prometheus metric library for NGINX https://github.com/knyar/nginx-lua-prometheus  [1]

Install nginx and set up a test


Let me start on a fresh Ubuntu 16.04 install and just install nginx


  > sudo apt-get install nginx



Do a quick test


  > curl localhost





My server happens to live at 192.168.0.82 so doing a quick test to make sure I can get to it by opening http://192.168.0.82/




OK all is working!

Log files and simulating errors


Since my real goal to get a count of 200, 404, and 500 status I am going to simulate them as best I can.

First let me see where the default log files are



  > sudo vi /etc/nginx/nginx.conf



And there is the default log file for nginx /var/log/nginx/access.log

Now just a quick grep to see how many 200, 404, and 500 codes I have


  > cat /var/log/nginx/access.log | grep " 200 " | wc -l
 

  > cat /var/log/nginx/access.log | grep " 404 " | wc -l





  > cat /var/log/nginx/access.log | grep " 500 " | wc -l






Looks like no 500s




500 error


Let me update the nginx.conf file to output a 500 error for a specific location.


  > sudo vi /etc/nginx/nginx.conf

I updated it to.


user www-data;
worker_processes 4;
pid /var/run/nginx.pid;



events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}


http {

  include /etc/nginx/mime.types;
  index    index.html index.htm;

  default_type application/octet-stream;

  sendfile     on;
  tcp_nopush   on;
  tcp_nodelay on;
  server_names_hash_bucket_size 128;
  keepalive_timeout    70;
  types_hash_max_size 2048;


  gzip on;
  gzip_disable "msie6";

  proxy_buffering    off;
  proxy_set_header   X-Real-IP $remote_addr;
  proxy_set_header   X-Scheme $scheme;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header   Host $http_host;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  server {
    listen       80;
    server_name _;
    index index.html index.htm index.nginx-debian.html;

    root /var/www/html;

        location / {
           try_files $uri $uri/ =404;
        }

        location /500 {
           return 500;
        }
  }
}


Restart nginx


  > sudo systemctl restart nginx.service

Now you should get a 500 error if you hit /500


  > curl -I localhost/500




Now I should see 500 errors in my log


  > cat /var/log/nginx/access.log | grep " 500 " | wc -l







Simple test script


Now for a simple script


  > vi nginx_test.sh

(also put this as a gist at https://gist.github.com/patmandenver/60defed1b2b699178da6a143840edb7c )


#!/bin/bash
#
# Author: Patrick Bailey
# License: MIT
#
# Set the percentage of 200, 404, and 500 errors you want
# Set the hourly rate you want
# Set a URL for 200, 404 and 500
#
# Then run it.  it will suffle the 200, 404, and 500 urls
# (Just for testing)
#
#####################################

#Number of codes to send per type
#It will keep looping
PERC_200=60
PERC_404=30
PERC_500=40

PER_HOUR=75000

URL="http://localhost"
URL_404="http://localhost/404.html"
URL_500="http://localhost/500"

TOTAL_200=0
TOTAL_404=0
TOTAL_500=0

START=$(date +%s%N)

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT

function ctrl_c() {
   echo ""
   echo "Statistics "
   echo "200 Total: $TOTAL_200"
   echo "404 Total: $TOTAL_404"
   echo "500 Total: $TOTAL_500"
   exit 0
}

ARRAY=()
function create_array() {
  ARRAY=()
  for i in $( eval echo {1..$PERC_200} )
  do
    ARRAY+=($URL)
  done
  for i in $( eval echo {1..$PERC_404} )
  do
    ARRAY+=($URL_404)
  done
  for i in $( eval echo {1..$PERC_500} )
  do
    ARRAY+=($URL_500)
  done

  ARRAY=($(shuf -e "${ARRAY[@]}"))
}

create_array
NUM=0
TIME_CHECK=0
OLD_PAUSE_TIMER=0
TIME=$START
NUM_MSGS=10
#Number of nanoseconds it should take between runs
NUM_NSEC=$((3600 * $NUM_MSGS * 1000000000/$PER_HOUR))
#NUM_SEC=$(($NUM_NSEC/1000000000))
#NUM_MSEC=$(($NUM_SEC))$((($NUM_NSEC - $NUM_SEC*1000000000)/1000000 ))


while :
do
  LOCAL_URL=${ARRAY[$NUM]}
  curl -s -o /dev/null -w '%{http_code}' $LOCAL_URL
  echo""

  if [ "$LOCAL_URL" == "$URL" ]
    then
    ((TOTAL_200+=1))
  fi
  if [ "$LOCAL_URL" == "$URL_404" ]
    then
    ((TOTAL_404+=1))
  fi
  if [ "$LOCAL_URL" == "$URL_500" ]
    then
    ((TOTAL_500+=1))
  fi


  ((NUM+=1))
  #Get a new random array
  if [ $NUM == ${#ARRAY[@]} ]
    then
    create_array
    NUM=0
  fi

  ((TIME_CHECK+=1))
  if [ $TIME_CHECK == 1000 ]
    then
    TIME_CHECK=0
  fi

  if ! (($TIME_CHECK % $NUM_MSGS)); then
    TIME_PRIOR=$TIME
    TIME=$(date +%s%N)

    #Get total rate
    TOTAL=$(($TOTAL_200 + $TOTAL_404 + $TOTAL_500))
    TOTAL_TIME_NS=$(($TIME - $START))
    TOTAL_RATE=$((3600*1000000000*$TOTAL/$TOTAL_TIME_NS))
    echo "TOTAL RATE: $TOTAL_RATE"
    CURRENT_RATE=$((3600*1000000000*$NUM_MSGS/($TIME - $TIME_PRIOR)))
    echo "CURRENT RATE: $CURRENT_RATE"

    #Calculate pause time
    PAUSE_NSEC=$(($NUM_NSEC - ($TIME - $TIME_PRIOR)))
    PAUSE_NSEC=$(($OLD_PAUSE_TIMER + $PAUSE_NSEC))
    if (( $PAUSE_NSEC > 0 ))
    then
      PAUSE_SEC=$(($PAUSE_NSEC/1000000000))
      PAUSE_TIMER=$(($PAUSE_SEC)).$((($PAUSE_NSEC - $PAUSE_SEC*1000000000)/1000000 ))
      OLD_PAUSE_TIMER=$PAUSE_NSEC
      sleep $PAUSE_TIMER
    fi
  fi
done


Chmod it


  > chmod u+x nginx_test.sh

Run it


  > ./nginx_test.sh

It will run continuously but when you ctrl+c to get out of it you will get some stats.




Now that I have something to test with I can get about to installing the nginx Prometheus exporter.




Nginx with lua support


Looks like this tool needs nginx with lua support installed.    To easily install this on Ubuntu 16.04 all you need to do is install nginx-extras



  > sudo apt-get install nginx-extras

Now let me create a location to put this git repo


  > sudo mkdir /opt/nginx-lua-prometheus

And clone it over



  > sudo git clone https://github.com/knyar/nginx-lua-prometheus.git \
    /opt/nginx-lua-prometheus/



Now let's add this feature to nginx.conf


  > sudo vi /etc/nginx/nginx.conf

Inside the http block add the following



lua_shared_dict prometheus_metrics 10M;
lua_package_path "/opt/nginx-lua-prometheus/?.lua";
init_by_lua '
  prometheus = require("prometheus").init("prometheus_metrics")
  metric_requests = prometheus:counter(
    "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
  metric_latency = prometheus:histogram(
    "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  metric_connections = prometheus:gauge(
    "nginx_http_connections", "Number of HTTP connections", {"state"})
';
log_by_lua '
  local host = ngx.var.host:gsub("^www.", "")
  metric_requests:inc(1, {host, ngx.var.status})
  metric_latency:observe(ngx.now() - ngx.req.start_time(), {host})
';

The highlighted part is the location of the local git repo I just downloaded.
Now add a new server section to post the metrics


  server {
    listen 9113;
    allow 192.168.0.0/16;
    allow 127.0.0.1/32;
    deny all;
    location /metrics {
      content_by_lua '
        metric_connections:set(ngx.var.connections_reading, {"reading"})
        metric_connections:set(ngx.var.connections_waiting, {"waiting"})
        metric_connections:set(ngx.var.connections_writing, {"writing"})
        prometheus:collect()
      ';
    }
  }

I used port 9113 see https://github.com/prometheus/prometheus/wiki/Default-port-allocations [2]

The allow section is nice but I usually do this on the server level or at the aws/openstack level.
Now restart nginx


  > sudo systemctl restart nginx

Now see if you can see any metrics

Open localhost:9113/metrics


  > curl localhost:9113/metrics






Or in my case
http://192.168.0.82:9113/metrics





Now let me run my test to see if I can get 200, 404, and 500 status' on this page.


  > ./nginx_test.sh


Now let me see what I get.


  > curl -s localhost:9113/metrics | grep status





Looks like its working J




References


[1]        Prometheus Metric Library Github
https://github.com/knyar/nginx-lua-prometheus
Accessed 8/2017
[2]        Prometheus Exporters
https://github.com/prometheus/prometheus/wiki/Default-port-allocations
Accessed 8/2017


2 comments:

  1. SplitMay 15, 2018 at 8:36 AM

    This comment has been removed by the author.

    ReplyDelete
  2. Dieta BaraoOctober 3, 2018 at 7:20 AM

    Hi, thank you so much for sharing your tutorial!
    I was left with a question regarding the section below.
    Where should I insert it? in prometheus.yml of Prometheus-Server? or nginx_exporter.yml?

    Thank you!
    Now add a new server section to post the metrics


    server {
    listen 9113;
    allow 192.168.0.0/16;
    allow 127.0.0.1/32;

    ReplyDelete