I recently wrote an article where I installed Prometheus on Ubuntu 20.04 http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html [1]

Now I want to start getting some real data into it to have something to graph!

One good first step is to use Prometheus Node Exporter https://prometheus.io/docs/guides/node-exporter/ [2]
https://github.com/prometheus/node_exporter [3]

There are two good reasons to start with node exporter. One, it will provide tons on Unix information about your server… Memory usage, disk usage, kernel info etc. Two, you can use it to suck up other data you create with your own cron jobs that create prometheus formatted data and put it in the correct folder.

Download and setup

Using https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04 [4]

First see what version of ubuntu you are on

> lsb_release -a

On this particular server I have set up a second drive located at /prometheus and that is where I put prometheus data

Create new user for node exporter

> sudo useradd --no-create-home --shell /bin/false node_exporter

Head over to https://github.com/prometheus/node_exporter

Go find the latest (click on latest)

Right click on this guy and get the copy the link in my case its https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.darwin-amd64.tar.gz

> wget https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz

Untar it and install

> tar xvf node_exporter-0.15.1.linux-amd64.tar.gz
> sudo cp node_exporter-0.15.1.linux-amd64/node_exporter /usr/local/bin
> sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create a folder to use for extra metrics

> sudo mkdir -p /prometheus/metrics
> sudo chown node_exporter:node_exporter /prometheus/metrics

SystemD setup

Create the systemD file

> sudo vi /lib/systemd/system/node_exporter.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/prometheus/metrics
Restart=always
RestartSec=10s
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Enable the service so it will auto start on reboot

> sudo systemctl enable node_exporter

> sudo systemctl status node_exporter

Wahoo

Test it out

> sudo systemctl start node_exporter
> sudo systemctl status node_exporter

Now pull ports and test.

> ssh prometheus -L 9090:localhost:9090 –L 9100:localhost:9100

http://localhost:9100/metrics

Wahoo data.

Now to scrape it

The data is available now but we still need to tell prometheus to scrape the data.

So let’s tweak the prometheus settings. (Prometheus is running on the same server)

> sudo vi /prometheus/pheus/prometheus.yml

Now add another thing to scrape on this same server.

- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']

Restart Prometheus

> sudo systemctl restart prometheus

Now let’s check prometheus and see if its getting the data.

http://localhost:9090/

Open Targets

Looks like its scraping it.

Go to Graph

Search for

node_cpu{mode="system"}

(click Execute)

There is data!

Custom Data

If I look at /lib/systemd/system/node_exporter.service

> sudo vi /lib/systemd/system/node_exporter.service

I have set the variable collector.textfile.directory to look at the folder /prometheus/metrics for “extra” prometheus formatted data to ingest and have node exporter show at :9100/metrics

So let me make a simple python program and create a cron job to run it.

> sudo mkdir /prometheus/code
> sudo vi /prometheus/code/prometheus_random.py

#!/usr/bin/env python3
#
# Random stuff
#
##########################

import sys
import os
import random

base_dir="/prometheus/metrics/"

##########################
# Main
##########################
if __name__ == '__main__':

#Test if base folder exist

try:
  if not os.path.exists(base_dir):
  print("folder '" + base_dir + "' does not exist")
  sys.exit(1)
  else:
  prom_data = "# HELP my_test_data_01 just some test data\n"
  prom_data += "# TYPE my_test_data_01 gauge\n"
  prom_data += "my_test_data_01 " + str(random.randint(1,100)) + "\n"

  prom_data += "# HELP my_test_data_02 just some test data\n"
  prom_data += "# TYPE my_test_data_02 gauge\n"
  prom_data += "my_test_data_02 " + str(random.randint(1,100)) + "\n"

      prom_data += "# HELP my_test_data_03 just some test data\n"
  prom_data += "# TYPE my_test_data_03 gauge\n"
  prom_data += "my_test_data_03 " + str(random.randint(1,100)) + "\n"

      f = open(base_dir + "/my_data.prom", 'w')
  f.write(prom_data)
  f.close()

except Exception as e:
print("Exception {0}", e)
sys.exit(1)

> sudo chown node_exporter:node_exporter /prometheus/code/prometheus_random.py
> sudo chown node_exporter:node_exporter /prometheus/code
> sudo chmod u+x /prometheus/code/prometheus_random.py

Let me switch to the node_exporter user and run the python code.

> sudo su node_exporter -s /bin/bash
> /prometheus/code/prometheus_random.py

Now check the output

> cat /prometheus/metrics/my_data.prom