I am still poking around
Prometheus. So far I am liking it a lot.
In this article I am
going to go over how to get it connected to JMX in Zookeeper to start getting
metrics out of those systems for monitoring.
I did another article on how to get it set up with Kafka ( see http://www.whiteboardcoder.com/2017/04/prometheus-and-jmx.html [1] )
I some of information
for this article at https://blog.rntech.co.uk/2016/10/20/monitoring-apache-kafka-with-prometheus/ [2] and https://www.robustperception.io/monitoring-kafka-with-prometheus/ [3]
Be forewarned, this is
going to be a long article as I poke at this tool to get it working the way I
want it to.
Prometheus JMX Exporter
JXM Exporter located at https://github.com/prometheus/jmx_exporter/ [3] on GitHub.
It is a lightweight http server that exposes JMX data as Prometheus
compatible metrics that it can then scrape.
VisualVM and JMX
First thing I want to do
is to turn jmx on for the zookeeper.
Once I have this set up then I can download and use VisualVM to connect
to zookeeper and see some stats.
I have a very basic set
up of kafka/zookeeper poking around I found an environment file that is used in
the init script /etc/zookeeper/conf/environment
From
/etc/init.d/zookeeper
I am going to edit that
file so that it will turn on JMX
|
> sudo vi /etc/zookeeper/conf/environment
|
And update the JAVA_OPTS
section with this
|
#Should retrieve
local IP address
IP_ADDR=`ip
route get 8.8.8.8 | awk '{print $NF; exit}'`
JAVA_OPTS="$JAVA_OPTS
-Djava.rmi.server.hostname=$IP_ADDR"
JAVA_OPTS="$JAVA_OPTS
-Dcom.sun.management.jmxremote.port=9696"
JAVA_OPTS="$JAVA_OPTS
-Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="$JAVA_OPTS
-Dcom.sun.management.jmxremote.ssl=false"
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Let's see if that worked.
|
> sudo ps -Af | grep 9696
|
That appears to be working.
Now onto using VisualVM to connect into it.
Download Visual VM
And Download
Unzip it and start it
up.
Accept the license
Add a JMX Connection.
Enter in the IP address
and port (In my case the kafka/zookeeper
server lives at 192.168.0.140) . Then
click OK
Now you should have
this.
Double Click on it.
Click on the monitor tab
and you can now see stuff scrolling by.
Go to Tools ->
Plugins
Select the Available
Plugins tab and checkbox the VisualVM-MBeans and click Install.
Next
Accept the license and
click Install
Finish
Close the connection
Now you should have an
MBeans tab
There are your mbeans J
This is a standalone zookeeper so my specific zookeeper
mbeans are very limited.
Here is a zookeeper page that lists their mbeans in JMX
The bottom section is the standalone mbeans.
There are some values.
Install Prometheus JMX exporter and get it set up
Head over here https://github.com/prometheus/jmx_exporter/#running
[5]
Download the jar file
I am going to make a folder to do all this work in.
|
> cd
> mkdir jmx_exporter_zookeeper
> cd jmx_exporter_zookeeper
> wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.9/jmx_prometheus_javaagent-0.9.jar
|
Create a configuration
file in yaml format
|
> vi zookeeper_kafkfa.yml
|
This is going to be a very very basic yaml file to start
with. It will read in all the mbeans
data and use default setttings for them.
|
---
lowercaseOutputName: true
|
Adding jmx exporter as javaagent to zookeeper
Edi the same zookeeper
environment file again.
|
> sudo vi /etc/zookeeper/conf/environment
|
And add this to the
JAVA_OPTS section
|
##JMX Exporter
Agent
JMX_DIR="/home/patman/jmx_exporter_zookeeper"
JAVA_OPTS="$JAVA_OPTS
-javaagent:$JMX_DIR/jmx_prometheus_javaagent-0.9.jar=1234:$JMX_DIR/zookeeper_kafkfa.yml"
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Now try and curl the
/metrics
|
> curl localhost:1234/metrics
|
Wahoo it worked!
But we have at least two problems.
1.
We have all the metrics, and we may not want
them all
2.
All zookeepermetrics are being set as gauges,
some need to be labeled as counters.
So how do we fix that?
We fix it by tweaking the config yaml file we made.
Filtering the data
First thing run a few curls to see how much data points we
are getting.
This will list all the
individual metrics and remove the comments/settings that start with a # symbol
|
> curl -s localhost:1234/metrics | grep -v "^#"
|
Get a total count
|
> curl -s localhost:1234/metrics | grep -v "^#" |
wc -l
|
I have a total of 287 records.
These records consist of generic java jvm info like memory
size etc and some is specific to zookeeper.
Get a total count of just zookeeper data
|
> curl -s localhost:1234/metrics | grep -v "^#" |
grep zookeeper | wc -l
|
I can add a whitelist, that will filter on class names, to
the yaml file
Here is my first go at it.
|
> vi zookeeper_kafkfa.yml
|
Update it to the following
|
---
lowercaseOutputName: true
whitelistObjectNames:
["java.lang.OperatingSystem:*"]
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
See how many stats I have now
|
> curl -s localhost:1234/metrics | grep -v "^#" |
wc -l
|
Down to 48 and if you look at them all they are just
java_lang stats.
Second go at it (just getting
|
> vi zookeeper_kafkfa.yml
|
Update it to the
following (only get the zookeeper
metrics)
|
---
lowercaseOutputName: true
whitelistObjectNames: ["org.apache.ZooKeeperService:*"]
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
See how many stats I have now
|
> curl -s localhost:1234/metrics | grep -v "^#" |
wc -l
|
I got 69 back … and if you look at it.
I have zookeeper metrics and I also have the java jvm
data. Turns out no matter what whitelist
you set you will always get the jvm data.
Which is fine by me.
JMX trickiness
Now for the fun part.
I need to further filter out the zookeeper data and also set some as
counters.
I am going to start with this one metric StandaloneServer_port2181:PacketsReceived.
This metric is a counter, not a gauge, it always increases.
I want to fix my yaml file so I only get this zookeeper
metric and it shows up as a counter.
Running this curl I can see it shows up as a gauge.
|
> curl -s localhost:1234/metrics | grep -B 2
packetsreceived
|
I also happen to see the other specific packets received per
server connected at
I think I will try and get both of them if I can, but only
those from the zookeeper data.
Fun part
To filter out this data you need to know three things from
the JMX data.
I need to pick out three
parts to be used in a whitelistObjectNames
1. Domain
2. Type
3. Name
Select Standalone
Click on Metadata and look at ObjectName
1.
Domain
In this case the domain
= org.apache.ZooKeeperService
2.
Type
This appears to have not
type
In this case type = *
3.
Name
In this case the name0 =
StandaloneServer_port2181
I can use these first
three parts and create a whitelistObjectNames.
|
> vi zookeeper_kafkfa.yml
|
Here is a simple yaml
file using that.
|
---
lowercaseOutputName: true
whitelistObjectNames:
["org.apache.ZooKeeperService:name0=StandaloneServer_port2181,*"]
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Running this curl I can see it shows up as a gauge.
|
> curl -s localhost:1234/metrics | grep -v "#" |
grep zoo | wc -l
|
This gave me 21 , so no reduction yet.
More tweaking setting rules
|
> vi zookeeper_kafkfa.yml
|
Update it to.
|
---
lowercaseOutputName: true
#whitelistObjectNames:
["org.apache.ZooKeeperService:*"]
whitelistObjectNames: ["org.apache.ZooKeeperService:name0=StandaloneServer_port2181,*"]
rules:
- pattern:
org.apache.ZooKeeperService<name0=(.+)><>(PacketsReceived)
name: zookeeper_server_$1
type: COUNTER
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Running this curl
|
> curl -s localhost:1234/metrics | grep -v "#" |
grep zoo | wc -l
|
Down to 2!
If I look at it
I can see I have two metrics that are both marked as
counters. So that part is good. The second metric is coming from here.
I do want this data but I do not like how it is being named
zookeeper_server_standaloneserver_port2181_name1_connections_name2_0:0:0:0:0:0:0:1_name3_0x15c1706051b0000
I need to fiddle with the config file again.
|
> vi zookeeper_kafkfa.yml
|
Update it to.
|
---
lowercaseOutputName: true
#whitelistObjectNames:
["org.apache.ZooKeeperService:*"]
whitelistObjectNames:
["org.apache.ZooKeeperService:name0=StandaloneServer_port2181,*"]
rules:
- pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsReceived)
name: zookeeper_server_$2_$3_$5
type: COUNTER
- pattern:
org.apache.ZooKeeperService<name0=(.+)><>(PacketsReceived)
name: zookeeper_server_$1_$2
type: COUNTER
|
I added a new rule that
will find this second one. The Patterns
are applied in order and the first one that fits will be used.
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Running this curl
|
> curl -s localhost:1234/metrics | grep -v "#" |
grep zoo
|
Exactly what I wanted J
More tweaking All Counters
Let me see which other data points I want that are COUNTERS.
Looks like I only want PacketsReceived and PacketsSent for
all my COUNTERS.
|
> vi zookeeper_kafkfa.yml
|
Update it to.
|
---
lowercaseOutputName: true
whitelistObjectNames: ["org.apache.ZooKeeperService:name0=StandaloneServer_port2181,*"]
rules:
- pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsReceived|PacketsSent)
name: zookeeper_server_$2_$3_$5
type: COUNTER
- pattern:
org.apache.ZooKeeperService<name0=(.+)><>(PacketsReceived|PacketsSent)
name: zookeeper_server_$1_$2
type: COUNTER
|
I added a new rule that
will find this second one. The Patterns
are applied in order and the first one that fits will be used.
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Running this curl
|
> curl -s localhost:1234/metrics | grep -v "#" |
grep zoo
|
Wahoo working!
More tweaking All Gauges
Let me see which other data points I want that are just
normal gauges?
AvgRequestLatency
MaxRequestLatency
MinRequestLatency
NumAliveConnections
OutstandingRequests
AvgLatency
MaxLatency
MinLatency
OutstandingRequests
|
> vi zookeeper_kafkfa.yml
|
Update it to.
|
---
lowercaseOutputName: true
whitelistObjectNames:
["org.apache.ZooKeeperService:name0=StandaloneServer_port2181,*"]
rules:
- pattern:
org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(PacketsReceived|PacketsSent)
name: zookeeper_server_$2_$3_$5
type: COUNTER
- pattern:
org.apache.ZooKeeperService<name0=(.+)><>(PacketsReceived|PacketsSent)
name: zookeeper_server_$1_$2
type: COUNTER
- pattern: org.apache.ZooKeeperService<name0=(.+).name1=(.+).name2=(.+).name3=(.+)><>(AvgLatency|MaxLatency|MinLatency|OutstandingRequests)
name:
zookeeper_server_$2_$3_$5
- pattern: org.apache.ZooKeeperService<name0=(.+)><>(AvgRequestLatency|MaxRequestLatency|MinRequestLatency|NumAliveConnections|OutstandingRequests)
name:
zookeeper_server_$1_$2
|
Now restart zookeeper
|
> sudo /etc/init.d/zookeeper restart
|
Running this curl
|
> curl -s localhost:1234/metrics | grep -v "#" |
grep zoo | sort
|
Wahoo working!
References
[1] Prometheus and JMX
[2] Monitoring Apache Kafka with
Prometheus
[3] Monitoring Kafka with
Prometheus
[4] Zookeeper Mbeans
[5] Prometheus git page
very good document ... very helpfull
ReplyDeleteReally, good documentation. thanks you!
ReplyDelete