I recently had to get up
to speed on setting up and using syslog-ng. which I wrote up on how to install
it and get it logging a simple python app.
See http://www.whiteboardcoder.com/2017/03/syslog-ng-install-and-configure.html [1] I
also did a write up on how to get it to listen on a port http://www.whiteboardcoder.com/2017/03/syslog-ng-listen-on-port.html [2]
In this post I am going
to go over overcoming a few issues in getting raw json log outputs to a file.
The first problem (extra info)
First my set up in my /etc/syslog-ng/syslog-ng.conf here is my set up to listen and create a log
for my app at /var/log/myapp.log
|
source s_myapp {
system();
};
destination d_myapp {
file("/var/log/myapp.log"); };
filter f_myapp { facility(local0) and not level(debug); };
log { source (s_myapp); filter(f_myapp);
destination(d_myapp); };
|
These are the relevant parts of my syslog-ng.conf file for
this.
Now If I use the logger tool to send a simple json message…
|
> logger -p local0.info
'{"num":2, "notes":"My message"}'
|
And as I tail the log
|
> tail -f /var/log/myapp.log
|
I do get out my json, but I also get out some extra
information that includes a timestamp, hostname, and username. (my hostname happens to be syslog)
But I just want the raw json! How do I get this done?
You can use templates https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-macros.html
[3]
Edit syslog-ng.conf
|
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
Update this piece with an inline template.
|
destination d_myapp { file("/var/log/myapp.log" template("${MSG}\n")); };
|
Now reload the
syslog-ng services
|
> sudo systemctl reload syslog-ng
|
Send another message
|
> logger -p local0.info
'{"num":2, "notes":"My message"}'
|
Tailing the log again
|
> tail -f /var/log/myapp.log
|
Wahoo that worked.
The second problem … Python
The way python sends the log I get a little issue.
With this code….
|
#!/usr/bin/python3
import
sys
import
syslog
import
logging
import
logging.handlers
#Set
up Logger
logger
= logging.getLogger("10x13")
logger.setLevel(logging.DEBUG)
handler
= logging.handlers.SysLogHandler(address='/dev/log',facility=16)
log_format
= logging.Formatter('{"time":
"%(asctime)s", "note":"%(message)s"}')
handler.setFormatter(log_format)
logger.addHandler(handler)
logger.info("Info
Log Message")
logger.debug("Debug
Log Message")
logger.error("Error
log Message")
|
If I run this bit of code…
|
> ./log_test.py
|
I am getting the first part of my json lopped off.
Turns out the way python is sending the message if we choose
to use a template it will split the raw json into two variable MSGHDR and
MSG. One simple tweak to account for
that is to edit the template.
Edit syslog-ng.conf
|
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
And create the most simple network source just network()
|
destination d_myapp {
file("/var/log/myapp.log"
template("${MSGHDR}${MSG}\n"));
};
|
Now reload the
syslog-ng services
|
> sudo systemctl reload syslog-ng
|
Now run the program
|
> ./log_test.py
|
There we go!
Hmmm… I should probably add level of log to the output.
Here is my updated code.
|
#!/usr/bin/python3
import
sys
import
syslog
import
logging
import
logging.handlers
#Set
up Logger
logger
= logging.getLogger("10x13")
logger.setLevel(logging.DEBUG)
handler
= logging.handlers.SysLogHandler(address='/dev/log',facility=16)
log_format = logging.Formatter('{"level": "%(levelname)s",
"time": "%(asctime)s",
"note":"%(message)s"}')
handler.setFormatter(log_format)
logger.addHandler(handler)
logger.info("Info
Log Message")
logger.debug("Debug
Log Message")
logger.error("Error
log Message")
|
Much better J
References
[1] syslog-ng install and
configure.
[2] syslon-ng listen on a port
[3] syslog-ng templates
No comments:
Post a Comment