Syslog-ng listen on port

I recently had to get up to speed on setting up and using syslog-ng. which I wrote up on how to install it and get it logging a simple python app.  See http://www.whiteboardcoder.com/2017/03/syslog-ng-install-and-configure.html [1]

In this post I am going to go over how to get syslog-ng to listen on a port for incoming logs.

Getting it to listen on a port

First I need to do a little research on how network works in with sources https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-sources-network.html [2]

Edit syslog-ng.conf

> sudo vi /etc/syslog-ng/syslog-ng.conf

And create the most simple network source just network()

source s_network {     network( transport(tcp) port(601)); };

This source will listen on the default port 601 and on TCP. It will also listen from any IP address.

Now let me create the log section

log { source(s_network); filter(f_network); destination(d_network); };

Next create a filter section

filter f_network { facility(local0) and not level(debug); };

Next create a destination section

destination d_network { file("/var/log/app_network.log" ); };

Now reload  the syslog-ng services

> sudo systemctl reload syslog-ng

To double check that the conf file was correctly formatted.

> tail -f /var/log/syslog

And you should see something like this.

On the top there is an error.

Now do a quick check and make sure syslog-ng is listening on that port.

> sudo netstat -peant | grep syslog-ng

Yep all looks good.

Looking at the docs I see this note.

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see Section 6.11, Collecting messages using the IETF syslog protocol (syslog() driver).

So it is listening but expecting a specific protocol to come in.  What is the simplest way to test this?

A simple way to send a message is to use the logger command https://linux.die.net/man/1/logger [3]

Sending logs to the port

Tail the /var/log/syslog

> tail -f /var/log/syslog

Now Send a log message to the port

> logger -p local0.info --server 127.0.0.1 --tcp --port 601 "My Test Message"

You should see 3 messages in syslog

Mar 25 07:38:56 syslog syslog-ng[2359]: Syslog connection accepted; fd='13', client='AF_INET(127.0.0.1:39584)', local='AF_INET(0.0.0.0:601)' Mar 25 07:38:56 127.0.0.1 1 2017-03-25T07:38:56.889550-06:00 syslog patman - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="313500"] My Test Message Mar 25 07:38:56 syslog syslog-ng[2359]: Syslog connection closed; fd='13', client='AF_INET(127.0.0.1:39584)', local='AF_INET(0.0.0.0:601)'

The first messages just says that the connection was accepted.  The last that the connection is was closed.  The one in the middle is the actual message.

Mar 25 07:38:56 127.0.0.1 1 2017-03-25T07:38:56.889550-06:00 syslog patman - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="313500"] My Test Message

There is my message.

Now just tail the /var/log/app_network.app

> tail -f /var/log/app_network.log

For me that log is a little too verbose.

That can be changed by a few tweaks in the sylog-ng.con file

Simplify the message

You can use templates to set the output of the message see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-macros.html [4]

I am going to do a simple inline template

Here are a few variables

Variable	Definition
${DATE}	Date-related macros, FULLDATE, C_FULLDATE, R_FULLDATE, S_FULLDATE
${HOST}
${HOUR12}	AMPM
${HOUR}	Date-related macros
${ISODATE}
${LEVEL}	LEVEL_NUM, PRIORITY or LEVEL
${MSGHDR}	Templates and macros, MSG or MESSAGE
${MSGONLY}	MSG or MESSAGE
${MSG}	Message representation in syslog-ng OSE, MSG or MESSAGE, substr
${PID}	Comparing macro values in filters

For a fuller list go to https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/syslog-ng-parameter-index.html [5]

Here are some variables ${ISODATE} ${HOST} ${MSGHDR}${MSG}\n. (The ${MSGHDR}${MSG}

Edit syslog-ng.conf

> sudo vi /etc/syslog-ng/syslog-ng.conf

I am going to use MSG and LEVEL

Update this line

destination d_network {    file("/var/log/app_network.log" template("${LEVEL}::${MSG}\n")); };

Now reload  the syslog-ng services

> sudo systemctl reload syslog-ng

Now Send a log message to the port

> logger -p local0.info --server 127.0.0.1 --tcp --port 601 "My Test Message"

Not exactly what I expected I do see the info:: (level) there.   But the message is rather robust.  Is logger adding this additional information to the message?

Poking around I finally figured it out.   Here is how to get logger to not send that extra information.

> logger -p local0.info --server 127.0.0.1 --tcp --port 601\       --rfc3164 "My Test Message"

Wahoo that worked !

Now since this is listening on a port other servers can send messages to this port.

I can run this command from cygwin on another machine and it works (The logger is an older version and by default uses the rfc3164 standard)

> logger -p local0.info --server 192.168.0.52 --tcp --port 601 \      "From another server"

Works!

Python code sending to a port

Now that I got that working I wanted to create a simple python logger example to send messages to that port.

Looks like for python the only option to send logs on a port is via UDP, so I need to tweak the syslog-ng.conf file to also listen to udp

> sudo vi /etc/syslog-ng/syslog-ng.conf

And create the most simple network source just network()

source s_network {     network( transport(tcp) port(601));     network( transport(udp) port(602)); };

Now do a quick check and make sure syslog-ng is listening on that port.

> sudo netstat -peanut | grep syslog-ng

There we go

Now for the code

> vi port_log_test.py

 Here is my code

#!/usr/bin/python3 import sys import syslog import logging import logging.handlers #Set up Logger logger = logging.getLogger("10x13") logger.setLevel(logging.DEBUG) handler = logging.handlers.SysLogHandler(address=('localhost',602),facility=16) log_format = logging.Formatter('%(asctime)s, %(message)s') handler.setFormatter(log_format) logger.addHandler(handler) logger.info("Info Log Message") logger.debug("Debug Log Message") logger.error("Error log Message")

Make it executable and run it

> chmod u+x port_log_test.py   >./port_log_test.py

Now looking at the logs

 

Done deal!    The debug message was filtered out by the syslog f_network filter settings

filter f_network { facility(local0) and not level(debug); };

Wahoo it's all working !!

Oh and now since it is listening on UDP I can send a logger message over UDP

> logger -p local0.info --server 127.0.0.1 --udp --port 602 \       --rfc3164 "My UDP Message"

Nice!

References

[1]        syslog-ng install and configure.

http://www.whiteboardcoder.com/2017/03/syslog-ng-install-and-configure.html
Accessed 3/2017

[2]        network

https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-sources-network.html
Accessed 3/2017

[3]        Logger man page

https://linux.die.net/man/1/logger
Accessed 3/2017

[4]        Syslog-ng templates

https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-macros.html
Accessed 3/2017

[5]        Syslog-ng symbols

https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/syslog-ng-parameter-index.html
Accessed 3/2017