How to ignore (discard) certain syslog messages in rsyslogd using filters

Written by - 2 comments

Published on - Listed in Linux ELK


A central syslog server, using rsyslogd, was being flooded with syslog messages from surrounding servers. Docker logs in particular were filling up the central syslog server's /var/log/syslog in a short amount of time:

root@syslog:~# grep "docker" /var/log/syslog
Jan  7 13:16:18 docker11 systemd[1]: run-docker-runtime\x2drunc-moby-b5f31be74b826201bd84ce712581bfceffaf192b024d68375d94f0947af4ff07-runc.0OXJAR.mount: Succeeded.
Jan  7 13:16:18 docker01 systemd[1]: run-docker-runtime\x2drunc-moby-3ee4af24c1c3cd3f79b579fc81ac0d5af34e78647185aa4e8a2c02d3903b0ec0-runc.KjU36D.mount: Succeeded.
Jan  7 13:16:18 docker23 systemd[1]: run-docker-runtime\x2drunc-moby-b20db8928161fc671b942fc82f98ea3a86ac6c9d09c15a880ecdb416d16700c5-runc.N89XuM.mount: Succeeded.
Jan  7 13:16:18 docker11 systemd[1]: run-docker-runtime\x2drunc-moby-ae1234beb6d40cdc6bdb0e8da07461c6958c130959e1cc5f4cebc037cdce5bab-runc.wTfEVk.mount: Succeeded.
Jan  7 13:16:19 docker24 systemd[1]: run-docker-runtime\x2drunc-moby-6aa9b35b3053ce45cf77ba6707a1e183374bce314c69fb999dd2091f1a6fc083-runc.Z6TlqD.mount: Succeeded.
[...]

A ton of mount:Succeeded log events were logged, multiple times per second.

This is fine on the Docker host's local syslog file, but on the central syslog server (which forwards the logs to an ELK stack) this is just too much (non-helpful) information.

Luckily rsyslogd is capable to discard syslog messages using filters

Discarding logs which contain a certain content

In the following example the syslog message field (referred to as ":msg") can be checked if the content contains a certain string:

root@syslog:~# cat /etc/rsyslog.d/10-filter-docker-syslog.conf
# Filter out messages like these:
# Jan  7 13:08:44 docker01 systemd[1]: run-docker-runtime\x2drunc-moby-4cb10df07f04c27fc12255faf5d8d58acdc4ca5fc99b7d59088048022b1d2f38-runc.MprpAa.mount: Succeeded.
# Jan  7 13:14:31 docker23 systemd[1]: var-lib-docker-overlay2-9fcf3bf476a8337799f1f3a58c7f74ce88856f300a62ae63c5549fb6d0e89714-merged.mount: Succeeded.
:msg, contains, "run-docker-runtime"    stop
:msg, contains, "var-lib-docker-overlay2"    stop

The "stop" at the end is the action which rsyslog should take. In this case (stop) the message is simply ignored/discarded. 

Note: In previous rsyslog versions, the hyphen character (~) was used instead of stop

Discarding logs of a specific program/process name

The same works also with other syslog fields. For example if all syslog messages from the systemd process should be discarded:

root@syslog:~# cat /etc/rsyslog.d/10-filter-systemd.conf
# Filter out all systemd messages:
:programname, isequal, "systemd"    stop

Discarding logs from a specific syslog server

Syslog messages from a specific syslog server can also be ignored:

root@syslog:~# cat /etc/rsyslog.d/10-filter-remote-syslogserver.conf
# Filter out messages from a spammy syslog server:
:fromhost-ip, isequal, "10.10.2.133"    stop

More filters using properties and conditions

In general there are many possibilities to create rsyslog filters. Basically the syntax is the following:

:field, condition, "search string" action

A full list of field names can be found in the rsyslog properties documentation.

A full list of possible conditions can be found in the compare operations documentation.

Always make sure to reload or restart rsyslogd after a config change.

The config order is important!

Last but not least it's important to note that these "filter configs" should be named correctly. Rsyslog (by default) reads all *.conf files from the /etc/rsyslog.d/ directory in an alphabetical order. The filters should happen before the file "50-default.conf" is loaded. This is the config responsible for writing the syslog messages into files.

Therefore if you start all your filters with a number prior to 50, the filters should work. In the following example all filter configs start with a 10:

root@syslog:~# ll /etc/rsyslog.d/
total 32
--w----r-T 1 root root  706 Aug 14  2017 01-json.conf
-rw-r--r-- 1 root root  456 Jan  7 13:15 10-filter-docker-syslog.conf
-rw-r--r-- 1 root root   93 Jan  7 14:42 10-filter-remote-syslogserver.conf

-rw-r--r-- 1 root root  314 Sep  8  2015 20-ufw.conf
-rwxr-xr-x 1 root root  255 Jun 29  2017 21-cloudinit.conf
-rw-r--r-- 1 root root 1124 Jan 30  2018 50-default.conf
--w----r-T 1 root root  108 Jan  7 13:30 99-remote.conf
-rw-r--r-- 1 root root  242 Apr 13  2016 postfix.conf



Add a comment

Show form to leave a comment

Comments (newest first)

D McKeon from Oregon, USA wrote on Jun 25th, 2022:

Thanks for an informative post. FYI,
the hyphen '-' or dash character is %2d,
the '~' at %7e can be called 'tilde'

dec oct hex 76543210 char
45 55 2d 00101101 -
126 176 7e 01111110 ~


noWinToday from wrote on May 24th, 2022:

nice, but i can't get it to work :(
i had it used some years ago, but now i've inherited a server and cannot get it working again.
rsyslog.conf contains this:
:msg, contains, "Connection discarded" stop
some templates like this:
template(name="stonegate" type="string" string="/var/log/rsyslog/stonegate/SG_%$YEAR%-%$MONTH%-%$DAY%.log")
a ruleset with some actions like this:
ruleset(name="remote-syslog-514") {
if ( $fromhost-ip == '172.18.2.60' or $fromhost-ip == '172.18.173.20' ) then
{
action(type="omfile" DynaFile="stonegate" dynaFileCacheSize="50" asyncWriting="on" flushInterval="2" ioBufferSize="1024")
stop
}
else if ( $fromhost-ip ...
module(load="imudp")
input(type="imudp" port="514" ruleset="remote-syslog-514")
module(load="imptcp")
input(type="imptcp" port="514" ruleset="remote-syslog-514")


RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   


Update cookies preferences