Application (Docker/Kubernetes) containers and STDOUT logging

Written by - 1 comments

Published on - Listed in Docker Containers Linux ELK

In our Docker container environment (on premise, using Rancher) I have configured the Docker daemon to forward STDOUT logs from the containers to a central Logstash using GELF.

For applications logging by default to STDOUT this works out of the box. But for some hand-written applications this might require some additional work.

In this particular example the application simply logged into a local log file on the AUFS filesystem (/tmp/application.log). But all these log messages of course never arrive in the ELK stack because they were not logged to STDOUT but written into a file.

The developer then adjusted the Dockerfile and instead of creating the log file, created a symlink:

# forward logs to docker log collector
RUN ln -sf /dev/stdout /tmp/application.log

To be honest, I thought this would do the trick. But once the new container image was deployed, the application logs didn't arrive in our ELK stack. Why?

I went into the container and tested myself:

root@af8e2147f8ba:/app# cd /tmp/

root@af8e2147f8ba:/tmp# ls -la
total 12
drwxrwxrwt  3 root root 4096 Jan 15 12:55 .
drwxr-xr-x 54 root root 4096 Jan 15 12:57 ..
lrwxrwxrwx  1 root root   11 Jan 15 12:52 application.log -> /dev/stdout
drwxr-xr-x  3 root root 4096 Jan 15 12:52 npm-6-d456bc8a

Yes, there is the  application log file, which is a symlink to /dev/stdout. Should work, right? Let's try this:

root@af8e2147f8ba:/tmp# echo "test test test" > application.log
test test test

Although I saw "test test test" appearing in the terminal, this message never made it into the ELK stack. On my research why, I came across a VERY GOOD explanation by user "phemmer" on this GitHub issue:

"The reason this doesn't work is because /dev/stdout is a link to STDOUT of the process accessing it. So by doing foo > /dev/stdout, you're saying "redirect my STDOUT to my STDOUT". Kinda doesn't do anything :-).
And since /var/log/test.log is a symlink to it, the same thing applies. What you want is to redirect output to STDOUT of PID 1. PID 1 is the process launched by docker, and its STDOUT will be what docker picks up."

So to sum this up, we need to use the STDOUT of PID 1 (the container itself), otherwise the message won't be picked up by the Docker daemon.

Let's try this inside the still running container:

root@af8e2147f8ba:/tmp# rm application.log
root@af8e2147f8ba:/tmp# ln -sf /proc/1/fd/1 /tmp/application.log
root@af8e2147f8ba:/tmp# echo 1 2 3 > application.log

And hey, my 1 2 3 appeared in Kibana!

Docker container logs STDOUT logging

I slightly modified the Dockerfile with that new knowledge:

# forward logs to docker log collector
RUN ln -sf /proc/1/fd/1 /tmp/application.log

Note: /proc/1 obviously is PID 1. fd/1 is stdout, as you might know from typical cron jobs, e.g. */5 * * * * 2>&1. fd/2 would be STDERR by the way.

After the new container image was built, deployed and started, the ELK stack is now getting the application logs:

Container logs appearing in ELK stack

Add a comment

Show form to leave a comment

Comments (newest first)

Carl J. Mosca from wrote on May 30th, 2019:

Thank you. Excellent tip!!

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder