Monitoring memory usage of a LXC container (comparing 1.x vs 2.x)

Written by Claudio Kuenzler - 0 comments

Published on July 18th 2018 - Listed in Monitoring Linux LXC Virtualization

---

Monitoring Linux Containers (LXC, also known as System Containers to separate from the Docker world) is as important as you monitor your LXC host. But the usage view inside a container is sometimes "unreal". 

Back in 2013, when I started the monitoring plugin check_lxc, the only way to really check the memory usage of a container was to check the current cgroup values of the container:

root@lxchost:~# /usr/lib/nagios/plugins/check_lxc.sh -n container1 -t mem
LXC container1 OK - Used Memory: 6187 MB|mem=6488358912B;0;0;0;0

In the background of check_lxc.sh, the cgroup values of container1 are read. But why so complicated and not just run a classic check_mem.pl inside the container?

To answer that question, take a look at the following picture:

Focus on the memory usage; both the LXC host (top), running LXC 1.x, and the LXC container (bottom) show the exact same values.

Or to see it in text form:

root@lxchost:~# free -m
             total       used       free     shared    buffers     cached
Mem:         32176      30911       1264        119       1855      21921
-/+ buffers/cache:       7135      25041
Swap:         3814        165       3649


root@container1:~# free -m
             total       used       free     shared    buffers     cached
Mem:         32176      30911       1264        119       1855      21921
-/+ buffers/cache:       7135      25041
Swap:         3814        165       3649

The container only sees the same values as the host. But the container itself only uses 6187 MB according to cgroups, not 7135 MB.

That's why you should use check_lxc on the host to get a more accurate memory usage of the containers.

Until recently.

Now that I'm working on a new LXC environment on Debian Stretch, there's a newer LXC version (LXC 2.x). Something immediately caught my eye the first time I ran (h)top:

Focus again on the memory usage. This time the LXC host (top) and the LXC container (bottom) have different values. True, the (cpu) load and the swap usage is still the same on both host and container, but it's already something!

Doing the same check with free:

root@lxchost:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          64421       21419         361         179       42639       42173
Swap:         15258         668       14590


root@container1:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          64421        1038       62770         179         612       62770
Swap:         15258         668       14590

Note: These are different hosts and different containers than the values seen above from LXC 1.x.

Both host and container show the total capacity of memory and swap, but the used column clearly shows a difference.
But don't be fooled: The calculation on the available memory (last column) is kind of wrong in the container. That is because the container cannot know about the other containers running beside it and is therefore unaware of other memory consumers.

What about check_lxc in that case?

root@lxchost:~# /usr/lib/nagios/plugins/check_lxc.sh -n container1 -t mem
LXC container1 OK - Used Memory: 1596 MB|mem=1673646080B;0;0;0;0

The host tells us the container is using 1596 MB, which is almost the same value as 1038 (used) + 612 (buff/cache) (=1650 MB).

The big question now: Can check_mem.pl be used inside the container and give accurate alerts?

root@container1:~# ./check_mem.pl -u -w 90 -c 95
OK - 2.6% (1693488 kB) used.|TOTAL=65967908KB;;;; USED=1693488KB;59371117;62669512;; FREE=64274420KB;;;; CACHES=443236KB;;;;

The answer is: No. Because the check_mem.pl plugin (as of today) makes a calculation based on the "free" output from above. And as long as these are kind of incorrect, the container's consumption of resources (disk, memory, cpu) should still be monitored on the host.

If you'd create a script/plugin which only checks the "used" value, you're probably good to go though.

But let's focus on the good news: When you're logged into the container and you run (h)top you now see the (more or less) correct memory consumption of the container. That's already a big improvement and really helpful.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.