Debian Wheezy LXC container not running on Stretch host with Kernel 4.18 bpo anymore (a vsyscall story)

Written by Claudio Kuenzler - 0 comments

Published on May 13th 2019 - last updated on May 15th 2019 - Listed in Linux LXC Security

---

While migrating an old host still running with Debian Wheezy (7) and one remaining Linux container (also running with Wheezy), I came across an interesting problem.

The problem

The new host runs with Debian Stretch (9) and a Kernel 4.18 from stretch-backports to better support the AMD Ryzen processor (see article Linux going AMD Ryzen with Debian 9 (Stretch)). After the rootfs of the container was synced from the Wheezy LXC host to the Stretch LXC host, the container didn't start up. Or better said, it started up, but then died a few seconds later:

root@stretch ~ # lxc-start -n wheezycontainer -d
root@stretch ~ # lxc-ls -f|grep wheezycontainer
wheezycontainer  RUNNING 1         -      192.168.15.177 -   
root@stretch ~ # lxc-ls -f|grep wheezycontainer
wheezycontainer  STOPPED 1         -      -              -   

To figure out what's going on, I decided to chroot into the container and see for myself.

root@stretch ~ # mount --bind /proc /var/lib/lxc/wheezycontainer/rootfs/proc
root@stretch ~ # mount --bind /sys /var/lib/lxc/wheezycontainer/rootfs/sys
root@stretch ~ # mount --bind /dev /var/lib/lxc/wheezycontainer/rootfs/dev
root@stretch ~ # chroot /var/lib/lxc/wheezycontainer/rootfs /bin/sh
\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]
\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]
\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]ps auxf
Signal 11 (SEGV) caught by ps (procps-ng version 3.3.3).
ps:display.c:59: please report this bug

Holy sh!t, I did not see that coming! All the commands I tried returned in a segmentation fault:

\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]top
signal 11 (SEGV) was caught by top, please
see http://www.debian.org/Bugs/Reporting

The reason

A look into dmesg (at least this command works!) revealed something interesting:

 \[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]dmesg | tail
[494840.491954] ps[24332] vsyscall attempted with vsyscall=none ip:ffffffffff600400 cs:33 sp:7ffe33393128 ax:ffffffffff600400 si:1000 di:0
[494880.210536] top[24794] vsyscall attempted with vsyscall=none ip:ffffffffff600400 cs:33 sp:7ffc63fe5158 ax:ffffffffff600400 si:2244e90 di:7ffc63fe5178
\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]$ \[\033[00m\]

Here we can see that both commands (ps and top) used before attempted to use a vsyscall. What exactly is a vsyscall? On my research I came across a very good explanation in a mailing list post by Nathaniel Smith. Here are some excerpts of it:

The "vsyscall" mechanism is a clever hack/trick that Linux uses to speed up some syscalls (most notably gettimeofday), and involves the kernel injecting some code in processes' memory maps, that glibc then knows to call. [...] the kernel always injected it at the same fixed address, so glibc was hard-coded to just "know" that e.g. gettimeofday was at 0xffffffffff600400. [...]

In these less trusting times, these hard-coded addresses are considered a security risk (they violate ASLR etc.), so they were deprecated a long time ago.

[...] if you try to run an old binary that blindly uses the hardcoded addresses then it will segfault as soon as it tries to call gettimeofday.

Debian has recently flipped the switch to disable this on their kernels, so if you're running a recent Debian testing or unstable (kernel 4.8 or better) [...]

The workaround is to reboot and add the option 'vsyscall=emulate' to the kernel command line.

Another good hint can be found in a public forum post which describes the same dmesg log entries.

I'm running several Debian Stretch servers and this Ryzen machine is the only one using the 4.18 Kernel from backports, this Wheezy container issue only happens on the newer Kernel. But in general that means that in the future (probably with Debian Buster becoming stable) we will see compatibility issues with Wheezy containers.

The Workarounds

So what's the solution? There is no satisfying solution to this. There are workarounds:

• As mentioned by Nathaniel Smith above, you can add the option vsyscall=emulate to the Kernel's cmdline (most of the times in /etc/default/grub on Debian systems). But this will allow vsyscalls which violate ASLR. And ASLR (address space layout randomization) is a good mechanism against zero day exploits!
• You can still run the Wheezy container as a full virtual machine, as long as you install a Kernel inside the container.
  
• Upgrade or migrate the Wheezy container and the software running on it to a newer Debian. Of course that's wishful thinking and is not always possible (depending on the age of the running software).
  

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.