LXC 2.0 container not starting on Debian 9 Stretch when using cgroup limits

Written by - 0 comments

Published on February 18th 2019 - Listed in LXC Linux


I just hit a problem on a Debian 9 (Stretch) machine with the latest LXC 2.0.7 (package 2.0.7-2+deb9u2 from Debian repos) installed.

When I tried to run a LXC container with cgroup limits enabled, I got the following error:

# grep limit /var/lib/lxc/container/config
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G

# lxc-start -n container -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for container
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "container".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "container".
 lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

After some research I came across a very interesting thread in the linuxcontainers.org forums. There was indeed a problem in the 2.0.7 version but it was fixed in 2.0.8. The problem with Debian? Stretch still runs with 2.0.7 and has since quite a long time (over a year) according to the changelog:

lxc (1:2.0.7-2+deb9u2) stretch; urgency=medium

  * 0005-debian-Use-iproute2-instead-of-iproute.patch: pull iproute2 instead
    of iproute, fixing the creation of testing and unstable containers after
    the iproute binary package was dropped.

 -- Antonio Terceiro   Mon, 29 Jan 2018 20:23:36 -0200

lxc (1:2.0.7-2+deb9u1) stretch; urgency=medium

  * 0003-lxc-debian-don-t-hardcode-valid-releases.patch: don't
    hardcode list of valid Debian releases. Allows creating stable, buster,
    testing, and unstable containers.
  * 0004-lxc-debian-don-t-write-C.-locales-to-etc-locale.gen.patch: don't
    insert C.* locales into /etc/locale.gen (Closes: #879595)

 -- Antonio Terceiro   Fri, 27 Oct 2017 15:13:31 -0200

lxc (1:2.0.7-2) unstable; urgency=high

  * use bash-completion's pkg-config support and don't move files around
  * ignore lxc-test-cloneconfig if kernel has no overlay support
  * CVE-2017-5985: Ensure target netns is caller-owned (Closes: #857295)

 -- Evgeni Golov   Sat, 11 Mar 2017 09:47:20 +0100

lxc (1:2.0.7-1) unstable; urgency=medium

  * New upstream version 2.0.7
    + Closes: #847909, #847894, #847466

 -- Evgeni Golov   Mon, 23 Jan 2017 22:03:24 +0100

 According to the thread in the forums, the problem was fixed in lxcfs, partitcularly in the package libpam-cgfs. The discussion in the LXC forums also led to the report of Debian bug #867619. However this bug was only reported on the upcoming Debian 10 (Buster). According to one of the maintainers (Evgeni Golov) this was fixed in 2.0.7-2. The problem? The latest available package version in Stretch is as of today (February 18th 2019) still 2.0.7-1:

# apt-cache show libpam-cgfs
Package: libpam-cgfs
Source: lxcfs
Version: 2.0.7-1+deb9u1
Installed-Size: 47
Maintainer: pkg-lxc
Architecture: amd64
Depends: libc6 (>= 2.14), libfuse2 (>= 2.2), libpam0g (>= 0.99.7.1), libpam-runtime (>= 1.0.1-6), systemd | cgroupfs-mount
Conflicts: libpam-cgm
Description-en: PAM module for managing cgroups for LXC
 LXCFS provides a FUSE based filesystem to improve the LXC experience
 within the containers.
 .
 This provides a Pluggable Authentication Module (PAM) to provide
 logged-in users with a set of cgroups which they can administer.
 This allows for instance unprivileged containers, and session
 management using cgroup process tracking.
Description-md5: e709f3eddd48d5ce8595be4d003fd4f5
Homepage: https://linuxcontainers.org
Section: admin
Priority: optional
Filename: pool/main/l/lxcfs/libpam-cgfs_2.0.7-1+deb9u1_amd64.deb
Size: 18332
MD5sum: df18b81dc8e1dabffa7be5eaf586dc01
SHA256: 76e265bfb9a361db019c2fc1dc2ad6cf2b58cc62528f160c1107b77a6377af00

So how can this be tackled?

There are several possibilities:

Note: I haven't tried these yet! Stand by!

1) Use a manually fixed and prepared package of libpam-cgfs from the Ubuntu suite, packaged by Stéphane Graber:

https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu4/+build/12785691/+files/libpam-cgfs_2.0.7-0ubuntu4_amd64.deb

However these packages were made for an Ubuntu system, although they should be (pretty much) compatible with Debian Stretch.

2) Use the 2.0.7-2 packages from Debian maintainer Evgeni Golov:

https://people.debian.org/~evgeni/tmp/lxcfs/

However, these packages were made for Debian 10. To be tested...

3) Use Debian Stretch backports. 

stretch-backports offers the 2.0.8 version of LXC and related packages:

  • lxcfs (2.0.8-1~bpo9+1)
  • libpam-cgfs (2.0.8-1~bpo9+1)

Interestingly the last comment in the lxcfs package in backports (Pierre-Elliott Bécue) wrote this in the changelog:

lxcfs (2.0.8-1~bpo9+1) stretch-backports; urgency=medium

  * Team upload
  * Rebuild for stretch-backports.
  * This backport release is an alternative to 2.0.7-1 that has a couple of
    issues, and shouldn't have reached stable.
    See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more
    intel.

 -- Pierre-Elliott Bécue   Sat, 17 Nov 2018 09:01:07 +0100

 "This backport release is an alternative to 2.0.7-1 that has a couple of issues, and shouldn't have reached stable. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more intel."

Ah, and here is the mentioned bug again ;-)

So let's try and see which of these potential solutions work.

Update February 19th 2019:
Today I continued my tests and finally got the cgroup limits to work on Debian Stretch with the following packages installed:

# dpkg -l|egrep "(lxc|libpam-cgfs)"
ii  liblxc1          1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools (library)
ii  libpam-cgfs      2.0.7-1+deb9u1      amd64        PAM module for managing cgroups for LXC
ii  lxc              1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools
ii  lxcfs            2.0.7-1+deb9u1      amd64        FUSE based filesystem for LXC
ii  python3-lxc      1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools (Python 3.x bindings)

On another Debian Stretch server I also successfully tested it with a newer lxcfs package from debian stretch-backports (2.0.8-1~bpo9+1).

Additional Kernel parameters (cgroup_enable=memory swapaccount=1) were set in /etc/default/grub.

However as soon as I touched lxcfs (package upgrade or downgrade) I needed a reboot, otherwise I would get the following error when trying to start a container with cgroup limits:

# lxc-start -n test -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for test
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "test".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "test".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

Because lxcfs doesn't run anymore once the package was touched:

# systemctl status lxcfs
? lxcfs.service - FUSE filesystem for LXC
   Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-02-19 07:17:38 CET; 13h ago
     Docs: man:lxcfs(1)
 Main PID: 31389 (code=exited, status=1/FAILURE)
      CPU: 5ms

After a reboot:

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=UUID=XXXXXXXX-XXXX-XXXX-XXXXXXXXXXXX ro quiet cgroup_enable=memory swapaccount=1

# lxc-start -n test -d
# lxc-ls -f
NAME   STATE   AUTOSTART GROUPS IPV4           IPV6
test   RUNNING 0         -      192.168.12.199 -    

The following cgroup limits were set by the way:

lxc.cgroup.cpuset.cpus = 1-12
lxc.cgroup.cpu.shares = 1024
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.