LXC 2.0 container not starting on Debian 9 Stretch when using cgroup limits

Written by - 0 comments

Published on - Listed in LXC Linux


I just hit a problem on a Debian 9 (Stretch) machine with the latest LXC 2.0.7 (package 2.0.7-2+deb9u2 from Debian repos) installed.

When I tried to run a LXC container with cgroup limits enabled, I got the following error:

# grep limit /var/lib/lxc/container/config
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G

# lxc-start -n container -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for container
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "container".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "container".
 lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

After some research I came across a very interesting thread in the linuxcontainers.org forums. There was indeed a problem in the 2.0.7 version but it was fixed in 2.0.8. The problem with Debian? Stretch still runs with 2.0.7 and has since quite a long time (over a year) according to the changelog:

lxc (1:2.0.7-2+deb9u2) stretch; urgency=medium

  * 0005-debian-Use-iproute2-instead-of-iproute.patch: pull iproute2 instead
    of iproute, fixing the creation of testing and unstable containers after
    the iproute binary package was dropped.

 -- Antonio Terceiro   Mon, 29 Jan 2018 20:23:36 -0200

lxc (1:2.0.7-2+deb9u1) stretch; urgency=medium

  * 0003-lxc-debian-don-t-hardcode-valid-releases.patch: don't
    hardcode list of valid Debian releases. Allows creating stable, buster,
    testing, and unstable containers.
  * 0004-lxc-debian-don-t-write-C.-locales-to-etc-locale.gen.patch: don't
    insert C.* locales into /etc/locale.gen (Closes: #879595)

 -- Antonio Terceiro   Fri, 27 Oct 2017 15:13:31 -0200

lxc (1:2.0.7-2) unstable; urgency=high

  * use bash-completion's pkg-config support and don't move files around
  * ignore lxc-test-cloneconfig if kernel has no overlay support
  * CVE-2017-5985: Ensure target netns is caller-owned (Closes: #857295)

 -- Evgeni Golov   Sat, 11 Mar 2017 09:47:20 +0100

lxc (1:2.0.7-1) unstable; urgency=medium

  * New upstream version 2.0.7
    + Closes: #847909, #847894, #847466

 -- Evgeni Golov   Mon, 23 Jan 2017 22:03:24 +0100

 According to the thread in the forums, the problem was fixed in lxcfs, partitcularly in the package libpam-cgfs. The discussion in the LXC forums also led to the report of Debian bug #867619. However this bug was only reported on the upcoming Debian 10 (Buster). According to one of the maintainers (Evgeni Golov) this was fixed in 2.0.7-2. The problem? The latest available package version in Stretch is as of today (February 18th 2019) still 2.0.7-1:

# apt-cache show libpam-cgfs
Package: libpam-cgfs
Source: lxcfs
Version: 2.0.7-1+deb9u1
Installed-Size: 47
Maintainer: pkg-lxc
Architecture: amd64
Depends: libc6 (>= 2.14), libfuse2 (>= 2.2), libpam0g (>= 0.99.7.1), libpam-runtime (>= 1.0.1-6), systemd | cgroupfs-mount
Conflicts: libpam-cgm
Description-en: PAM module for managing cgroups for LXC
 LXCFS provides a FUSE based filesystem to improve the LXC experience
 within the containers.
 .
 This provides a Pluggable Authentication Module (PAM) to provide
 logged-in users with a set of cgroups which they can administer.
 This allows for instance unprivileged containers, and session
 management using cgroup process tracking.
Description-md5: e709f3eddd48d5ce8595be4d003fd4f5
Homepage: https://linuxcontainers.org
Section: admin
Priority: optional
Filename: pool/main/l/lxcfs/libpam-cgfs_2.0.7-1+deb9u1_amd64.deb
Size: 18332
MD5sum: df18b81dc8e1dabffa7be5eaf586dc01
SHA256: 76e265bfb9a361db019c2fc1dc2ad6cf2b58cc62528f160c1107b77a6377af00

So how can this be tackled?

There are several possibilities:

Note: I haven't tried these yet! Stand by!

1) Use a manually fixed and prepared package of libpam-cgfs from the Ubuntu suite, packaged by Stéphane Graber:

https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu4/+build/12785691/+files/libpam-cgfs_2.0.7-0ubuntu4_amd64.deb

However these packages were made for an Ubuntu system, although they should be (pretty much) compatible with Debian Stretch.

2) Use the 2.0.7-2 packages from Debian maintainer Evgeni Golov:

https://people.debian.org/~evgeni/tmp/lxcfs/

However, these packages were made for Debian 10. To be tested...

3) Use Debian Stretch backports. 

stretch-backports offers the 2.0.8 version of LXC and related packages:

  • lxcfs (2.0.8-1~bpo9+1)
  • libpam-cgfs (2.0.8-1~bpo9+1)

Interestingly the last comment in the lxcfs package in backports (Pierre-Elliott Bécue) wrote this in the changelog:

lxcfs (2.0.8-1~bpo9+1) stretch-backports; urgency=medium

  * Team upload
  * Rebuild for stretch-backports.
  * This backport release is an alternative to 2.0.7-1 that has a couple of
    issues, and shouldn't have reached stable.
    See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more
    intel.

 -- Pierre-Elliott Bécue   Sat, 17 Nov 2018 09:01:07 +0100

 "This backport release is an alternative to 2.0.7-1 that has a couple of issues, and shouldn't have reached stable. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more intel."

Ah, and here is the mentioned bug again ;-)

So let's try and see which of these potential solutions work.

Update February 19th 2019:
Today I continued my tests and finally got the cgroup limits to work on Debian Stretch with the following packages installed:

# dpkg -l|egrep "(lxc|libpam-cgfs)"
ii  liblxc1          1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools (library)
ii  libpam-cgfs      2.0.7-1+deb9u1      amd64        PAM module for managing cgroups for LXC
ii  lxc              1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools
ii  lxcfs            2.0.7-1+deb9u1      amd64        FUSE based filesystem for LXC
ii  python3-lxc      1:2.0.7-2+deb9u2    amd64        Linux Containers userspace tools (Python 3.x bindings)

On another Debian Stretch server I also successfully tested it with a newer lxcfs package from debian stretch-backports (2.0.8-1~bpo9+1).

Additional Kernel parameters (cgroup_enable=memory swapaccount=1) were set in /etc/default/grub.

However as soon as I touched lxcfs (package upgrade or downgrade) I needed a reboot, otherwise I would get the following error when trying to start a container with cgroup limits:

# lxc-start -n test -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for test
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "test".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "test".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

Because lxcfs doesn't run anymore once the package was touched:

# systemctl status lxcfs
? lxcfs.service - FUSE filesystem for LXC
   Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-02-19 07:17:38 CET; 13h ago
     Docs: man:lxcfs(1)
 Main PID: 31389 (code=exited, status=1/FAILURE)
      CPU: 5ms

After a reboot:

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=UUID=XXXXXXXX-XXXX-XXXX-XXXXXXXXXXXX ro quiet cgroup_enable=memory swapaccount=1

# lxc-start -n test -d
# lxc-ls -f
NAME   STATE   AUTOSTART GROUPS IPV4           IPV6
test   RUNNING 0         -      192.168.12.199 -    

The following cgroup limits were set by the way:

lxc.cgroup.cpuset.cpus = 1-12
lxc.cgroup.cpu.shares = 1024
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   


Update cookies preferences