I just hit a problem on a Debian 9 (Stretch) machine with the latest LXC 2.0.7 (package 2.0.7-2+deb9u2 from Debian repos) installed.
When I tried to run a LXC container with cgroup limits enabled, I got the following error:
# grep limit /var/lib/lxc/container/config
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G
# lxc-start -n container -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for container
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "container".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "container".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
After some research I came across a very interesting thread in the linuxcontainers.org forums. There was indeed a problem in the 2.0.7 version but it was fixed in 2.0.8. The problem with Debian? Stretch still runs with 2.0.7 and has since quite a long time (over a year) according to the changelog:
lxc (1:2.0.7-2+deb9u2) stretch; urgency=medium
* 0005-debian-Use-iproute2-instead-of-iproute.patch: pull iproute2 instead
of iproute, fixing the creation of testing and unstable containers after
the iproute binary package was dropped.
-- Antonio Terceiro
lxc (1:2.0.7-2+deb9u1) stretch; urgency=medium
* 0003-lxc-debian-don-t-hardcode-valid-releases.patch: don't
hardcode list of valid Debian releases. Allows creating stable, buster,
testing, and unstable containers.
* 0004-lxc-debian-don-t-write-C.-locales-to-etc-locale.gen.patch: don't
insert C.* locales into /etc/locale.gen (Closes: #879595)
-- Antonio Terceiro
lxc (1:2.0.7-2) unstable; urgency=high
* use bash-completion's pkg-config support and don't move files around
* ignore lxc-test-cloneconfig if kernel has no overlay support
* CVE-2017-5985: Ensure target netns is caller-owned (Closes: #857295)
-- Evgeni Golov
lxc (1:2.0.7-1) unstable; urgency=medium
* New upstream version 2.0.7
+ Closes: #847909, #847894, #847466
-- Evgeni Golov
According to the thread in the forums, the problem was fixed in lxcfs, partitcularly in the package libpam-cgfs. The discussion in the LXC forums also led to the report of Debian bug #867619. However this bug was only reported on the upcoming Debian 10 (Buster). According to one of the maintainers (Evgeni Golov) this was fixed in 2.0.7-2. The problem? The latest available package version in Stretch is as of today (February 18th 2019) still 2.0.7-1:
# apt-cache show libpam-cgfs
Package: libpam-cgfs
Source: lxcfs
Version: 2.0.7-1+deb9u1
Installed-Size: 47
Maintainer: pkg-lxc
Architecture: amd64
Depends: libc6 (>= 2.14), libfuse2 (>= 2.2), libpam0g (>= 0.99.7.1), libpam-runtime (>= 1.0.1-6), systemd | cgroupfs-mount
Conflicts: libpam-cgm
Description-en: PAM module for managing cgroups for LXC
LXCFS provides a FUSE based filesystem to improve the LXC experience
within the containers.
.
This provides a Pluggable Authentication Module (PAM) to provide
logged-in users with a set of cgroups which they can administer.
This allows for instance unprivileged containers, and session
management using cgroup process tracking.
Description-md5: e709f3eddd48d5ce8595be4d003fd4f5
Homepage: https://linuxcontainers.org
Section: admin
Priority: optional
Filename: pool/main/l/lxcfs/libpam-cgfs_2.0.7-1+deb9u1_amd64.deb
Size: 18332
MD5sum: df18b81dc8e1dabffa7be5eaf586dc01
SHA256: 76e265bfb9a361db019c2fc1dc2ad6cf2b58cc62528f160c1107b77a6377af00
So how can this be tackled?
There are several possibilities:
Note: I haven't tried these yet! Stand by!
1) Use a manually fixed and prepared package of libpam-cgfs from the Ubuntu suite, packaged by Stéphane Graber:
However these packages were made for an Ubuntu system, although they should be (pretty much) compatible with Debian Stretch.
2) Use the 2.0.7-2 packages from Debian maintainer Evgeni Golov:
https://people.debian.org/~evgeni/tmp/lxcfs/
However, these packages were made for Debian 10. To be tested...
3) Use Debian Stretch backports.
stretch-backports offers the 2.0.8 version of LXC and related packages:
Interestingly the last comment in the lxcfs package in backports (Pierre-Elliott Bécue) wrote this in the changelog:
lxcfs (2.0.8-1~bpo9+1) stretch-backports; urgency=medium
* Team upload
* Rebuild for stretch-backports.
* This backport release is an alternative to 2.0.7-1 that has a couple of
issues, and shouldn't have reached stable.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more
intel.
-- Pierre-Elliott Bécue
"This backport release is an alternative to 2.0.7-1 that has a couple of issues, and shouldn't have reached stable. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867619 for more intel."
Ah, and here is the mentioned bug again ;-)
So let's try and see which of these potential solutions work.
Update February 19th 2019:
Today I continued my tests and finally got the cgroup limits to work on Debian Stretch with the following packages installed:
# dpkg -l|egrep "(lxc|libpam-cgfs)"
ii liblxc1 1:2.0.7-2+deb9u2 amd64 Linux Containers userspace tools (library)
ii libpam-cgfs 2.0.7-1+deb9u1 amd64 PAM module for managing cgroups for LXC
ii lxc 1:2.0.7-2+deb9u2 amd64 Linux Containers userspace tools
ii lxcfs 2.0.7-1+deb9u1 amd64 FUSE based filesystem for LXC
ii python3-lxc 1:2.0.7-2+deb9u2 amd64 Linux Containers userspace tools (Python 3.x bindings)
On another Debian Stretch server I also successfully tested it with a newer lxcfs package from debian stretch-backports (2.0.8-1~bpo9+1).
Additional Kernel parameters (cgroup_enable=memory swapaccount=1) were set in /etc/default/grub.
However as soon as I touched lxcfs (package upgrade or downgrade) I needed a reboot, otherwise I would get the following error when trying to start a container with cgroup limits:
# lxc-start -n test -F
lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1949 Permission denied - Error setting memory.memsw.limit_in_bytes to 68G for test
lxc-start: start.c: lxc_spawn: 1190 Failed to setup cgroup limits for container "test".
lxc-start: start.c: __lxc_start: 1346 Failed to spawn container "test".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
Because lxcfs doesn't run anymore once the package was touched:
# systemctl status lxcfs
? lxcfs.service - FUSE filesystem for LXC
Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-02-19 07:17:38 CET; 13h ago
Docs: man:lxcfs(1)
Main PID: 31389 (code=exited, status=1/FAILURE)
CPU: 5ms
After a reboot:
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=UUID=XXXXXXXX-XXXX-XXXX-XXXXXXXXXXXX ro quiet cgroup_enable=memory swapaccount=1
# lxc-start -n test -d
# lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6
test RUNNING 0 - 192.168.12.199 -
The following cgroup limits were set by the way:
lxc.cgroup.cpuset.cpus = 1-12
lxc.cgroup.cpu.shares = 1024
lxc.cgroup.memory.limit_in_bytes = 64G
lxc.cgroup.memory.memsw.limit_in_bytes = 68G
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder