Updating Ubuntu with cross versions on LXC hosts and containers: Watch the gotchas!

Written by - 0 comments

Published on - last updated on April 29th 2019 - Listed in Linux Ubuntu LXC Containers


In a previous post not long ago, I wrote that this April 2019 will be the (official) end of the long term support (LTS) of Ubuntu 14.04, codenamed Trusty. I wrote a step by step guide how to upgrade an Ubuntu 14.04 server via 16.04 to 18.04 (see It is 2019, time to upgrade your Ubuntu 14.04 Trusty machines!). This guide covers the basics how to upgrade the Ubuntu distribution and all packages coming from Ubuntu repositories and should work in most setups.

However there are always the more complex scenarios out there. One of such a scenario is one of our LXC environments. The LXC hosts are currently running on Ubuntu 14.04, most of the LXC containers were installed with Ubuntu 16.04.
Note: This is by default not possible due to the different init system of host and containers. However there's a workaround by removing systemd and manually installing upstart-sysv. See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for more details.

In the past few days I've been testing which would be the best order to upgrade the Ubuntu versions. Should I start with the containers first, followed by the host? Or should the host have the newest version first and then I upgrade the containers? What about intermediate steps? Will the containers still run on an old 14.04 when the host is on 18.04 and vice versa?

After a couple of upgrades in a TEST environment, I came to the following conclusion and compatibility matrix:

LXC Host LXC Guest Compatibility Notes
14.04 14.04 yes -
14.04 16.04
no* System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to talk to init daemon.
14.04 18.04 no System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to talk to init daemon.
16.04 14.04 yes -
16.04 16.04 yes -
16.04 18.04 yes -
18.04 14.04 no upstart init not started, applications could be started manually though. 
e.g. /etc/init.d/ssh start
https://github.com/lxc/lxc/issues/903
18.04 16.04 yes -
18.04 18.04 yes -

* again:  See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for a workaround.

Luckliy there is 16.04 as an intermediate version available! It's the version in between being capable of talking correctly to the older upstart and newer systemd init systems!

The worst situations happened when the host staid at 14.04 and all the containers were upgraded but also when the host was upgraded to 18.04 but all the containers remained at 14.04.
Both situations caused the containers to now work correctly. Although they were started by LXC itself, the init system of the containers didn't work anymore due to an incompatibility between Systemd and the previous (Ubuntu-default) init system upstart.

TL;DR

When you need to upgrade an LXC environment (both host and containers need upgrades) and you run multiple versions across host and containers, these are the steps you should consider as your upgrade plan:

1) Upgrade the LXC hosts from 14.04 to 16.04. This will make sure that all containers from 14.04 to 18.04 will start correctly.
2) Upgrade all the containers from 14.04 to 16.04.
3) Upgrade the LXC hosts from 16.04 to 18.04. Containers with 16.04 run on SystemD by default, so they run fine on a host with 18.04.
4) Upgrade all the containers from 16.04 to 18.04.

This will help you reduce the downtimes of the whole LXC environment and therefore your applications.

Updated April 29th 2019:

Here's a handy way to find out under which Linux distribution and which version your containers are running. On the host run os-prober:

root@lxchost:~# os-prober
/dev/mapper/vglxc-dpm--heske01--t:Ubuntu 16.04.2 LTS (16.04):Ubuntu:linux
/dev/mapper/vglxc-inf--jira01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu1:linux
/dev/mapper/vglxc-inf--ldap01--t:Ubuntu 16.04.1 LTS (16.04):Ubuntu2:linux
/dev/mapper/vglxc-onl--mysql01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu3:linux
/dev/mapper/vglxc-okapi--app01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu4:linux
/dev/mapper/vglxc-onl--redis01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu5:linux
/dev/mapper/vglxc-onlkc--web01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu6:linux
/dev/mapper/vglxc-shop--web01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu7:linux
/dev/mapper/vglxc-st--cdb01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu8:linux
/dev/mapper/vglxc-ver--dks01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu9:linux


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder