In a previous post not long ago, I wrote that this April 2019 will be the (official) end of the long term support (LTS) of Ubuntu 14.04, codenamed Trusty. I wrote a step by step guide how to upgrade an Ubuntu 14.04 server via 16.04 to 18.04 (see It is 2019, time to upgrade your Ubuntu 14.04 Trusty machines!). This guide covers the basics how to upgrade the Ubuntu distribution and all packages coming from Ubuntu repositories and should work in most setups.
However there are always the more complex scenarios out there. One of such a scenario is one of our LXC environments. The LXC hosts are currently running on Ubuntu 14.04, most of the LXC containers were installed with Ubuntu 16.04.
Note: This is by default not possible due to the different init system of host and containers. However there's a workaround by removing systemd and manually installing upstart-sysv. See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for more details.
In the past few days I've been testing which would be the best order to upgrade the Ubuntu versions. Should I start with the containers first, followed by the host? Or should the host have the newest version first and then I upgrade the containers? What about intermediate steps? Will the containers still run on an old 14.04 when the host is on 18.04 and vice versa?
After a couple of upgrades in a TEST environment, I came to the following conclusion and compatibility matrix:
|LXC Host||LXC Guest||Compatibility||Notes|
||no*||System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to talk to init daemon.
|14.04||18.04||no||System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to talk to init daemon.
|18.04||14.04||no|| upstart init not started, applications could be started manually though.
e.g. /etc/init.d/ssh start
* again: See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for a workaround.
Luckliy there is 16.04 as an intermediate version available! It's the version in between being capable of talking correctly to the older upstart and newer systemd init systems!
The worst situations happened when the host staid at 14.04 and all the containers were upgraded but also when the host was upgraded to 18.04 but all the containers remained at 14.04.
Both situations caused the containers to now work correctly. Although they were started by LXC itself, the init system of the containers didn't work anymore due to an incompatibility between Systemd and the previous (Ubuntu-default) init system upstart.
When you need to upgrade an LXC environment (both host and containers need upgrades) and you run multiple versions across host and containers, these are the steps you should consider as your upgrade plan:
1) Upgrade the LXC hosts from 14.04 to 16.04. This will make sure that all containers from 14.04 to 18.04 will start correctly.
2) Upgrade all the containers from 14.04 to 16.04.
3) Upgrade the LXC hosts from 16.04 to 18.04. Containers with 16.04 run on SystemD by default, so they run fine on a host with 18.04.
4) Upgrade all the containers from 16.04 to 18.04.
This will help you reduce the downtimes of the whole LXC environment and therefore your applications.
Updated April 29th 2019:
Here's a handy way to find out under which Linux distribution and which version your containers are running. On the host run os-prober:
/dev/mapper/vglxc-dpm--heske01--t:Ubuntu 16.04.2 LTS (16.04):Ubuntu:linux
/dev/mapper/vglxc-inf--jira01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu1:linux
/dev/mapper/vglxc-inf--ldap01--t:Ubuntu 16.04.1 LTS (16.04):Ubuntu2:linux
/dev/mapper/vglxc-onl--mysql01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu3:linux
/dev/mapper/vglxc-okapi--app01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu4:linux
/dev/mapper/vglxc-onl--redis01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu5:linux
/dev/mapper/vglxc-onlkc--web01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu6:linux
/dev/mapper/vglxc-shop--web01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu7:linux
/dev/mapper/vglxc-st--cdb01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu8:linux
/dev/mapper/vglxc-ver--dks01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu9:linux
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder