On a smaller Icinga 2 single-master installation, it was time to run an upgrade. This Icinga 2 setup was still running with an outdated 2.10 version; meanwhile 2.12 was released a few days ago. At the same time the Icingaweb 2.7 version can also be upgraded to a newer 2.8 release. Let's do this!
Usually this is an easy task, managed by simply running apt-get upgrade. And usually this always works. But not this time. What happened?
The command apt-show-versions is a nice helper to see which packages will be upgraded - and it also shows the currently installed version. This information is always helpful in case of a rollback.
root@icinga2:~# apt-show-versions -u | grep icinga
icinga2:amd64/icinga-stretch 2.10.4-1.stretch upgradeable to 2.12.0-1.stretch
icinga2-bin:amd64/icinga-stretch 2.10.4-1.stretch upgradeable to 2.12.0-1.stretch
icinga2-common:all/icinga-stretch 2.10.4-1.stretch upgradeable to 2.12.0-1.stretch
icinga2-doc:all/icinga-stretch 2.11.3-1.stretch upgradeable to 2.12.0-1.stretch
icinga2-ido-mysql:amd64/icinga-stretch 2.10.4-1.stretch upgradeable to 2.12.0-1.stretch
icingacli:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
icingaweb2:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
icingaweb2-common:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
icingaweb2-module-doc:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
icingaweb2-module-monitoring:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
php-icinga:all/icinga-stretch 2.7.3-1.stretch upgradeable to 2.8.1-1.stretch
During the apt-get upgrade command, the Icinga2 packages wanted to overwrite certain files in conf.d. In our setups we usually empty these files, to not interfere with other global apply rules. Hence the selection N (to not overwrite the existing configs) was chosen.
But right after, when the new packages are supposed to be installed, a lot of errors showed up:
root@icinga2:~# apt-get upgrade
[...]
Configuration file '/etc/icinga2/conf.d/hosts.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** hosts.conf (Y/I/N/O/D/Z) [default=N] ? N
Configuration file '/etc/icinga2/conf.d/notifications.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** notifications.conf (Y/I/N/O/D/Z) [default=N] ? N
Configuration file '/etc/icinga2/conf.d/services.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** services.conf (Y/I/N/O/D/Z) [default=N] ? N
Installing new version of config file /etc/icinga2/features-available/opentsdb.conf ...
Installing new version of config file /etc/icinga2/scripts/mail-host-notification.sh ...
Installing new version of config file /etc/icinga2/scripts/mail-service-notification.sh ...
Installing new version of config file /etc/icinga2/zones.d/README ...
Installing new version of config file /etc/logrotate.d/icinga2 ...
Job for icinga2.service failed because the control process exited with error code.
See "systemctl status icinga2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript icinga2, action "start" failed.
icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
00-restart.conf, limits.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2020-08-07 16:14:50 CEST; 5ms ago
Process: 14319 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=203/EXEC)
Process: 14314 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 14319 (code=exited, status=203/EXEC)
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Main process exited, code=exited, status=203/EXEC
Aug 07 16:14:50 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Service hold-off time over, scheduling restart.
Aug 07 16:14:50 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed to reset devices.list: Operation not permitted
Aug 07 16:14:50 icinga2 systemd[1]: Starting Icinga host/service/network monitoring system...
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Main process exited, code=exited, status=203/EXEC
Aug 07 16:14:50 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Service hold-off time over, scheduling restart.
Aug 07 16:14:50 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed to reset devices.list: Operation not permitted
Aug 07 16:14:50 icinga2 systemd[1]: Starting Icinga host/service/network monitoring system...
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Main process exited, code=exited, status=203/EXEC
Aug 07 16:14:50 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Service hold-off time over, scheduling restart.
Aug 07 16:14:50 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed to reset devices.list: Operation not permitted
Aug 07 16:14:50 icinga2 systemd[1]: Starting Icinga host/service/network monitoring system...
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Main process exited, code=exited, status=203/EXEC
Aug 07 16:14:50 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:50 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Service hold-off time over, scheduling restart.
Aug 07 16:14:51 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Failed to reset devices.list: Operation not permitted
Aug 07 16:14:51 icinga2 systemd[1]: Starting Icinga host/service/network monitoring system...
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Main process exited, code=exited, status=203/EXEC
Aug 07 16:14:51 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Service hold-off time over, scheduling restart.
Aug 07 16:14:51 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Start request repeated too quickly.
Aug 07 16:14:51 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:14:51 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
dpkg: error processing package icinga2-common (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
icinga2-common
E: Sub-process /usr/bin/dpkg returned an error code (1)
Interesting (to see it positively)!
The new deb packages were downloaded and can be found in /var/cache/apt/archives. Hence a manual installation maybe shows more information why the installation through apt resulted in an error?
root@icinga2:/var/cache/apt/archives# dpkg -i icinga2-common_2.12.0-1.stretch_all.deb
(Reading database ... 42266 files and directories currently installed.)
Preparing to unpack icinga2-common_2.12.0-1.stretch_all.deb ...
Unpacking icinga2-common (2.12.0-1.stretch) over (2.12.0-1.stretch) ...
Setting up icinga2-common (2.12.0-1.stretch) ...
Job for icinga2.service failed because the control process exited with error code.
See "systemctl status icinga2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript icinga2, action "start" failed.
icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
00-restart.conf, limits.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2020-08-07 16:26:23 CEST; 6ms ago
Process: 29766 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=203/EXEC)
Process: 29761 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 29766 (code=exited, status=203/EXEC)
Aug 07 16:26:23 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:26:23 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:26:23 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
dpkg: error processing package icinga2-common (--install):
subprocess installed post-installation script returned error exit status 1
Processing triggers for systemd (232-25+deb9u12) ...
Errors were encountered while processing:
icinga2-common
The package installation failed again. At this point a problem in the configuration within /etc/icinga2/ which would lead to a non-successful Icinga 2 restart was suspected. But a quick view in the mentioned journalctl, revealed something else (only the relevant part is shown here):
root@icinga2:~# journalctl -xe
[...]
-- Unit icinga2.service has begun starting up.
Aug 07 16:26:24 icinga2 systemd[29820]: icinga2.service: Failed at step EXEC spawning /usr/sbin/icinga2: No such file or directory
-- Subject: Process /usr/sbin/icinga2 could not be executed
-- Defined-By: systemd
[...]
No such file or directory? Of the Icinga 2 binary itself? No wonder a restart wouldn't work. Let's try to install the package containing the binary (icinga2-bin) manually:
root@icinga2:~# apt-get install icinga2-bin
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
icinga2-bin : Depends: libboost-context1.67.0 but it is not installable
Depends: libboost-coroutine1.67.0 but it is not installable
Depends: libboost-date-time1.67.0 but it is not installable
Depends: libboost-filesystem1.67.0 but it is not installable
Depends: libboost-program-options1.67.0 but it is not installable
Depends: libboost-regex1.67.0 (>= 1.67.0-10) but it is not installable
Depends: libboost-system1.67.0 but it is not installable
Depends: libboost-thread1.67.0 but it is not installable
E: Unable to correct problems, you have held broken packages.
Well, well, well. Finally an error message which makes sense! As it is a package dependency issue, it is actually surprising that apt did not show this at the beginning...
Time for Internet research! It did not take long, until a thread in the Icinga 2 community was found. It basically describes very similar problems although described a bit differently. The final quote however reveals the solution:
i was missing the debian strecht backports repository. After adding it and doing an apt-get update I was able to install everything.
Further research confirms, that the release notes of Icinga 2.11 mentioned "boost" especially. And the upgrading notes to 2.11 also contain special notes concerning Boost 1.66+:
After this information was digested, it is obvious that the needed boost packages only exist in stretch-backports repositories. Therefore they need to be enabled:
root@icinga2:~# echo "deb http://deb.debian.org/debian stretch-backports main" > /etc/apt/sources.list.d/backports.list
root@icinga2:~# apt-get update
And the icinga2-bin package can be installed:
root@icinga2:~# apt-get install icinga2-bin
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
dbconfig-common icinga2-doc libboost-atomic1.62.0 libboost-chrono1.62.0 libboost-date-time1.62.0 libboost-program-options1.62.0 libboost-regex1.62.0
libboost-system1.62.0 libboost-thread1.62.0 libyajl2 php-htmlpurifier
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
libboost-atomic1.67.0 libboost-chrono1.67.0 libboost-context1.67.0 libboost-coroutine1.67.0 libboost-date-time1.67.0 libboost-filesystem1.67.0
libboost-program-options1.67.0 libboost-regex1.67.0 libboost-system1.67.0 libboost-thread1.67.0
The following NEW packages will be installed:
icinga2-bin libboost-atomic1.67.0 libboost-chrono1.67.0 libboost-context1.67.0 libboost-coroutine1.67.0 libboost-date-time1.67.0 libboost-filesystem1.67.0
libboost-program-options1.67.0 libboost-regex1.67.0 libboost-system1.67.0 libboost-thread1.67.0
0 upgraded, 11 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
Need to get 6,076 kB of archives.
After this operation, 39.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Unpacking icinga2-bin (2.12.0-1.stretch) ...
Setting up icinga2-common (2.12.0-1.stretch) ...
Job for icinga2.service failed because the control process exited with error code.
See "systemctl status icinga2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript icinga2, action "start" failed.
icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
00-restart.conf, limits.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2020-08-07 16:31:11 CEST; 5ms ago
Process: 30443 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=1/FAILURE)
Process: 30438 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 30443 (code=exited, status=1/FAILURE)
Aug 07 16:31:11 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system.
Aug 07 16:31:11 icinga2 systemd[1]: icinga2.service: Unit entered failed state.
Aug 07 16:31:11 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
dpkg: error processing package icinga2-common (--configure):
subprocess installed post-installation script returned error exit status 1
Setting up libboost-atomic1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
dpkg: dependency problems prevent configuration of icinga2-bin:
icinga2-bin depends on icinga2-common (= 2.12.0-1.stretch); however:
Package icinga2-common is not configured yet.
dpkg: error processing package icinga2-bin (--configure):
dependency problems - leaving unconfigured
Setting up libboost-program-options1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-context1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-date-time1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up libboost-system1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-regex1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-filesystem1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-thread1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-chrono1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Setting up libboost-coroutine1.67.0:amd64 (1.67.0-13+deb10u1~bpo9+1) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Errors were encountered while processing:
icinga2-common
icinga2-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)
Oh, come on! What now?
Although an error is shown at the end, at least this time, the Icinga 2 binary is installed and available so a config check can be done:
root@icinga2:~# icinga2 daemon -C
[2020-08-07 16:31:58 +0200] information/cli: Icinga application loader (version: r2.12.0-1)
[2020-08-07 16:31:58 +0200] information/cli: Loading configuration file(s).
[2020-08-07 16:31:58 +0200] information/ConfigItem: Committing config item(s).
[2020-08-07 16:31:58 +0200] critical/config: Error: Could not load library 'libmysql_shim.so.2.12.0': libmysql_shim.so.2.12.0: cannot open shared object file: No such file or directory
[2020-08-07 16:31:58 +0200] critical/config: 1 error
[2020-08-07 16:31:58 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
An issue with libmysql sounds like a problem with the IDO package? Let's check the current status of all icinga packages:
root@icinga2:~# dpkg -l|grep icinga
ii icinga-l10n 1.0.0-1.stretch all l10n (short for Localization) provides all translations available for Icinga.
iU icinga2-bin 2.12.0-1.stretch amd64 host and network monitoring system - daemon
iF icinga2-common 2.12.0-1.stretch all host and network monitoring system - common files
ii icinga2-doc 2.12.0-1.stretch all host and network monitoring system - documentation
rc icinga2-ido-mysql 2.10.4-1.stretch amd64 host and network monitoring system - MySQL support
ii icingacli 2.8.1-1.stretch all simple CLI tool for Icingaweb2 and its modules
ii icingaweb2 2.8.1-1.stretch all simple and responsive web interface for Icinga
ii icingaweb2-common 2.8.1-1.stretch all simple and responsive web interface for Icinga - common files
ii icingaweb2-module-doc 2.8.1-1.stretch all simple and responsive web interface for Icinga - documentation module
ii icingaweb2-module-monitoring 2.8.1-1.stretch all simple and responsive web interface for Icinga - monitoring module
ii php-icinga 2.8.1-1.stretch all PHP library to communicate with and use Icinga
Indeed! The icinga2-ido-mysql package is in removed (rc) state. This explains the missing libmysql module!
A proper re-installation of this package should do it:
root@icinga2:~# apt-get install icinga2-ido-mysql
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
icinga2-doc libboost-atomic1.62.0 libboost-chrono1.62.0 libboost-date-time1.62.0 libboost-program-options1.62.0 libboost-regex1.62.0 libboost-system1.62.0
libboost-thread1.62.0 libyajl2 php-htmlpurifier
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
icinga2-ido-mysql
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
2 not fully installed or removed.
Need to get 120 kB of archives.
After this operation, 356 kB of additional disk space will be used.
Get:1 https://packages.icinga.com/debian icinga-stretch/main amd64 icinga2-ido-mysql amd64 2.12.0-1.stretch [120 kB]
Fetched 120 kB in 0s (240 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package icinga2-ido-mysql.
(Reading database ... 42328 files and directories currently installed.)
Preparing to unpack .../icinga2-ido-mysql_2.12.0-1.stretch_amd64.deb ...
Unpacking icinga2-ido-mysql (2.12.0-1.stretch) ...
Setting up icinga2-common (2.12.0-1.stretch) ...
Setting up icinga2-bin (2.12.0-1.stretch) ...
Setting up icinga2-ido-mysql (2.12.0-1.stretch) ...
dbconfig-common: writing config to /etc/dbconfig-common/icinga2-ido-mysql.conf
dbconfig-common: flushing administrative password
W: APT had planned for dpkg to do more than it reported back (8 vs 12).
Affected packages: icinga2-common:amd64
Not only did this install the missing IDO modules, it also correctly set up the icinga2-common and icinga2-bin packages which failed after the previous installations.
Icinga 2 is up again!
Lesson learned? Definitely. If you use Icinga 2 on Debian Stretch, make sure you enable the stretch-backports repositories or you will run into dependency problems once you want to install Icinga 2 2.11 or higher!
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Container Containers CouchDB DB DNS Database Databases Docker ELK ElasticSearch Elasticsearch Filebeat FreeBSD GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java KVM Kibana Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher SSL Security Shell SmartOS Solaris Surveillance SystemD TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder