How to fix broken apt dist-upgrade when /var is out of disk space

Written by - 0 comments

Published on - Listed in Linux


When running apt-get dist-upgrade (or the newer apt upgrade), the command downloads the update packages (.deb) into /var/cache/apt/archives. Once downloaded, apt runs through the package installers. Depending on each package, there might be additional pre- or post-install steps to be done.

One of that is the upgrade of a linux-image (Kernel) or a Kernel related package (e.g. linux-firmware). It launches update-initramfs in the background, updating the initramfs for the new Kernel version. This process uses /var/tmp to build the new initramfs.

Diskspace (/var) full during upgrade

If you happen to be in the unlucky situation that your /var partition (or root partition if you don't have a separate partition for /var) is filled before the apt process is completed, you run into a dangerous situation:

root@ubuntu:~# apt-get update && apt-get dist-upgrade
[...]
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-97-generic
depmod: ERROR: Could not create index 'modules.dep'. Output is truncated: No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f/etc/ld.so.conf': No space left on device
cp: cannot create directory '/var/tmp/mkinitramfs_ELDs7f/etc/ld.so.conf.d': No space left on device
/sbin/ldconfig.real: Writing of cache data failed: No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f//usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so': No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f//usr/lib/x86_64-linux-gnu/libresolv-2.31.so': No space left on device
update-initramfs: Generating /boot/initrd.img-5.4.0-96-generic
cp: error writing '/var/tmp/mkinitramfs_cAbtxh//usr/lib/x86_64-linux-gnu/libkmod.so.2.3.5': No space left on device
E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-96-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
[...]
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-125-generic
Processing triggers for systemd (245.4-4ubuntu3.17) ...
Processing triggers for dbus (1.12.16-2ubuntu2.2) ...
Errors were encountered while processing:
 linux-firmware
 linux-image-generic
 linux-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)

A reboot of the server in this situation means trouble: The Ubuntu server will most likely not be able to start anymore.

Make or increase available space

Luckily there's no need to panic, the errors above can be fixed. But first you need to find out why /var is full. This could be because of a lot of logs in /var/log, it could be huge temp files in /var/tmp, it could be anything. In this particular situation the cause was a Logstash process logging a lot of errors into /var/log/logstash, filling up the /var partition very quickly. Deleting the Logstash (plain) logs gave enough available disk space back.

Another alternative is of course to increase the /var partition. That's easy when you are using Logical Volumes - it might be trickier if you have /var on your root partition.

Once you got some space back on /var, you can continue with fixing apt.

Fixing the previously broken upgrade

apt, respectively dpkg, know which packages were broken during the upgrade and will attempt to launch the failed process (e.g. postinstall scripts) again.

This can be initiated with dpkg --configure -a:

root@ubuntu:~# dpkg --configure -a
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-125-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-97-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-96-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-94-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-42-generic
Setting up linux-image-generic (5.4.0.125.126) ...
Setting up linux-generic (5.4.0.125.126) ...

The initramfs was successfully built (this time). With another apt-get dist-upgrade you can verify that there is indeed no problem left:

root@ubuntu:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-5.4.0-94 linux-headers-5.4.0-94-generic linux-headers-5.4.0-96 linux-headers-5.4.0-96-generic linux-headers-5.4.0-97 linux-headers-5.4.0-97-generic linux-image-5.4.0-94-generic linux-image-5.4.0-96-generic
  linux-image-5.4.0-97-generic linux-modules-5.4.0-94-generic linux-modules-5.4.0-96-generic linux-modules-5.4.0-97-generic linux-modules-extra-5.4.0-94-generic linux-modules-extra-5.4.0-96-generic linux-modules-extra-5.4.0-97-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder