apt-get update error: Could not get lock (SystemD and unattended APT)

Written by - 0 comments

Published on - last updated on April 24th 2021 - Listed in Linux Systemd


On an Ubuntu 16.04 Xenial machine I got the following error:

 # apt-get update
Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

A quick look at the processes showed that a daily run of apt-get update, managed by systemd, seemed hanging:

# ps auxf| grep -i apt
root     28044  0.0  0.0   4508  1700 ?        Ss   03:34   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily
root     28077  0.0  0.1  44628  7344 ?        S    03:34   0:03  \_ apt-get -qq -y update
_apt     28081  0.2  1.4 237124 57924 ?        S    03:34   1:44      \_ /usr/lib/apt/methods/https
_apt     28082  0.0  0.1  43212  5844 ?        S    03:34   0:00      \_ /usr/lib/apt/methods/http
_apt     28083  0.0  0.1  43276  5584 ?        S    03:34   0:00      \_ /usr/lib/apt/methods/http
_apt     28588  0.0  0.1  41036  5432 ?        S    03:36   0:00      \_ /usr/lib/apt/methods/gpgv

I tried to see what these processes are doing... Well for some reason the main process (PID 28077) seemed to have run in a timeout and was stuck in a loop:

# strace -s 1000 -f -p 28077
strace: Process 28077 attached
select(10, [5 6 7 9], [], NULL, {0, 76562}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}) = 0 (Timeout)
select(10, [5 6 7 9], [], NULL, {0, 500000}^Cstrace: Process 28077 detached

So what is causing this timeout? Let's check what exactly SystemD did at 03:34 this morning:

Jun 23 03:34:19 onl-lb04-s systemd[1]: Started Daily apt activities.
Jun 23 03:34:19 onl-lb04-s systemd[1]: apt-daily.timer: Adding 9h 3min 10.786515s random time.
Jun 23 03:34:19 onl-lb04-s systemd[1]: apt-daily.timer: Adding 6h 10min 44.777700s random time.
Jun 23 03:34:19 onl-lb04-s systemd[1]: Starting Daily apt activities...

WTF? SystemD seems to have added a total of 15 hours 13mins to the process? So this is why the process keeps hanging in a timeout and therefore locking apt?

There's quite some information on the Internet (if one knows what to look for) concerning this "problem". Obviously I'm not the only one stumbling on the automatic apt updates/upgrades which were enabled by default since Ubuntu 16.04 Xenial. Some good reads:

In general the apt folks agree that the current setup with adding random times to execute is not a good idea:

"We should think about this a bit more" - Julian Andres Klode (apt maintainer)

In the second bug 1686470 (which serves as general brainstorming and technical re-setup of the whole automatic apt update/upgrade) a definitive solution seems to be implemented. But it is yet to be released on Xenial:

Ubuntu Xenial Bug Automatic APT Unattended Updates

Until a definitive fix (re-setup of the automatic apt process) is released, there are the following workarounds:

  • Manually kill such apt processes which appear "hanging" (due to the random and sometimes huge added time)
  • Disable automatic updates/upgrades in /etc/apt/apt.conf.d/20auto-upgrades (by setting both values to 0)
  • If you want automatic apt updates and/or upgrades on your Xenial system, do so the legacy way using cron
  • Wait until bug 1686470 is fixed

Additional research points to apt proxy

Updated on June 26th 2017

After additional research it seems the timeout (seen in strace above) didn't come from the added random time, but rather from apt itself trying to establish a connection with a proxy. apt on this particular server was set up to use a http proxy:

# cat /etc/apt/apt.conf
Acquire::http::Proxy "http://myproxyserver.local:8080";

However apt-get update got stuck on a https repository:

# apt-get update
Ign:1 https://packagecloud.io/varnishcache/varnish5/ubuntu xenial InRelease
0% [Waiting for headers] [Waiting for headers]^C                        

Reason seems to be some problem in the firewall rules as I wasn't able to communicate with the proxy server, even with curl:

# export https_proxy=http://myproxyserver.local:8080
# curl https://www.claudiokuenzler.com/robots.txt
curl: (56) Proxy CONNECT aborted





Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder