HAProxy in Ubuntu 18.04 Docker image not starting: cannot bind UNIX socket

Written by - 4 comments

Published on - last updated on December 5th 2019 - Listed in Docker HAProxy Linux Kubernetes Containers

HAProxy usually works like a charm. In fact HAProxy is (probably) one of the most stable open source software capable of handling thousands of simultaneous connections. Using HAProxy for more than 7 years now, you can imagine my surprise when I was not able to run HAProxy in a Docker/application container.

Umm... what did you do?

Very simple stuff actually. Ubuntu 18.04 was used as base image. In Dockerfile a simple installation of the haproxy package, followed by the RUN of the haproxy process, was added:

FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler

# install packages
RUN apt-get update \
  && apt-get install -y -qq haproxy

CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]

Once the image was successfully built on Docker Hub, I wanted to deploy the container:

root@host:~# docker run napsty/haproxy:latest
Unable to find image 'napsty/haproxy:latest' locally
latest: Pulling from napsty/haproxy
7ddbc47eeb70: Pull complete
c1bbdc448b72: Pull complete
8c3b70e39044: Pull complete
45d437916d57: Pull complete
8e8788d95679: Pull complete
Digest: sha256:37d36ca920099c8d8288b1207fa7705bc5c169cc8747cf5afd599ab138c316e4
Status: Downloaded newer image for napsty/haproxy:latest
[ALERT] 337/164645 (1) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]

The error stating "cannot bind UNIX socket" is unclear without additional information.

Debugging inside the container

Let's get into this container to see what is actually happening:

root@harbor:~# docker run -it nzzonline/haproxy:devel bash

root@d8e6cf9fa2d5:/# stat /usr/sbin/haproxy
  File: /usr/sbin/haproxy
  Size: 1634568       Blocks: 3200       IO Block: 4096   regular file
Device: fd0eh/64782d    Inode: 682455      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-04 16:22:23.299177985 +0000
Modify: 2019-12-02 15:38:31.000000000 +0000
Change: 2019-12-04 16:23:27.219793835 +0000
 Birth: -

root@d8e6cf9fa2d5:/# stat /etc/haproxy/haproxy.cfg
  File: /etc/haproxy/haproxy.cfg
  Size: 1276          Blocks: 8          IO Block: 4096   regular file
Device: fd0eh/64782d    Inode: 401588      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-04 16:22:23.099176057 +0000
Modify: 2019-10-28 12:01:03.000000000 +0000
Change: 2019-12-04 16:23:27.019791909 +0000
 Birth: -

Both files /usr/sbin/haproxy and /etc/haproxy/haproxy.cfg do exist. But why would HAProxy not start? Launching the RUN command manually should show more:

root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
[ALERT] 337/162548 (13) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]

And indeed - HAProxy fails to start because it cannot bind to the Unix socket on /run/haproxy/admin.sock. Interestingly, this directory does not even exist:

root@d8e6cf9fa2d5:/# stat /run/haproxy
stat: cannot stat '/run/haproxy': No such file or directory

What if the directory is added manually?

root@d8e6cf9fa2d5:/# mkdir /run/haproxy
root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root@d8e6cf9fa2d5:/# ps auxf
root         1  0.1  0.0  18508  3516 pts/0    Ss   16:23   0:00 bash
haproxy     17  0.0  0.0  54284  1256 ?        Ss   16:27   0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root        18  0.0  0.0  34400  2884 pts/0    R+   16:27   0:00 ps auxf

Ha! HAProxy is running now!

How to solve this situation?

An obvious workaround is to add the creation of the directory already in Dockerfile - at the creation of the image:

FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler

# install packages
RUN apt-get update \
  && apt-get install -y -qq haproxy

# add missing socket path
RUN mkdir /run/haproxy

CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]

Once the image is built on Docker Hub, a docker pull will download the changed image and the container hopefully starts up this time:

root@host:~# docker pull napsty/haproxy:latest
devel: Pulling from napsty/haproxy
7ddbc47eeb70: Already exists
c1bbdc448b72: Already exists
8c3b70e39044: Already exists
45d437916d57: Already exists
af6d65a824f2: Already exists
cbf99dcc6c4f: Pull complete
b6229eac0350: Pull complete
a5d72fba4c6a: Pull complete
Digest: sha256:c03d7f770fa82ab77bcb2fa021b49300526367b47dc16d2ec9fe89c6be24febd
Status: Downloaded newer image for napsty/haproxy:latest

root@host:~# docker run napsty/haproxy:latest

No errors anymore!

Real solution?

That's the bigger challenge. "Many minds == many opinions"; some may think I'm completely wrong but in my opinion the haproxy package coming from the Ubuntu repositories should be responsible to create this path during the installation (postinst). I created Ubuntu bug #1855140, we'll see how this will be (eventually) solved.

One big question remains though: Why do installations of the haproxy package on "normal" Ubuntu 18.04 work but seem to fail every time when deployed in a Docker container? This can be double-checked in the deb source package.

Analysis of the haproxy source deb package

Note: This package analysis was done after the discussion in the comments (see comments below the article).

After a deb-src repository is added to /etc/apt/sources.list, the haproxy source package can be installed:

root@linux:~# grep src /etc/apt/sources.list
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic main restricted

root@linux:~# apt-get update

root@linux:~# apt-get source haproxy
Reading package lists... Done
NOTICE: 'haproxy' packaging is maintained in the 'Git' version control system at:
Please use:
git clone https://salsa.debian.org/haproxy-team/haproxy.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 2,122 kB of source archives.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (dsc) [2,280 B]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (tar) [2,055 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (diff) [65.2 kB]
Fetched 2,122 kB in 1s (2,280 kB/s)
dpkg-source: info: extracting haproxy in haproxy-1.8.8
dpkg-source: info: unpacking haproxy_1.8.8.orig.tar.gz
dpkg-source: info: unpacking haproxy_1.8.8-1.debian.tar.xz
dpkg-source: info: applying 0002-Use-dpkg-buildflags-to-build-halog.patch
dpkg-source: info: applying haproxy.service-start-after-syslog.patch
dpkg-source: info: applying haproxy.service-add-documentation.patch
dpkg-source: info: applying haproxy.service-use-environment-variables.patch
W: Download is performed unsandboxed as root as file 'haproxy_1.8.8-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

Inside the haproxy-1.8.8 folder the structure of the source package can be seen:

root@linux:~# cd haproxy-1.8.8/
root@linux:~/haproxy-1.8.8# ll
total 652
drwxr-xr-x 12 root root   4096 Dec  5 14:15 ./
drwx------  6 root root   4096 Dec  5 14:15 ../
-rw-r--r--  1 root root 490605 Apr 19  2018 CHANGELOG
drwxr-xr-x 18 root root   4096 Apr 19  2018 contrib/
-rw-r--r--  1 root root  41508 Apr 19  2018 CONTRIBUTING
drwxr-xr-x  6 root root   4096 Apr 19  2018 debian/
drwxr-xr-x  5 root root   4096 Apr 19  2018 doc/
drwxr-xr-x  2 root root   4096 Apr 19  2018 ebtree/
drwxr-xr-x  3 root root   4096 Apr 19  2018 examples/
-rw-r--r--  1 root root    788 Apr 19  2018 .gitignore
drwxr-xr-x  6 root root   4096 Apr 19  2018 include/
-rw-r--r--  1 root root   2029 Apr 19  2018 LICENSE
-rw-r--r--  1 root root   3089 Apr 19  2018 MAINTAINERS
-rw-r--r--  1 root root  36682 Apr 19  2018 Makefile
drwxr-xr-x  6 root root   4096 Dec  5 14:15 .pc/
-rw-r--r--  1 root root  15356 Apr 19  2018 README
-rw-r--r--  1 root root   2713 Apr 19  2018 ROADMAP
drwxr-xr-x  2 root root   4096 Apr 19  2018 scripts/
drwxr-xr-x  2 root root   4096 Apr 19  2018 src/
-rw-r--r--  1 root root     14 Apr 19  2018 SUBVERS
drwxr-xr-x  2 root root   4096 Apr 19  2018 tests/
-rw-r--r--  1 root root     24 Apr 19  2018 VERDATE
-rw-r--r--  1 root root      6 Apr 19  2018 VERSION

Building deb packages is not trivial, but if one has gotten into it (for whatever reason) there are a couple of important files to look at. The first file worth to check out is debian/haproxy.dirs:

root@linux:~/haproxy-1.8.8# cat debian/haproxy.dirs

A couple of paths are mentioned here, but the (now famous) /run/haproxy does not appear in that list. However it appears in haproxy.tmpfile:

root@linux:~/haproxy-1.8.8# cat debian/haproxy.tmpfile
d /run/haproxy 2775 haproxy haproxy -

What exactly is this tmpfile and what is its purpose? The last time I personally was involved in more deb package building was during the Debian Wheezy (7) release. This is a pre-systemd Debian release. Might tmpfile be related to systemd? Interestingly this exact information can be found in the package's changelog:

root@linux:~/haproxy-1.8.8# grep -rni tmpfile *
debian/changelog:980:      tmpfiles.d config for systemd.

By taking a closer look at the documentation of dh_systemd_enable it's clearly written:

    If this exists, it is installed into usr/lib/tmpfiles.d/package.conf in the package build directory. (The tmpfiles.d mechanism is currently only used by systemd.)

There's no mistake: "currently only used by systemd". And as it is common knowledge: Systemd does not run in a Docker container. 

Further information can also be found on the Ubuntu manpages for tmpfiles.d:

systemd-tmpfiles uses the configuration files from the above directories to describe the creation, cleaning and removal of volatile and temporary files and directories which usually reside in directories such as /run or /tmp.

This sums it up pretty clearly: The /run/haproxy directory is only created by using systemd-tmpfiles, which does not exist in a Docker container. As the haproxy is installed during the Docker image building (which happens inside the Docker container), there is no Systemd available. Hence the directory /run/haproxy is never created.

Add a comment

Show form to leave a comment

Comments (newest first)

Hadret from wrote on Dec 5th, 2019:

Ah, nice catch! So the package maintainers never foresaw HAProxy being in environment without systemd available. Nice, thanks for sharing!

ck from Switzerland wrote on Dec 5th, 2019:

Hadret, I analysed the source package and it is indeed a problem of the package. It relies on Systemd to create the /run/haproxy directory. I updated the article to show my findings.

ck from Switzerland wrote on Dec 5th, 2019:

Hadret, thanks for checking! I am running douzens of HAProxy installations, some of them on Ubuntu 18.04 (LXC or VM's, too) and never had that problem before. So yes, I agree with you - it must have something to do between the package and the fact that this happens in a Docker container. Maybe the haproxy.tmpfile you mentioned is only triggered by systemd? Systemd does not exist within a Docker container and maybe this causes the problem? I will have to look at the deb src package I guess.

Hadret from wrote on Dec 5th, 2019:

I dug a bit out of sheer curiosity. It's not a problem with the package -- it's a problem with Docker container. Not sure where is it exactly, but the package does handle /run/haproxy folder creation. In the debian folder of the package you can find haproxy.tmpfile that is responsible for creating it:

d /run/haproxy 2775 haproxy haproxy -

What's more, if you install haproxy package on a VM (default/fresh/clean OS), the directory is being created successfully:

root@haproxy:~# ls -ld /run/haproxy
drwxrwsr-x 2 haproxy haproxy 60 Dec 5 13:26 /run/haproxy

It even matches the setgid (2755) setting. My bet is on Docker not handling that well on the package installation. For whatever reason -- I was able to create the directory and give it ownership and appropriate permissions inside the container.