HAProxy 1.5 init script broken in Ubuntu 14.04.3 LTS

Written by - 0 comments

Published on - Listed in Linux


Yesterday I got stuck on a weird problem where HAProxy suddenly started to behave stangely.

On a new server installation with Ubuntu 14.04.3 LTS and HAProxy 1.5 installed through trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1), HAProxy showed random HTTP 503 errors. But the backend servers themselves were all OK. Another side effect was that not even the stats page was shown correctly (also with 503).

When I checked with netstat if the admin/stats port was correct, it hit me:

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4221/haproxy   
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

It seems that HAProxy is started twice, listening twice on each port?! I checked the procs and indeed, there are two HAProxy processes running:

ps aux | grep haproxy
haproxy   4221  0.0  0.1  29868  2984 ?        Ss   12:06   0:03 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy   4653  0.0  0.1  29732  2980 ?        Ss   12:22   0:02 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

I remembered that I restarted HAProxy a few minutes before the problems started, so I checked if this works correctly:

service haproxy stop
 * Stopping haproxy haproxy                                                                                   [ OK ]

service haproxy status
haproxy not running.

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4221/haproxy   
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

Damn, the init script (/etc/init.d/haproxy) is lying! It didn't stop HAProxy at all. Only by killing the HAProxy processes I was able to terminate the processes and release the listening ports:

killall haproxy

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

 It seems that the init script was changed in the HAProxy 1.5 of trusty-backports, according to the changelog:

haproxy (1.5.4-1ubuntu2) utopic; urgency=medium

  * debian/haproxy.init: Backport of vivid stop routine,
    uses start-stop-daemon to reliable terminate all haproxy processes
    and return the proper exit code. (LP: #1462495)

 -- Jorge Niedbalski   Mon, 08 Jun 2015 15:52:13 -0500

As I can't rely on non-working init scripts, I switched back to 1.4 (the original trusty package). Luckly I don't need SSL in this HAProxy so I can live with the 1.4 version as well.

Ubuntu bug report can be found here: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1494141


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder