HAProxy 1.5 init script broken in Ubuntu 14.04.3 LTS

Written by Claudio Kuenzler - 0 comments

Published on September 10th 2015 - Listed in Linux

Yesterday I got stuck on a weird problem where HAProxy suddenly started to behave stangely.

On a new server installation with Ubuntu 14.04.3 LTS and HAProxy 1.5 installed through trusty-backports (1.5.4-1ubuntu2.1~ubuntu14.04.1), HAProxy showed random HTTP 503 errors. But the backend servers themselves were all OK. Another side effect was that not even the stats page was shown correctly (also with 503).

When I checked with netstat if the admin/stats port was correct, it hit me:

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4221/haproxy   
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

It seems that HAProxy is started twice, listening twice on each port?! I checked the procs and indeed, there are two HAProxy processes running:

ps aux | grep haproxy
haproxy   4221  0.0  0.1  29868  2984 ?        Ss   12:06   0:03 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
haproxy   4653  0.0  0.1  29732  2980 ?        Ss   12:22   0:02 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

I remembered that I restarted HAProxy a few minutes before the problems started, so I checked if this works correctly:

service haproxy stop
 * Stopping haproxy haproxy                                                                                   [ OK ]

service haproxy status
haproxy not running.

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      4221/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4653/haproxy   
tcp        0      0 0.0.0.0:8099            0.0.0.0:*               LISTEN      4221/haproxy   
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

Damn, the init script (/etc/init.d/haproxy) is lying! It didn't stop HAProxy at all. Only by killing the HAProxy processes I was able to terminate the processes and release the listening ports:

killall haproxy

netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      956/nginx      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      855/sshd       
tcp6       0      0 :::22                   :::*                    LISTEN      855/sshd       

 It seems that the init script was changed in the HAProxy 1.5 of trusty-backports, according to the changelog:

haproxy (1.5.4-1ubuntu2) utopic; urgency=medium

  * debian/haproxy.init: Backport of vivid stop routine,
    uses start-stop-daemon to reliable terminate all haproxy processes
    and return the proper exit code. (LP: #1462495)

 -- Jorge Niedbalski   Mon, 08 Jun 2015 15:52:13 -0500

As I can't rely on non-working init scripts, I switched back to 1.4 (the original trusty package). Luckly I don't need SSL in this HAProxy so I can live with the 1.4 version as well.

Ubuntu bug report can be found here: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1494141

Add a comment

Comments (newest first)

No comments yet.