RabbitMQ queue monitoring and fixing check_rabbitmq_queue monitoring plugin

Written by - 0 comments

Published on - Listed in Monitoring Coding Perl Linux


To monitor RabbitMQ queues, there are a couple of ways (scripts and monitoring plugins) but they all (usually) check the queues using the RabbitMQ API.

Viewing the queues from the API

The following example uses curl to access the RabbitMQ API and view all queues:

ck@rabbitmq:~$ curl http://localhost:15672/api/queues -u monitoring:secret
[{"arguments":{"x-dead-letter-exchange":"rabbitmq-prod-new-dlx"},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.01809491112502785,"avg_ack_ingress_rate":0.01809491112502785,"avg_egress_rate":0.01809491112502785,"avg_ingress_rate":0.01809491112502785,"delta":["delta",0,0,0,0],"len":0,"mode":"default","next_seq_id":4403,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_capacity":1.0 [...]

The response is a (pretty large) JSON output, containing all queues including their statistics.

From that JSON output the different queues can be listed using the "name" key. Using the jshon parser:

ck@rabbitmq:~$ curl -s http://localhost:15672/api/queues -u monitoring:secret | jshon -a -e name
"billrun.completion"
"billrun.worker"
"invoice.download"
"invoice.dunning"
"invoice.post"
"invoice.processor"
"invoice.worker"

Or if you prefer the jq parser:

ck@rabbitmq:~$ curl -s http://localhost:15672/api/queues -u monitoring:secret | jq -r ".[].name"
"billrun.completion"
"billrun.worker"
"invoice.download"
"invoice.dunning"
"invoice.post"
"invoice.processor"
"invoice.worker"

As root user you can also use the rabbitmqctl command to list the queues:

root@rabbitmq:~# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name    messages
invoice.processor    0
invoice.dunning    0
billrun.completion    0
invoice.download    0
invoice.worker    0
billrun.worker    0

Queues monitoring with check_rabbitmq_queue.pl

There are a couple of RabbitMQ scripts or monitoring plugins available, but all of them seem to be dated and sadly haven't received updates in years. One of the plugins which still works with a recent RabbitMQ version (tested with 3.9) is the Perl script check_rabbitmq_queue. This script is part of a bundle of Perl scripts (nagios-plugins-rabbitmq), doing all kinds of checks on a RabbitMQ server.

ck@rabbitmq:~$ /usr/lib/nagios/plugins/check_rabbitmq_queue.pl -H localhost
RABBITMQ_QUEUE OK - All queues under the thresholds | messages=273.5294;; messages_ready=273.5294;; messages_unacknowledged=0.0000;; consumers=0.8824;;

The %2F issue

Under certain circumstances, the plugin might run into a problem, that an Access Refused error shows up:

ck@rabbitmq:~$ /usr/lib/nagios/plugins/check_rabbitmq_queue.pl -H localhost --port 15672 --username monitoring --password secret
RABBITMQ_QUEUE CRITICAL - Access Refused : http://localhost:15672/api/queues/%2F

The issue here is that the plugin tries to handle (potentially) multiple virtual hosts, the default vhost being "/". But inside the plugin code, the vhost name is going through a Perl function called uri_escape.

my $vhost=uri_escape($p->opts->vhost);

This function translates characters into a URL-compatible format. A space in the URL (" ") would become %20 for example. But in this case %2F is the code for the forward slash ("/"), which is an important part of the URL itself and should not be translated.

This situation can be improved and fixed by only doing uri_escape on vhost names not matching "/":

my $vhost='/';
$vhost.=uri_escape($p->opts->vhost) if $p->opts->vhost ne "/";

And later in the code the URL to check the queue(s) is adjusted to the new default $vhost value:

my $url = "";
if ($queue eq "all"){
    $url = sprintf("http%s://%s:%d/api/queues%s", ($p->opts->ssl ? "s" : ""), $hostname, $port, $vhost);
} else{
    $url = sprintf("http%s://%s:%d/api/queues%s/%s", ($p->opts->ssl ? "s" : ""), $hostname, $port, $vhost, $queue);
}

After adding this fix, the queues can be monitored again:

ck@rabbitmq:~$ /usr/lib/nagios/plugins/check_rabbitmq_queue.pl -H localhost --port 15672 --username monitoring --password secret
RABBITMQ_QUEUE OK - All queues under the thresholds | messages=0.8636;; messages_ready=0.8636;; messages_unacknowledged=0.0000;; consumers=0.5682;;

This code change was submitted as pull request to the upstream project. But given that it has been 6 years since the last commit, I doubt this will be merged.

As an alternative, you can download the check_rabbitmq_queue plugin with the %2F fix from the pull request branch.


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder