Icinga2 graphing with InfluxDB and Grafana

Written by - 1 comments

Published on - last updated on March 3rd 2021 - Listed in Icinga Monitoring DB

As you can imagine from my previous post (icinga2-classicui shows Whoops error on new Icinga 2.8 install) I'm currently setting up a renewed monitoring architecture running on Ubuntu 16.04 Xenial.

To my big surprise PNP4Nagios doesn't run anymore on Xenial because Xenial ships with PHP 7.0 but PNP4Nagios is now running with PHP 7.0 (see this issue on github). There are surely ways around it like installing special PHP repositories to install older PHP versions, but I decided to test another graphing method: Grafana as user interface (grapher) and InfluxDB as database. 

Note in advance: Getting your graphs shown in Grafana certainly requires much more work than with PNP4Nagios, especially if you heavily rely on nrpe checks like me.

Part I: Preparation of InfluxDB

Let's install the InfluxDB right from the Ubuntu repos:

root@inf-mon02-t:~# apt-get install influxdb influxdb-client

After the installation, influxdb's daemon (influxd) is launched automatically and is listening:

root@inf-mon02-t:~# netstat -lntup | grep influx
tcp        0      0*               LISTEN      25027/influxd
tcp6       0      0 :::8083                 :::*                    LISTEN      25027/influxd
tcp6       0      0 :::8086                 :::*                    LISTEN      25027/influxd
tcp6       0      0 :::8088                 :::*                    LISTEN      25027/influxd

Data will be stored in /var/lib/influxdb:

root@inf-mon02-t:~# ll /var/lib/influxdb/
total 16
drwxr-xr-x 3 influxdb influxdb 4096 Dec  8 12:11 data
drwx------ 2 influxdb influxdb 4096 Dec  8 12:11 hh
drwxr-xr-x 3 influxdb influxdb 4096 Dec  8 12:11 meta
drwx------ 3 influxdb influxdb 4096 Dec  8 12:11 wal

InfluxDB can now be accessed using the InfluxDB client (influx):

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
name: databases

> quit

Let's prepare a database for Icinga2 and create a database user for it:

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
name: databases

> CREATE USER icinga2 WITH PASSWORD 'mysupersecretpassword'
> GRANT ALL ON icinga2 TO icinga2
database        privilege
icinga2         ALL PRIVILEGES

> quit

Part II: Tell Icinga2 to use InfluxDB for performance data

The following Icinga2 features need to be enabled:

root@inf-mon02-t:~# icinga2 feature enable influxdb
root@inf-mon02-t:~# icinga2 feature enable perfdata

This requires a restart of Icinga2:

root@inf-mon02-t:~# service icinga2 restart

Verify, that the features are really enabled:

root@inf-mon02-t:~# icinga2 feature list
Disabled features: debuglog elasticsearch gelf graphite ido-mysql livestatus opentsdb syslog
Enabled features: api checker command compatlog influxdb mainlog notification perfdata statusdata

In theory that should already be enough. From the official Icinga2 documentation:

"By default the InfluxdbWriter feature expects the InfluxDB daemon to listen at on port 8086."

Important to add a note here: The default database Icinga2 wants to write into is called "icinga2".

Now let's check if InfluxDB is really receiving data from Icinga2's.

Note: InfluxDB doesn't use the term "table" as classical SQL databases. This is called a "measurement" in InfluxDB terminology and is "comparable" to a table. It's my first day with InfluxDB so this is very likely not technically correct, but helpful to understand the query syntax.

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
> USE icinga2
Using database icinga2
name: measurements

> SELECT * FROM ping4
name: ping4
time                    hostname        metric  service value
1512732076000000000     inf-jira01-t    pl      ping4   0
1512732076000000000     inf-jira01-t    rta     ping4   0.003722
1512732076000000000     inf-mon02-t     pl      ping4   0
1512732076000000000     inf-mon02-t     rta     ping4   4.1e-05
1512732138000000000     inf-mon02-t     rta     ping4   4.6e-05
1512732138000000000     inf-jira01-t    rta     ping4   0.00366
1512732138000000000     inf-mon02-t     pl      ping4   0
1512732138000000000     inf-jira01-t    pl      ping4   0
1512732200000000000     inf-mon02-t     pl      ping4   0
1512732200000000000     inf-mon02-t     rta     ping4   5.5e-05
1512732200000000000     inf-jira01-t    rta     ping4   0.003685
1512732200000000000     inf-jira01-t    pl      ping4   0
1512732262000000000     inf-jira01-t    rta     ping4   0.003784
1512732262000000000     inf-mon02-t     pl      ping4   0

> quit

Yes, there's data in there!

But all this collected data is only helpful if we can create graphs from it. Grafana is able to connect to an InfluxDB and create graphs from the data, so let's do this!

Part III: The fight to connect Grafana to InfluxDB

The installation of Grafana is very simple as Xenial already has Grafana packages in their repos.

root@inf-mon02-t:~# apt-get install grafana grafana-data

Grafana is listening on port tcp/3000:

root@inf-mon02-t:~# netstat -lntup|grep grafana
tcp6       0      0 :::3000                 :::*                    LISTEN      27579/grafana

This is the listener port of the Grafana-internal webserver. This is how Grafana looks in the browser:

Grafana Login

The default user and password is admin/admin and the default (and yet empty) dashboard is shown:

Grafana empty dashboard

To add the InfluxDB as a data source, click on Data Sources -> Add new:

Grafana add data source

Set a name for this new data source. I called it "InfluxDB-Icinga2".
On Ubuntu 16.04 Xenial an InfluxDB 0.9.x is installed, so set "Type" to "InfluxDB 0.9.x". 
In "Http settings" set the access to the InfluxDB's API port, which is "http://localhost:8086" and has a "direct" access. 
Grafana requires given user and password to connect to InfluxDB. Use the icinga2 user which was created above.

Grafana add new data source

Once the data source was addded, I clicked on "Test Connection" and got the following error in Chrome:

Unknown error
InfluxDB Error: Cannot read property 'message' of null

In Firefox I got a slightly different error:

Unknown error
InfluxDB Error: err.data is null

Grafana data source error err data is null

I wasn't able to find a solution to this, but it led me to believe that a lot of bugs have been fixed since the Grafana version for Xenial (2.6.0) was released.
The current version is 4.6.2 so I decided to install this one. But first, Grafana (from the Ubuntu repos) needs to be uninstalled again:

root@inf-mon02-t:~# apt-get remove grafana grafana-data
root@inf-mon02-t:~# apt-get purge grafana grafana-data
root@inf-mon02-t:~# rm -rf /usr/share/grafana

I downloaded the current release's Debian package and installed it:

root@inf-mon02-t:~# wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.6.2_amd64.deb
root@inf-mon02-t:~# sudo dpkg -i grafana_4.6.2_amd64.deb

At the end of the installation of the deb package, the following notes show up:

### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server

Hmm... could have been placed into the deb package, too. 
So let's do this:

root@inf-mon02-t:~# sudo /bin/systemctl daemon-reload
root@inf-mon02-t:~# sudo /bin/systemctl enable grafana-server
Synchronizing state of grafana-server.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
root@inf-mon02-t:~# sudo /bin/systemctl start grafana-server

And yes, Grafana is started:

root@inf-mon02-t:~# netstat -lntup| grep grafana
tcp6       0      0 :::3000                 :::*                    LISTEN      29922/grafana-serve

Back to the login with the browser. Wow - the interface has really changed a lot:

Grafana 4.6.2 dashboard


The new data source can now be added by clicking on the Grafana icon (top left) and then on "Data Sources".

Grafana menu
Grafana data source

Oh.. looks like the previously added InfluxDB is still here (because I didn't delete Grafana's own database in /var/lib/grafana before):

I selected this data source and clicked on "Save & Test". Argh. Another error:

Network Error: undefined(undefined)

Grafana data source error network error

Jeez, what now!? Eventually I came across this issue on Github and there user eighteen14k mentioned:

Oh this is just silly.

"Access: direct" - browser attempts to verify "data source"
"Access: proxy" - grafana verifies "data source".

That's the inverse of what I would expect.

What?! Seriously? Well I tried it and set the connection to "proxy" instead of "direct" and boom:

Grafana data source working

OMG! Seriously, this is, as eighteen14k said it, just silly!

Now that the connection from Grafana to InfluxDB is finally working, a new Dashboard needs to be added in Grafana.

Part IV: Create the Icinga2 dashboard

Luckily a dashboard is already prepared and ready to use from the Icinga2 team and can be downloaded here: https://grafana.com/dashboards/381.
The json config from this dashboard (ID 381) can be downloaded here: https://grafana.com/api/dashboards/381/revisions/1/download.
The content can then be pasted directly into Grafana, when importing a dashboard:

Grafana import dashboard
Grafana import dashboard
Grafana import dashboard

And fiiiinally, the graphs are showing up:

Grafana Icinga2 Dashboard

Note: For NRPE checks you will have to adapt the graphs because these performance data are stored in the "nrpe" measurements table.

Add a comment

Show form to leave a comment

Comments (newest first)

Cedric from Paris wrote on Feb 7th, 2024:

Merci Beaucoup pour votre tutoriel, ça ma beaucoup aider, j'ai suivi la procédure et ça bien marche.
Concernant grafana, je me suis basé sur la nouvelle version

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   

Update cookies preferences