Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Nagios/Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Very slow VMware Perl SDK with newer Perl libwww-perl version (solved)
Tuesday - Jun 21st 2016 - by - (0 comments)

For a new VMware and monitoring environment, I wanted to add the ESXi hosts to the monitoring. In the past years I successfully used the plugin "check_esx" from OP5 for this purpose. Meanwhile I had to learn that the plugin was renamed to "check_vmware_api" and that this plugin in turn had been forked at least twice.

Now there are the following plugins out there which do basically the same but slightly differ in development and code:

They all have one thing in common: They rely on the VMware Perl SDK in the background. And here starts the problem.

With all three plugins I experienced timeouts. When I showed a little bit more patience, I finally saw that it takes more than 3 minutes (!!!), to get some results:

$ time /tmp/check_vmware_esx.pl -H esx001 -u root -p secret --select=runtime
SOAP request error - possibly a protocol issue:


[...]

real    3m46.739s
user    0m0.199s
sys     0m0.022s

The most interesting part is the first line of the output: SOAP request error - possibly a protocol issue. After some research I stumbled across the following two pages:

The first site is a discussion in the VMware community forums where the problem is identified in the SDK itself. Seems the SDK was programmed with an old version of libwww-perl. On newer systems (and my monitoring server runs Ubuntu 14.04) this causes problems.

On the second site there is actually a nice workaround presented where the author, Bob Clary, manually installed an older libwww-perl version (5.837) into a separate location and exported the path in shell scripts as a wrapper around the Perl SDK.

While the workaround seems like the way to go, I didn't want to build wrappers around the SDK (and therefore maintain this after every potential update). If I work around it, then right in the file which causes problems. Which is, in my case, /usr/share/perl/5.18/VMware/VICommon.pm.

Before I did anything, I wanted to verify once again and directly from the Perl SDK itself, that the problem is not from one of the plugins but from the SDK:

time /usr/lib/vmware-vcli/apps/vm/vminfo.pl --server esx001 --username root --password secret
SOAP request error - possibly a protocol issue:
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
[...]

real    3m48.655s
user    0m0.184s
sys     0m0.028s

Here again it took more than 3 minutes until a result came back.

Now it's time to install the older libwww-perl version, as seen on the article from Bob Clary:

cd src/
mkdir libwww-perl
cd libwww-perl
wget https://github.com/libwww-perl/libwww-perl/archive/libwww-perl/5.837.tar.gz
tar -xzf 5.837.tar.gz
cd libwww-perl-libwww-perl-5.837/
perl Makefile.PL INSTALL_BASE=/opt/check_vmware_esx
make
make install

This installs the following Perl folder in /opt/check_vmware_esx/lib/perl5:

ll /opt/check_vmware_esx/lib/perl5/
total 96
drwxr-xr-x 2 root root  4096 Jun 21 10:14 Bundle
drwxr-xr-x 2 root root  4096 Jun 21 10:14 File
drwxr-xr-x 2 root root  4096 Jun 21 10:14 HTML
drwxr-xr-x 5 root root  4096 Jun 21 10:14 HTTP
drwxr-xr-x 5 root root  4096 Jun 21 10:14 LWP
-r--r--r-- 1 root root  9137 Sep 20  2010 lwpcook.pod
-r--r--r-- 1 root root 21343 Sep 20  2010 LWP.pm
-r--r--r-- 1 root root 25528 Sep 20  2010 lwptut.pod
drwxr-xr-x 3 root root  4096 Jun 21 10:14 Net
drwxr-xr-x 3 root root  4096 Jun 21 10:14 WWW
drwxr-xr-x 3 root root  4096 Jun 21 10:14 x86_64-linux-gnu-thread-multi

Now to the modification of /usr/share/perl/5.18/VMware/VICommon.pm (which was installed by the VMware Perl SDK installer). The begin of the file looks like this:

#
# Copyright 2006 VMware, Inc.  All rights reserved.
#

use 5.006001;
use strict;
use warnings;

use Carp qw(confess croak);
use XML::LibXML;
use LWP::UserAgent;
use LWP::ConnCache;
use HTTP::Request;
use HTTP::Headers;
use HTTP::Response;
use HTTP::Cookies;
use Data::Dumper;
[...]

To use the manually installed libwww-perl module (and not the default one installed from the Ubuntu repository), I simply added "use lib" right before the Perl version is defined:

#
# Copyright 2006 VMware, Inc.  All rights reserved.
#

use lib "/opt/check_vmware_esx/lib/perl5";

use 5.006001;
use strict;
use warnings;

use Carp qw(confess croak);
use XML::LibXML;
use LWP::UserAgent;
use LWP::ConnCache;
use HTTP::Request;
use HTTP::Headers;
use HTTP::Response;
use HTTP::Cookies;
use Data::Dumper;

Let's try the vminfo.pl command from above again:

time /usr/lib/vmware-vcli/apps/vm/vminfo.pl --server esx001 --username root --password secret

Information of Virtual Machine vm001

Name:            vm001
No. of CPU(s):           1
Memory Size:             4096
Virtual Disks:           1
Template:                0
vmPathName:              [esxz108_T2] vm001/vm001.vmx
Guest OS:                Microsoft Windows Server 2008 R2 (64-bit)
guestId:                 windows7Server64Guest
Host name:               vm001.local
IP Address:              10.10.20.157
VMware Tools:            VMware Tools is running and the version is current
Cpu usage:               39 MHz
Host memory usage:               4141 MB
Guest memory usage:              327 MB
Overall Status:          The entity is OK

Information of Virtual Machine vm002

[...]

real    0m1.498s
user    0m0.834s
sys     0m0.035s

Wow, what a change! All VM's running on esx001 were listed with detailed information. And all that in 1.498 seconds!

The Perl SDK seems fixed. Does that also apply on the plugins? I tried it with check_vmware_esx:

time /tmp/check_vmware_esx.pl -H esx001 -u root -p secret -S cpu --nosession
OK: CPU wait=424610.00 ms - CPU ready=97.00 ms - CPU usage=7.10%|'cpu_wait'=424610.00ms;;;; 'cpu_ready'=97.00ms;;;; 'cpu_usage'=7.10%;;;;

real    0m0.687s
user    0m0.351s
sys     0m0.029s

Mission accomplished!

 

Migrated web-application from PHP 5.4 to PHP 7: 3x faster!
Friday - Jun 10th 2016 - by - (0 comments)

This week a web-application was migrated from PHP 5.4 to a newer server with PHP 7. It required a few modifications of the applications, but it finally ran correctly.

The most interesting part: The application running under PHP 7 is now 3x faster than before! Take a look at the following graph which comes from our monitoring and measures the response time of the web-application:

PHP 5.4 vs. PHP 7 response times

Without a doubt you saw he "drop" of response time on Wednesday afternoon.

Here some additional information:
"Old" server was running on a CentOS 6 VM with Nginx and PHP-FPM 5.4. "New" server is a Ubuntu 16.04 LXC container with Nginx and PHP-FPM 7.

 

Create a fake mail server to test mail functions in applications
Tuesday - Jun 7th 2016 - by - (0 comments)

Developing and testing applications sometimes require you to go the "non-standard" way. A good example is when you have a web-application which has a mail function. The mails must correctly be sent but remember, this is just a test of the application and you don't want your users to get spammed with test mails. 

A fake/dummy mail server comes in handy. A developer suggested MailCatcher, which looked good to me except that it's written in ruby and depends on gems which I didn't want to put on this system. At least not if I find something easier...

Then I came across Dummy-SMTP, a simple listener written in python (so no additional software to be installed on my Ubuntu test servers) just saving all the received mails as text files in a folder.
By default, Dummy-SMTP runs on port 25 which requires root privileges to start with and it also must be started from within the correct folder. I made some modifications for the following purposes:

  • Launch the listener as non-root user
  • Launch the listener from anywhere with the absolute path

In my case, I defined the listener port to be 1025 and defined an absolute path: /srv/Dummy-SMTP which I chose where to run the listener.
To install Dummy-SMTP either clone the original repository or my forked one:

cd /srv/
git clone https://github.com/Napsty/Dummy-SMTP.git
chown -R appuser:appuser /srv/Dummy-SMTP

 Once the repository was cloned, become "appuser" if you aren't already and launch the listen.py file:

appuser@app01-test:~$ /srv/Dummy-SMTP/listen.py &
[1] 32189
Running fake smtp server on port 1025

Now I adapted the Postfix installation and defined that all mails should be relayed to this fake smtp server by setting the relayhost parameter:

cat /etc/postfix/main.cf|grep relayhost
relayhost = 127.0.0.1:1025

service postfix reload

From now on a mail sent from this server should be relayed to the dummy smtp server and be just stored as a text file in /srv/Dummy-SMTP/mails. Let's try this:

echo "Dummy SMTP Test" | mailx -s "Test" recipient@example.com

mail.log shows:

app01-test postfix/pickup[30289]: 3559024A6A: uid=0 from=
app01-test postfix/cleanup[4940]: 3559024A6A: message-id=<20160607120336.3559024A6A@app01-test.local>
app01-test postfix/qmgr[30290]: 3559024A6A: from=, size=385, nrcpt=1 (queue active)
app01-test postfix/smtp[4942]: 3559024A6A: to=, relay=127.0.0.1[127.0.0.1]:1025, delay=0.03, delays=0.02/0.01/0/0, dsn=2.0.0, status=sent (250 Ok)
app01-test postfix/qmgr[30290]: 3559024A6A: removed

And let's check out the Dummy SMTP mail folder:

ls /srv/Dummy-SMTP/mails
1465301016.24.txt

cat /srv/Dummy-SMTP/mails/1465301016.24.txt
Received: by app01-test.local (Postfix, from userid 0)
    id 3559024A6A; Tue,  7 Jun 2016 14:03:36 +0200 (CEST)
Subject: Test
To:
X-Mailer: mail (GNU Mailutils 2.99.98)
Message-Id: <20160607120336.3559024A6A@n
app01-test.local>
Date: Tue,  7 Jun 2016 14:03:36 +0200 (CEST)
From: root@nzzshop-app01-test (root)

Dummy SMTP Test

Neat! It worked.

The above mentioned Ruby alternative Mailcatcher has one nice feature though: The mails can be seen by browser. Well, this isn't very complicated either. The mail folder of the dummy smtp server is a given (/srv/Dummy-SMTP/mails) so by creating a simple "Alias" on the Apache running already on this test server, I was able to display all sent mails on the browser, too:

cat /etc/apache2/conf-enabled/dummysmtp.conf
Alias /mails "/srv/Dummy-SMTP/mails/"


    Require all granted
    Options +FollowSymLinks +Indexes
    AllowOverride All

service apache2 reload

Nothing fancy of course, but it works:

Fake dummy smtp server showing mails in browser


 

Raspberry Pi sd card issue after power outage (constant red led)
Tuesday - Jun 7th 2016 - by - (0 comments)

Last night we had a (announced) power outage at work. This morning everything seemed to work again except one thing: The Raspberry Pi 2 showing our monitoring status on the TV screen. Nooooo, we're blind!

The Raspberry Pi's LED was a constant red. No output on HDMI and also keyboard/mouse didn't show any connectivity. Even without any USB devices connected and power cable unplugged/replugged the Pi wouldn't boot. 

I came across a few pages with good information for troubleshooting booting issues:

My first fear was that the power outage caused a defect in the polyfuse and that I'd either have to replace it or wait some time (up until a couple of days) so it auto-resets itself. But a lot of the posts almost always blamed the SD card for boot issues. I didn't really think this could be the cause but I needed to make sure anyways.

So I inserted the micro sd card into my computer and voilà, there's definitely something bad:

[ 2882.284344] usb 2-3: USB disconnect, device number 2
[ 2902.054057] mmc0: new high speed SDHC card at address 59b4
[ 2902.060081] Driver 'mmcblk' needs updating - please use bus_type methods
[ 2902.060139] mmcblk0: mmc0:59b4 00000 7.35 GiB
[ 2902.063721]  mmcblk0: p1 p2 < p5 p6 > p3
[ 2904.227853] EXT4-fs (mmcblk0p3): recovery complete
[ 2904.235789] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
[ 2904.269231] FAT-fs (mmcblk0p5): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 2906.018193] EXT4-fs (mmcblk0p6): 9 orphan inodes deleted
[ 2906.018196] EXT4-fs (mmcblk0p6): recovery complete
[ 2906.065988] EXT4-fs (mmcblk0p6): mounted filesystem with ordered data mode. Opts: (null)
[ 2907.605639] mmcblk0: error -110 transferring data, sector 6016528, nr 56, cmd response 0x900, card status 0xb00
[ 2907.605643] mmcblk0: retrying using single block read
[ 2909.153337] mmcblk0: error -110 transferring data, sector 1830160, nr 32, cmd response 0x900, card status 0xb00
[ 2909.153349] mmcblk0: retrying using single block read
[ 2915.557948] mmcblk0: error -110 transferring data, sector 10471632, nr 8, cmd response 0x900, card status 0xb00
[ 2915.557953] mmcblk0: retrying using single block read
[ 2916.088688] mmcblk0: error -110 transferring data, sector 10472016, nr 8, cmd response 0x900, card status 0xb00
[ 2916.088694] mmcblk0: retrying using single block read

As dmesg suggested, I launched a fsck on /dev/mmcblk0p5 and got this:

fsck /dev/mmcblk0p5
fsck from util-linux 2.20.1
fsck.fat 3.0.26 (2014-03-07)
0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 1
Leaving filesystem unchanged.
/dev/mmcblk0p5: 74 files, 9965/30651 clusters

So a plain fsck didn't really repair the file system (Leaving filesystem unchanged). On one of the pages mentioned above, the following options were mentioned: -trawl.
From the --help output of fsck.fat, trawl means:

  -t       test for bad clusters
  -r       interactively repair the filesystem
  -a       automatically repair the filesystem
  -w       write changes to disk immediately
  -l       list path names

OK, looks legit to me. Let's roll:

fsck.fat -trawl /dev/mmcblk0p5
fsck.fat 3.0.26 (2014-03-07)
0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Checking file /boot0
Checking file /COPYING.linux (COPYIN~1.LIN)
Checking file /LICENCE.broadcom (LICENC~1.BRO)
Checking file /LICENSE.oracle (LICENS~1.ORA)
Checking file /bcm2708-rpi-b-plus.dtb (BCM270~1.DTB)
Checking file /bcm2708-rpi-b.dtb (BCM270~2.DTB)
Checking file /bcm2709-rpi-2-b.dtb (BCM270~3.DTB)
Checking file /bootcode.bin (BOOTCODE.BIN)
Checking file /cmdline.txt (CMDLINE.TXT)
Checking file /config.txt (CONFIG.TXT)
Checking file /fixup.dat (FIXUP.DAT)
Checking file /fixup_cd.dat (FIXUP_CD.DAT)
Checking file /fixup_db.dat (FIXUP_DB.DAT)
Checking file /fixup_x.dat (FIXUP_X.DAT)
Checking file /issue.txt (ISSUE.TXT)
Checking file /kernel.img (KERNEL.IMG)
Checking file /kernel7.img (KERNEL7.IMG)
Checking file /os_config.json (OS_CON~1.JSO)
Checking file /start.elf (START.ELF)
Checking file /start_cd.elf (START_CD.ELF)
Checking file /start_db.elf (START_DB.ELF)
Checking file /start_x.elf (START_X.ELF)
Checking file /overlays (OVERLAYS)
Checking file /bcm2708-rpi-cm.dtb (BCM270~4.DTB)
Checking file /overlays/.
Checking file /overlays/..
Checking file /overlays/hifiberry-amp-overlay.dtb (HIFIBE~1.DTB)
Checking file /overlays/tinylcd35-overlay.dtb (TINYLC~1.DTB)
Checking file /overlays/bmp085_i2c-sensor-overlay.dtb (BMP085~1.DTB)
Checking file /overlays/hifiberry-digi-overlay.dtb (HIFIBE~2.DTB)
Checking file /overlays/i2c-rtc-overlay.dtb (I2C-RT~1.DTB)
Checking file /overlays/ads7846-overlay.dtb (ADS784~1.DTB)
Checking file /overlays/rpi-display-overlay.dtb (RPI-DI~1.DTB)
Checking file /overlays/README
Checking file /overlays/hifiberry-dac-overlay.dtb (HIFIBE~3.DTB)
Checking file /overlays/at86rf233-overlay.dtb (AT86RF~1.DTB)
Checking file /overlays/hy28b-overlay.dtb (HY28B-~1.DTB)
Checking file /overlays/rpi-proto-overlay.dtb (RPI-PR~1.DTB)
Checking file /overlays/hy28a-overlay.dtb (HY28A-~1.DTB)
Checking file /overlays/dht11-overlay.dtb (DHT11-~1.DTB)
Checking file /overlays/mz61581-overlay.dtb (MZ6158~1.DTB)
Checking file /overlays/iqaudio-dacplus-overlay.dtb (IQAUDI~1.DTB)
Checking file /overlays/lirc-rpi-overlay.dtb (LIRC-R~1.DTB)
Checking file /overlays/w1-gpio-overlay.dtb (W1-GPI~1.DTB)
Checking file /overlays/pps-gpio-overlay.dtb (PPS-GP~1.DTB)
Checking file /overlays/mcp2515-can0-overlay.dtb (MCP251~1.DTB)
Checking file /overlays/rpi-dac-overlay.dtb (RPI-DA~1.DTB)
Checking file /overlays/piscreen-overlay.dtb (PISCRE~1.DTB)
Checking file /overlays/spi-bcm2835-overlay.dtb (SPI-BC~1.DTB)
Checking file /overlays/w1-gpio-pullup-overlay.dtb (W1-GPI~2.DTB)
Checking file /overlays/pitft28-resistive-overlay.dtb (PITFT2~1.DTB)
Checking file /overlays/enc28j60-overlay.dtb (ENC28J~1.DTB)
Checking file /overlays/iqaudio-dac-overlay.dtb (IQAUDI~2.DTB)
Checking file /overlays/hifiberry-dacplus-overlay.dtb (HIFIBE~4.DTB)
Checking file /overlays/gpio-poweroff-overlay.dtb (GPIO-P~1.DTB)
Checking file /overlays/i2c-gpio-overlay.dtb (I2C-GP~1.DTB)
Checking file /overlays/i2s-mmap-overlay.dtb (I2S-MM~1.DTB)
Checking file /overlays/mcp2515-can1-overlay.dtb (MCP251~2.DTB)
Checking file /overlays/mmc-overlay.dtb (MMC-OV~1.DTB)
Checking file /overlays/piscreen2r-overlay.dtb (PISCRE~2.DTB)
Checking file /overlays/pitft28-capacitive-overlay.dtb (PITFT2~2.DTB)
Checking file /overlays/pwm-2chan-overlay.dtb (PWM-2C~1.DTB)
Checking file /overlays/pwm-overlay.dtb (PWM-OV~1.DTB)
Checking file /overlays/raspidac3-overlay.dtb (RASPID~1.DTB)
Checking file /overlays/rpi-backlight-overlay.dtb (RPI-BA~1.DTB)
Checking file /overlays/rpi-ft5406-overlay.dtb (RPI-FT~1.DTB)
Checking file /overlays/rpi-sense-overlay.dtb (RPI-SE~1.DTB)
Checking file /overlays/sdhost-overlay.dtb (SDHOST~1.DTB)
Checking file /overlays/sdio-overlay.dtb (SDIO-O~1.DTB)
Checking file /overlays/smi-dev-overlay.dtb (SMI-DE~1.DTB)
Checking file /overlays/smi-nand-overlay.dtb (SMI-NA~1.DTB)
Checking file /overlays/smi-overlay.dtb (SMI-OV~1.DTB)
Checking file /overlays/spi-bcm2708-overlay.dtb (SPI-BC~2.DTB)
Checking file /overlays/spi-dma-overlay.dtb (SPI-DM~1.DTB)
Checking file /overlays/uart1-overlay.dtb (UART1-~1.DTB)
Checking file /overlays/vga666-overlay.dtb (VGA666~1.DTB)
Performing changes.
/dev/mmcblk0p5: 74 files, 9965/30651 clusters

Alright, another fsck please to see if the dirty bit was removed:

fsck /dev/mmcblk0p5
fsck from util-linux 2.20.1
fsck.fat 3.0.26 (2014-03-07)
/dev/mmcblk0p5: 74 files, 9965/30651 clusters

Yep, looks better.

Once I inserted the sd card into the Raspberry Pi again, I couldn't believe it - that thing booted again. So it really was the sd card to blame!

 

Presenting check_es_store plugin: Monitor ElasticSearch store (disk) usage
Thursday - Jun 2nd 2016 - by - (0 comments)

I'm happy to release yet another monitoring plugin intended for monitoring platforms like Nagios, Icinga etc: check_es_store.

What does it do?
The plugin connects to your ElasticSearch cluster node and checks the store/disk usage.

But that's what check_disk is for!
Well, not exactly. Lots of people (mainly developers) don't actually run their own ElasticSearch servers. If they did, yes, check_disk would be wise. But for all the other people having an ElasticSearch cluster in the cloud (SaaS) you know you have a certain size of a cluster you are paying for, but you don't actually know how much is currently used. Yes, there are surely tools and graphs from the ES cluster provider, but if you already have a monitoring, it makes sense to include this information right in there.

So in short: If you have a Nagios or Icinga monitoring and run an ElasticSearch cluster in the cloud; this plugin is for you.

Enjoy!

http://www.claudiokuenzler.com/nagios-plugins/check_es_store.php 

 

Limit http request methods in Nginx (authentication except for GET)
Tuesday - May 31st 2016 - by - (0 comments)

Was looking for a way to tell Nginx to let all GET requests through but all other requests (e.g. POST, PUT, etc) should be authenticated.

On my research I found a lot of examples using "if", like this one (found on https://www.acunetix.com/blog/articles/nginx-server-security-hardening-configuration-1/):

if ($request_method !~ ^(GET|HEAD|POST)$ )
{
    return 444;
}

However there are two problems.

1) if is evil: https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ 

2) basic auth does not work within an if clause in Nginx. So the following snippet doesn't work:

if ($request_method !~ ^GET$ )
{
  auth_basic "Login required";
  auth_basic_user_file /etc/nginx/auth/.htpasswd;
}

Belive me, I tried it and fell on my nose... On my research WHY it fails, I came across this page: https://t37.net/the-good-the-bad-and-the-ugly-of-virtual-hosting-with-nginx.html, which explains it nicely:

The reason why it fails is because if is not part of the general configuration module as one should believe. if is part of the rewrite module and auth_basic is another module. That’s one of the reason why the Nginx community thinks if is evil.

Eventually I came across the limit_except method, which is part of the Nginx core module. By using this method, one can define special limits for http request methods. The documentation (http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_except) describes the method in a bizarre way though:

Limits allowed HTTP methods inside a location. The method parameter can be one of the following: GET, HEAD, POST, PUT, DELETE, MKCOL, COPY, MOVE, OPTIONS, PROPFIND, PROPPATCH, LOCK, UNLOCK, or PATCH. Allowing the GET method makes the HEAD method also allowed. Access to other methods can be limited using the ngx_http_access_module and ngx_http_auth_basic_module modules directives

What I mean with bizarre: It's not clear whether the limitation is now for the mentioned request methods or for the unlisted request methods. But a configuration change quickly showed me that it works the following way:

  location /api/ {

    # Only allow GET, all other methods require basic auth
    limit_except GET {
      auth_basic           "Login required";
      auth_basic_user_file /etc/nginx/auth/.htpasswd;
    }

    include /etc/nginx/proxy.conf;
    proxy_pass       http://127.0.0.1:8080;
  }

The comment I added just above limit_except explains it: Set a limit for all http request methods (load the auth_basic stuff) EXCEPT the request method is GET.

Works nicely and I didn't have to use "if" either.

 

New version of check_esxi_hardware allows different CIM ports (NAT)
Tuesday - May 31st 2016 - by - (0 comments)

There's a new version (20160531) of the Monitoring Plugin to monitor the hardware of ESXi servers, check_esxi_hardware.py, out! This version contains a new feature which allows manually define the CIM port.

The default CIM port listens on tcp/5989. However if you have several ESXi servers behind one IP address (NAT) you can now use port forwardings on your firewall our router and monitor all of them, too.

The newly added parameter is "-C" (--cimport). Documentation of check_esxi_hardware is up to date, too.

Enjoy!

 

Shopware shows content correctly but with http status 503
Monday - May 30th 2016 - by - (0 comments)

Had a strange case of application problem today, when a webshop running on Shopware 5.x started to sporadically return http 503 status and therefore triggering our monitoring system and disabling the website in the reverse proxies (which are running in front of the application server). 

By fiddling around with local hosts entries and therefore bypassing the reverse proxies, something strange could be seen in the browser: The web shop is shown correctly, however the HTTP status from the application server is a 503 (see screenshot).

Shopware content error 503 

First it started off with sporadic 503 returns but after a while every request got a 503 status return. Neither in the Apache/PHP nor in the Shopware logs was a hint what would cause this. More or less by chance I decided to rename the cache folder (SHOPWARE_ROOT/var/cache) and create a new empty cache folder. And voilà, the application worked again and displayed the shop once again with a correct HTTP 200 status code.

 

SpamAssassin stopped working? Check loopback interface!
Thursday - May 26th 2016 - by - (0 comments)

Got the following error messages on a mailserver:

spamc[11629]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection timed out

A manual restart of spamassassin only resulted in the following error:

/etc/init.d/spamassassin restart
Restarting SpamAssassin Mail Filter Daemon: No /usr/bin/perl found running; none killed.
[12244] warn: server socket setup failed, retry 1: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 2: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 3: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 4: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 5: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 6: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 7: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 8: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] warn: server socket setup failed, retry 9: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
[12244] error: spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address
spamd: could not create INET socket on 127.0.0.1:783: Cannot assign requested address

First I suspected a hanging PID file somewhere or an already opened listener port 783 (but then usually an error "address already in use" is shown). But nope, port was free to be used.

Only when I checked ip addr show all became clear:

ip addr show
1: lo: mtu 16436 qdisc noop state DOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 8c:89:a5:15:6b:9d brd ff:ff:ff:ff:ff:ff
    inet 123.4.5.123/32 brd 123.4.5.255 scope global eth0
[...]

The loopback (lo) interface was DOWN! This happened because this server had a network problem after the boot and the activation of the loopback interface got forgotten (yet eth0 was working as it should).
After enabling the lo interface and restart SpamAssassin, all worked again:

ifup lo

/etc/init.d/spamassassin restart
Restarting SpamAssassin Mail Filter Daemon: No /usr/bin/perl found running; none killed.
spamd.

 

How to retrieve values from json output in bash
Friday - Apr 29th 2016 - by - (0 comments)

Sure, one could get a json output with curl and then do all kinds of grepping, sedding and awking. But there is actually a cool way, to get the value directly by accessing the json objects. The magic is done by jshon

Fortunately this (cli-)program doesn't need to be compiled manually, in Ubuntu it can be downloaded from the official repos:

sudo apt-get install jshon

Now let's do some json'ing. Let's take the ElasticSearch cluster stats for example. As you may know, you can get the current ES cluster statistics and information from the following URL: http://your.es.node:9200/_cluster/stats (for a human readable output you can use http://your.es.node:9200/_cluster/stats?human&pretty). 

The json output is separated into several objects. You can use jshon to list them:

$ curl "http://elasticsearch.local:9200/_cluster/stats" -s | jshon -k
timestamp
cluster_name
status
indices
nodes

As you see above, I directly accessed the ES URL and piped the output to jshon. The -k parameter returns a list of keys. In this case we got 5 keys.

The value I am looking for is within the "indices" key, so I first display the values of the full key with the -e parameter (returns json value):

$ curl "http://elasticsearch.local:9200/_cluster/stats" -s | jshon -e indices
{
 "count": 13,
 "shards": {
  "total": 61,
  "primaries": 61,
  "replication": 0.0,
  "index": {
   "shards": {
    "min": 1,
    "max": 5,
    "avg": 4.6923076923076925
   },
   "primaries": {
    "min": 1,
    "max": 5,
    "avg": 4.6923076923076925
   },
   "replication": {
    "min": 0.0,
    "max": 0.0,
    "avg": 0.0
   }
  }
 },
 "docs": {
  "count": 935249,
  "deleted": 196434
 },
 "store": {
  "size": "11.9gb",
  "size_in_bytes": 12860502682,
  "throttle_time": "3.7h",
  "throttle_time_in_millis": 13374212
 },
 "fielddata": {
  "memory_size": "7.9mb",
  "memory_size_in_bytes": 8328212,
  "evictions": 0
 },
 "filter_cache": {
  "memory_size": "78.1mb",
  "memory_size_in_bytes": 81991016,
  "evictions": 72989502
 },
 "id_cache": {
  "memory_size": "0b",
  "memory_size_in_bytes": 0
 },
 "completion": {
  "size": "0b",
  "size_in_bytes": 0
 },
 "segments": {
  "count": 453,
  "memory": "17.9mb",
  "memory_in_bytes": 18873642,
  "index_writer_memory": "0b",
  "index_writer_memory_in_bytes": 0,
  "index_writer_max_memory": "190.1mb",
  "index_writer_max_memory_in_bytes": 199421333,
  "version_map_memory": "0b",
  "version_map_memory_in_bytes": 0,
  "fixed_bit_set": "3.2mb",
  "fixed_bit_set_memory_in_bytes": 3448160
 },
 "percolate": {
  "total": 0,
  "get_time": "0s",
  "time_in_millis": 0,
  "current": 0,
  "memory_size_in_bytes": -1,
  "memory_size": "-1b",
  "queries": 0
 }
}

So this is the full output, but not the single value of what I actually wanted. To do this, the subkeys within the "indices" key can be shown and used the same way:

$ curl "http://elasticsearch.local:9200/_cluster/stats" -s | jshon -e indices -k
percolate
shards
count
store
docs
fielddata
filter_cache
id_cache
completion
segments

So now we got a nice list of the subkeys which we can access directly with an additional -e:

$ curl "http://elasticsearch.local:9200/_cluster/stats" -s | jshon -e indices -e store
{
 "size": "11.9gb",
 "size_in_bytes": 12860502682,
 "throttle_time": "3.7h",
 "throttle_time_in_millis": 13374212
}

And to get the final single value I wanted from the start, this can be retrieved by using yet another -e parameter:

$ curl "http://elasticsearch.local:9200/_cluster/stats" -s | jshon -e indices -e store -e "size_in_bytes"
12860502682

 


Go to Homepage home
Linux Howtos how to's
Nagios Plugins nagios plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7876 Days
until Death of Computers
Why?