Header RSS Feed
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Nagios/Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

MongoDB 3.x: Create log rotation script
Monday - Jul 6th 2015 - by - (0 comments)

On a recently installed MongoDB 3.x server, it seems that MongoDB now writes much more logs than the previous 2.x version.

According to the documentation, a log rotation, done by MongoDB itself, can be forced by sending the SIGUSR1 signal to the MongoDB process. When this is done, a rotated log file is created with a date extension:

root@mongoserver:/var/log/mongodb# ls -ltr
-rw-r--r-- 1 mongodb nogroup 168167263 Jul  6 10:31 mongod.log.2015-07-06T08-31-32
-rw-r--r-- 1 mongodb nogroup   1371461 Jul  6 10:32 mongod.log

My goal was to use send SIGUSR1 as a prescript command in logrotate and then compress the newly created file (mongod.log.2015-07-06T08-31-32 in this case). Unfortunately this doesn't work with logrotate, because the file extension, created by MongoDB, is too variable.
So I quickly created the following script which sends SIGUSR1 to the MongoDB PID, then compresses the rotated logfile:

# MongoDB Logrotate Script
# (c) Claudio Kuenzler www.claudiokuenzler.com
# 20150706 ck  Created script
# Variables
# Start
test -f $logpath/mongod.log || exit 1

echo "Starting MongoDB log rotation"
echo "Current logfile:"
ls -la $logpath/mongod.log
echo "Launching SIGUSR1 command"
kill -SIGUSR1 `cat $pidfile`
echo "Compressing new logfile"
find $logpath/ -name "mongod.log.$(date +%Y)*" ! -name "*.gz" -exec gzip {} +
echo "Finished MongoDB log rotation"
echo "-----------------------------------------------------"
) 2>&1 | tee -a ${logpath}/rotate.log

exit 0

This script is now run once a day by as a cronjob.

Successfully tested on Ubuntu 14.04 LTS with MongoDB 3.0.4 (installed from 10gen repo).

Side note: It's of course also possible to add a deletion of old rotated logfiles into the script, for example:

find $logpath/ -name "mongod.log.*gz" -mtime +7 -exec rm {} +


Samsung Galaxy S5 (G901F): Pain to install custom recovery or Cyanogenmod
Thursday - Jul 2nd 2015 - by - (0 comments)

A couple of months ago, I got a new phone from work, a Samsung Galaxy S5 G901F (also known as Samsung Galaxy S5 Plus). Ever since there are several points which greatly annoy me:

  • Samsung's Touchwiz interface is just a pain in the ass, it's bloody slow and looks ugly (trying to look iPhone-alike)
  • Already came across several bugs (Wifi tethering doesnt work, mail sync issues, ...)
  • Seen no updates so far - the phone is still on 4.4.4 but Android 5.1 is meanwhile out
  • Bloody customizations from the phone provider (Swisscom)
  • Other bloatware

Whenever such annoyances bugged me, I ditched the stock ROM and replaced it by CyanogenMod (successfully installed Cyanogenmod 7 on Motorola Milestone and later Cyanogenmod 9 on a Samsung Galaxy S2). So I decided to finally do it on the Samsung Galaxy S 5, too.

But that's easier said than done. 

Spoiler: The reason for all this is because the Samsung Galaxy S5 Plus G901F is not a klte device, but rather a kccat6 device. 

1. Cyanogenmod's install instructions for the Samsung Galaxy S5 don't work
The official documentation how to install CyanogenMod on the Samsung Galaxy S5 simply didn't work. When I tried to flash the custom recovery with heimdall, I got the following errors:

heimdall.exe flash --RECOVERY "C:\Program Files\Heimdall\recovery.img" --no-reboot
Heimdall v1.4.0

Copyright (c) 2010-2013, Benjamin Dobell, Glass Echidna

This software is provided free of charge. Copying and redistribution is

If you appreciate this software and you would like to support future
development please consider donating:

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
ERROR: Failed to send request to end PIT file transfer!
ERROR: Failed to download PIT file!
Ending session...
ERROR: Failed to send end session packet!
Releasing device interface...

2. Manual installation of a custom recovery is tricky
In order to install Cyanogenmod I first need to have a custom recovery on the phone. I tried different versions of TWRP, CWM but almost all of them showed an error message ("Could not do normal boot") when I tried to boot into the Recovery mode and it went straight to the Download Mode.
The issue here is, as I found out later (see Spoiler above), that the phone is not a klte device. And almost all firmwares, roms and root instructions are made for the klte type of the Samsung Galaxy S5. But the G901F is  a kccat6 device.
I was finally able to find a working CWM file which I could flash with odin.

3. How to manually flash a custom recovery (CWM) on your G901F
Eventually I came across the following post in the XDA Developers forums: SM - G901F Rooted at last ! CWM Recovery in English !
Halleluja! With the downloadedable file (g901f-cwm-recovery- I was finally able to put CWM on the phone (and it staid there after a reboot, too).
Here's a step by step instruction:

3.1 Download odin3 v3.09 (or newer) from http://odindownload.com/.

3.2 Download and install the Samsung USB drivers (SAMSUNG_USB_Driver_for_Mobile_Phones.zip) from http://developer.samsung.com/technical-doc/view.do?v=T000000117. You should probably reboot your computer afterwards just to make sure.

3.3 Unpack the odin zip file and launch the odin3 exe (Odin3 v3.09.exe).

3.4 Power off your Samsung Galaxy S5 G901F phone.

3.5 Boot your phone into the Download Mode by pressing the following buttons altogether: Volume Down + Home + Power until you see a warning triangle. Accept the warning by pressing the Volume Up button.

Samsung Galaxy S5 Plus Download Mode Samsung Galaxy S5 Plus Download Mode

3.6 Plug your phone with the USB cable to your computer. In Odin the device should now be detected by showing the communication port in ID:COM. In my case this was "0:[COM8]".

3.7 In Odin, disable the "Auto Reboot" option. On the right side (Files [Download]) click on the "PDA" or "AP" button and select the previously downloaded g901f-cwm-recovery- file. Then click on the Start button.
Odin will then display several log entries in the "Message" box. Wait until there is a green box "PASS" displayed above the communication port.
On your phone a blue bar should have appeared.

3.8 Disconnect your USB cable from the phone. Open your phone and take the battery out (yes, the phone is still running). Wait a few seconds and place your battery back in, close your phone.

3.9 Boot your phone into the Recovery Mode by pressing the following button combination: Volume Up + Home + Power. Keep it pressed until you see a small blue text appearing on top of the screen.

3.10 Congratz! You are now in CWM, a custom recovery.

CWM Recovery on G901F


4. Installation of CyanogenMod - fail
Now that the custom recovery/CWM is installed and working on the phone, I proceeded to (finally) install CyanogenMod. I booted the phone normally and copied the current CM12.1 nightly (cm-12.1-20150702-NIGHTLY-klte.zip) and the Google Apps package (gapps-5.1-2015-06-02-15-19.zip) to the phone's internal storage.

Placed cyanogenmod on device storage 

Afterwards I powered off the phone and booted into the Recovery Mode.

In CWM, I wanted to install CyanogenMod by using "install zip" -> "choose zip from /sdcard" -> select cm-12.1-20150702-NIGHTLY-klte.zip.

CyanogenMod installation G901F CyanogenMod installation G901F CyanogenMod installation G901F CyanogenMod installation G901F

The following error message then appeared on the phone:

This package is for device: kltexx, ltelra, kltetmo, kltecan, klteatt, klteub, klteacg, klte, kltekor, klteskt, kltektt; this device is kccat6

CyanogenMod installation G901F 

This is when I (finally!) found out that I have a Samsung Galaxy S5 Plus which isn't a klte device. Hence I could finally explain myself why so many trials to install a custom recovery failed.

Unfortunately to this day, there is no CyanogenMod version for this device (kccat6) available. A forum thread exists where you can add your demand: http://forum.cyanogenmod.org/topic/104658-samsung-galaxy-s5-plus-sm-g901f/

But at least now I'm prepared with CWM if CM ever comes to this device.

Additional notes: A working tutorial how to root your G901F can be found here: http://androidxda.com/root-samsung-galaxy-s5-lte-sm-g901f


New version of check_esxi_hardware fixes issue with patched pywbem
Friday - Jun 26th 2015 - by - (0 comments)

In the past few months I got more and more feedback concerning the following error when the Nagios plugin check_esxi_hardware was launched:

Unknown CIM Error: (0, 'SSL error: certificate verify failed')

As described in the FAQ of check_esxi_hardware, the problem often appeared after the system, on which the plugin runs, was updated. See also issue #7 on the github repo. But there is actually deeper history into that.

You're all aware that the core of check_esxi_hardware is the python module "pywbem". The upstream project appeared to be dead for a long time. When SSL security/vulnerability issues arose, some distributions manually fixed the code of pywbem which resulted in check_esxi_hardware failing because the syntax to connect to a CIM server has changed. Another problem was/still is, that the pywbem version never changed; it staid at 0.7.0 for several years, although there were many patches applied in several distros.

In September 2014 everything started to look better as it seemed, there was life in the pywbem project again. A new release of a version 0.8.0 seemed close. But once again there is no news anymore since the end of January 2015.

Nevertheless, the Nagios plugin check_esxi_hardware needed to be adapted for patched pywbems.
From today on version 20150626 fixes the compatibility issue and allows to be used with the "old" pywbem version as well as with patched pywbem 0.7.0 packages. And it is also ready for the new 0.8.0 release if that is ever to be shipped into distros.

The 20150626 version was successfully tested on the following Linux distros:

  • openSUSE 13.2 with python 2.7.8 and pywbem 0.8.0-dev.r728 (from upstream source)
  • Ubuntu 14.04 LTS (trusty) with python 2.7.6 and pywbem 0.7.0-4ubuntu1~14.04.1 (official Ubuntu repo)
  • Unbunt 12.04 LTS (precise) with python 2.7.3 and pywbem 0.7.0-4 (official Ubuntu repo)



HTTP content encoding and caching analysis with tcpdump and wireshark
Friday - Jun 26th 2015 - by - (0 comments)

To analyze a http caching problem I needed to grab the network connections and take a look into the http protocol and find possible problematic http requests. I used tcpdump to capture the tcp streams and wireshark to analyze the captured packets.

The following filters came in handy.

Show connections which requested www.example.com/ (the main domain) on the webserver

http.request.uri == "/" && http.host == "www.example.com" && ip.dst ==

Show connections which contain a HTTP 200 response code but don't contain the "Content-Encoding" http header:

http.response.code == 200 && !http.content_encoding

Show http responses where the content was gzip compressed:

http.content_encoding == gzip

Don't show http content, only headers (from http://www.askapache.com/hosting/debugging-http-cache-headers-wireshark.html):

http.response !=0 || http.request.method != "TRACE"

To be able to create filters with custom http headers, I first needed to add them to Wiresharks preferences:
Edit -> Preferences -> Protocols -> HTTP -> Custom HTTP headers fields -> Edit

I added the following additional headers:

  • Age: Age of Cache
  • X-Cache: Cache Type
  • X-Varnish-Hostname: Hostname of Varnish Cache

To use these custom http headers as a filter, you need to use the http.header prefix.

Show http responses which weren't cached, which don't contain the "Content-Encoding" header and which were treated by varnish-3 server:

http.header.X-Cache == MISS && !http.content_encoding && http.header.X-Varnish-Hostname == varnish-3

Show http responses which were cached (HIT) and treated by varnish-4 server:

http.header.X-Cache == HIT && http.header.X-Varnish-Hostname == varnish-4

Show http responses which pass through a varnish server (so the header X-Varnish-Hostname exists), have response code 200 and don't contain the "Content-Encoding" header:

http.header.X-Varnish-Hostname && http.response.code == 200 && !http.content_encoding


Change sort order in Icinga Classic UI with Icinga 2 Core
Thursday - Jun 18th 2015 - by - (0 comments)

On my Icinga 2 installation with the Icinga-Classic UI there is a new default sort of the services. Instead of having the services of a host sorted in alphabetical order, they're now sorted by the last check. Meaning: The service which was last recently checked is on top. 

The overview of the services of a host then looks like this:

Icinga Classic UI with Icinga 2: Service sort 

To change this behaviour back to the old ways (sort by alphabet by default), the cgi parameter sort_status_data_by_default needs to be enabled in the cgi.conf file:

cat cgi.cfg  | grep sort

According to the documentation, the sorting can be a little messed up when using Icinga 2 as core:

This option forces status.cgi to sort status data by default. Data will be sorted ascending. First by host name and then by service description. This is helpful if status data in status.dat is not sorted properly. This can occur especially if you use Icinga2. By default this options is switched off (0) to avoid possible higher cpu load in huge environments.


Use FTPES with pure-ftpd and solve connection timeout issues
Friday - Jun 12th 2015 - by - (0 comments)

In theory activating FTPES (FTP with explicit TLS encryption) on a pure-ftpd FTP server is pretty straightforward.

1. Create a file /etc/ssl/private/pure-ftpd.pem
This pem file contains both private key and certificate (can also contain the intermediate certificate as well). For security reasons set relevant permissions that only root is able to read this file.

echo -ne "-----BEGIN RSA PRIVATE KEY-----\nMIIE....\n-----END CERTIFICATE-----\n" >  /etc/ssl/private/pure-ftpd.pem

chmod 600 /etc/ssl/private/pure-ftpd.pem

2. Enable TLS encryption in pure-ftpd
On a current Ubuntu 14.04 LTS, the pure-ftpd config options are handled through files in /etc/pure-ftpd/conf. To enable TLS, a specific value has to be set in the file /etc/pure-ftpd/conf/TLS.

Enable TLS (but continue to allow normal plain-text connections):

echo 1 > /etc/pure-ftpd/conf/TLS

Enable TLS and allow only encrypted connections:

echo 2 > /etc/pure-ftpd/conf/TLS

3. Restart pure-ftpd

service pure-ftpd restart
Restarting ftp server: Running: /usr/sbin/pure-ftpd -l puredb:/etc/pure-ftpd/pureftpd.pdb -l pam -Y 1 -O clf:/var/log/pure-ftpd/transfer.log -A -8 UTF-8 -d -E -u 1000 -B

The output shows the currently defined pure-ftpd parameters, defined by the available config files in /etc/pure-ftpd.

So much for the setup itself. But now to the problem. As soon as I tried to connect to this pure-ftpd server with FTPES, I got the following error message in FileZilla:

Status:    Connection established, waiting for welcome message...
Status:    Initializing TLS...
Status:    Verifying certificate...
Status:    TLS connection established.
Status:    Connected
Status:    Retrieving directory listing...
Command:    PWD
Response:    257 "/" is your current location
Command:    TYPE I
Response:    200 TYPE is now 8-bit binary
Command:    PASV
Response:    227 Entering Passive Mode
Command:    MLSD
Error:    Connection timed out after 20 seconds of inactivity
Error:    Failed to retrieve directory listing

In pure-ftpd's verbose log I only saw the following entries:

Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [INFO] New connection from myclient.local
Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [DEBUG] Command [auth] [TLS]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [INFO] SSL/TLS: Enabled TLSv1/SSLv3 with AES256-GCM-SHA384, 256 secret bits cipher
Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [DEBUG] Command [user] [ftpuser]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [DEBUG] Command [pass] [<*>]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (?@myclient.local) [INFO] ftpuser is now logged in
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [opts] [UTF8 ON]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [pbsz] [0]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [prot] [P]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [pwd] []
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [type] [I]
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [pasv] []
Jun 12 10:49:36 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [DEBUG] Command [mlsd] []
Jun 12 10:51:33 ftpserver01 pure-ftpd: (ftpuser@myclient.local) [INFO] Logout.

When I tried the same connection without encryption, the connection worked.

It turns out that, as soon as encryption is used, the FTP connection is continued on a passive high port, which of course needs to be enabled in the firewall, too. A "normal" non-encrypted ftp connection keeps its connection on the server port 21.

In order to limit the number of passive high ports, I defined them in yet another pure-ftpd config file, called PassivePortRange:

echo "20000 20500" > /etc/pure-ftpd/conf/PassivePortRange

This means that additional (passive) FTP ports are defined between tcp/20000 and tcp/20500.
After a restart of pure-ftpd and modification of the incoming firewall rules, the connection finally worked:

Status:    Connection established, waiting for welcome message...
Status:    Initializing TLS...
Status:    Verifying certificate...
Status:    TLS connection established.
Status:    Connected
Status:    Retrieving directory listing...
Status:    Verifying certificate...
Status:    Directory listing of "/" successful

So keep in mind to (define and) create firewall rules for additional passive FTP ports when you want your FTP connections encrypted.


HAProxy: Make httpchk on different port than application
Thursday - Jun 11th 2015 - by - (0 comments)

Between an nginx web servers and its php-fpm upstream servers I wanted to add an additional check if php-fpm and mysql is running correctly on the upstream servers.

The current setup looks like this:

HTTP Request -> Nginx
                 |------> tcp/9000 on PHP Upstream 1 -> local mysql
                 |------> tcp/9000 on PHP Upstream 2 -> local mysql
                 |------> tcp/9000 on PHP Upstream 3 -> local mysql

 The problem is, that Nginx does not have a check function for its upstreams (yet in the open source version - but this seems to come soon).

In order to perform a check on the PHP+MySQL functionality, I placed a HAProxy in between:

HTTP Request -> Nginx
                 | -> tcp/9000 on localhost HAProxy
                                             |------> tcp/9000 on PHP Upstream 1 -> local mysql
                                             |------> tcp/9000 on PHP Upstream 2 -> local mysql
                                             |------> tcp/9000 on PHP Upstream 3 -> local mysql

Now how can I check the functionality of the PHP-FPM upstream servers? For this I installed nginx on the upstream servers, just for the purpose of local monitoring. The .php location (suffixes) are sent to the localhost php-fpm instance.
I wrote a check script in PHP which connects to the local MySQL instance and makes a write and a read operation, just to make sure everything works. If the check script succeeds, it echos "PHP and MySQL up".

Given that I have now nginx and the working check script on the upstream servers, how can I add this check in HAProxy?
A standard HAProxy configuration (which you will see in almost all tutorials) for a backend server looks like this:

listen phpfpm
  balance roundrobin
  mode tcp
  option  tcplog
  server phpupstream1 check
  server phpupstream2 check

The "check" option in the "server" definition just checks if port 9000 is available on the given server. With that alone I'm not able to execute the check script. I need to do my check via http to nginx on the upstream server, therefore I need to use port 80 for the check - but for the check only.
Great softwares offer great solutions. And HAProxy is great and the solution is there: It's simply called "port" and is part of the "server" syntax. It took me a while to figure out that "check" and "port" go together so it ist actually best described as "check port".

By adding "check port" to the server syntax, I am now able to perform checks on port 80 while the application passes the normal requests to port 9000:

listen phpfpm
  balance roundrobin
  mode tcp
  option  tcplog
  server phpupstream1 check port 80
  server phpupstream2 check port 80

But what about my php check script? Here I use httpchk to call the script and await the string "PHP and MySQL up":

listen phpfpm
  balance roundrobin
  mode tcp
  option  tcplog
  option httpchk GET /dbcheck.php HTTP/1.1\r\nHost:\ monitoring\r\nConnection:\ close
  http-check expect string PHP and MySQL up
  server phpupstream1 check port 80
  server phpupstream2 check port 80

Here we go. Now HAProxy does not only check a running tcp port of php-fpm, but rather a full stack of a working php-fpm daemon and mysql server.


Nginx talking to remote php-fpm backend server: No input file specified.
Wednesday - Jun 10th 2015 - by - (0 comments)

Sometimes you get a painful reminder, that you've forgotten the basics.

In a Nginx/PHP-FPM setup I wanted to add a new virtual host with a separate document root:

server {
  listen 80;
  server_name infra;
  error_page   500 502 503 504  /50x.html;
  access_log /var/log/nginx/default.access.log;
  error_log /var/log/nginx/default.error.log;
  root /var/www/default;
  index index.php;
  location = /50x.html {
      root   /usr/share/nginx/html;
  location / {
    try_files $uri $uri/ /index.php;
  # Pass PHP scripts to PHP-FPM
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass phpbackend:9000;
                fastcgi_index index.php;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;

All PHP requests are sent via the fastcgi_pass to an upstream server called "phpbackend". Looks simple enough and should work. But it didnt.

A manual check on the Host name "infra" with curl revealed that the php file info.php (which exists very well in the /var/www/default folder) was not shown/executed in php-fpm:

curl -H "Host: infra" http://mynginxhost.example.com/info.php
No input file specified.

The nginx error log showed the following entry:

2015/06/10 14:47:43 [error] 32148#0: *2302858 FastCGI sent in stderr: "Unable to open primary script: /var/www/default//info.php (No such file or directory)" while reading response header from upstream, client:, server: infra, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://phpbackend:9000", host: "infra"

But as soon as I changed the document root path from "/var/www/default" to "/var/www/phpmyadmin", the info.php (which also exists in the phpmyadmin folder) was viewable. 

I suspected a problem with the file permission, but these were all ok (meaning readable by nginx):

[root@nginx ~]# ll /var/www/
drwxr-xr-x   2 user user 4096  9. Jun 11:11 default
drwxrwxr-x. 10 user  user  4096 19. Mär 11:28 phpmyadmin
drwxrwxr-x.  2 user  user  4096 16. Apr 21:18 util

[root@nginx ~]# ll /var/www/default/
-rw-r--r-- 1 user user 12  5. Jun 16:07 index.html
-rwxrwxr-x 1 user  user  20 19. Mär 11:28 info.php

While I was focusing solely on the nginx server and tried a different set of fastcgi parameters, I finally received an important hint.
I completely forgot that the php files must exist on the php backend (upstream) server!

The file structure on the php backend server revealed, that the default folder (and its files) are missing:

[root@phpbackend ~]# ll /var/www/
drwxrwxr-x. 10 user user 4096 Mar 19 11:27 phpmyadmin
drwxrwxr-x.  2 user user 4096 Apr 16 21:18 util

So the default folder and its php script is missing, from there comes the error Unable to open primary script: /var/www/default//info.php (No such file or directory).
On the backend php server I created the missing folder and the php file:

[root@phpbackend ~]# mkdir -p /var/www/default
[root@phpbackend ~]# echo "" /var/www/default/info.php
[root@phpbackend ~]# chown -R user:user /var/www/default

And now the request worked.

curl -H "Host: infra" http://teaserlb001.nzz.ch/info.php
PHP Version 5.4.16

This will be stuck in my mind now for a couple of years - until I forget it again.


Ubuntu 14.04: mailq command not working in container (Permission denied)
Monday - Jun 8th 2015 - by - (0 comments)

Very strange but hit me today, when I installed a new LXC host, running Ubuntu 14.04 LTS and with the latest patches. 

After having installed an LXC container and added monitoring, the check_mailq plugin didn't work anymore. A manual verification confirmed, that the mailq command could not be executed:

root@container:~# mailq
postqueue: warning: close: Permission denied

The command works on the LXC host itself though. So "only" the mailq command in a container hits that problem.

According to Ubuntu bug #1390223 and to this mailing list post, this seems to be a bug in the recently for Trusty released kernel 3.16. The original trusty kernel (3.13) didn't have this issue (I'd have seen that in previously installed containers). 

Indeed, this recently installed LXC host already includes the newer 3.16 kernel:

root@lxchost001:~# uname -a
Linux shopprodlxc001 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Where does the kernel 3.16 come from?
This is where the big difference from Debian and Ubuntu come in. While Debian sticks with its original package versions (unless someone uses other apt sources or backports), Ubuntu often offers to install newer software packages, for example newer kernels. This has an advantage that new features can be enabled on a LTS system which was installed two years ago, but has the disadvantage of going a bit into "experimental". This is the classical example of the main difference between Debian and Ubuntu.

By checking the installed packages, one can see that the 3.16 kernel is actually the kernel version from Utopic Unicorn (aka Ubuntu 14.10):

root@lxchost001:~# dpkg -l | grep linux-image
ii  linux-image-3.16.0-30-generic       3.16.0-30.40~14.04.1   amd64   Linux kernel image for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-extra-3.16.0-30-generic 3.16.0-30.40~14.04.1   amd64   Linux kernel extra modules for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-generic-lts-utopic           amd64   Generic Linux kernel image

I am actually surprised that Ubuntu installs by default the newer available kernel and not the one intended for the LTS system. An upgrade can always easily be done with "apt-get dist-upgrade" but then the administrator knows about potential risks. If the newer kernel comes automatically on a new installed system, possible bugs can hit your system and you're in the same place as I am right now.

So let's go back to the LTS kernel 3.13 by installing the linux-image-generic-lts-trusty package (after having stopped the containers of course!):

root@lxchost001:~# apt-get install linux-image-generic-lts-trusty
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  linux-image-3.13.0-53-generic linux-image-extra-3.13.0-53-generic
Suggested packages:
  fdutils linux-doc-3.13.0 linux-source-3.13.0 linux-tools
The following NEW packages will be installed:
  linux-image-3.13.0-53-generic linux-image-extra-3.13.0-53-generic
  linux-image-generic linux-image-generic-lts-trusty
0 upgraded, 4 newly installed, 0 to remove and 3 not upgraded.
Need to get 51.9 MB of archives.
After this operation, 194 MB of additional disk space will be used.

When initramfs runs through as post-installation task, it finds two kernel versions:

Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
update-initramfs: Generating /boot/initrd.img-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.16.0-30-generic
Found initrd image: /boot/initrd.img-3.16.0-30-generic
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic

With the current status, there would be a "dual boot" possibility with both 3.13 and 3.16 kernels. I checked /boot/grub/grub.cfg and the defailt is still set to 3.16.
In my case I only want to use 3.13, so let's remove and purge 3.16:

root@lxchost001:~# apt-get remove linux-image-generic-lts-utopic linux-image-3.16.0-30-generic linux-image-extra-3.16.0-30-generic
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be REMOVED:
  linux-generic-lts-utopic linux-image-generic-lts-utopic
0 upgraded, 0 newly installed, 2 to remove and 1 not upgraded.
After this operation, 55.3 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 63127 files and directories currently installed.)
Removing linux-generic-lts-utopic ( ...
Removing linux-image-generic-lts-utopic ( ...
Removing linux-image-3.16.0-30-generic (3.16.0-30.40~14.04.1) ...
WARN: Proceeding with removing running kernel image.
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.16.0-30-generic /boot/vmlinuz-3.16.0-30-generic
update-initramfs: Deleting /boot/initrd.img-3.16.0-30-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.16.0-30-generic /boot/vmlinuz-3.16.0-30-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
The link /vmlinuz.old is a damaged link
Removing symbolic link vmlinuz.old
 you may need to re-run your boot loader[grub]
The link /initrd.img.old is a damaged link
Removing symbolic link initrd.img.old
 you may need to re-run your boot loader[grub]

The warnings are pretty good, so to avoid trying to boot a non-existant kernel, we update the grub configs:

root@lxchost001:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin

The system now needs to be booted with the 3.13 trusty kernel, so reboot:

root@lxchost001:~# reboot

Once the system is back up again, verify what kernel version is now being used:

root@lxchost001:~# uname -a
Linux shopprodlxc002 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@lxchost001:~# dpkg -l | grep linux-image
ii  linux-image-3.13.0-53-generic       3.13.0-53.89                     amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.16.0-30-generic       3.16.0-30.40~14.04.1             amd64        Linux kernel image for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-extra-3.13.0-53-generic 3.13.0-53.89                     amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.16.0-30-generic 3.16.0-30.40~14.04.1             amd64        Linux kernel extra modules for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-generic                            amd64        Generic Linux kernel image
ii  linux-image-generic-lts-trusty                     amd64        Generic Linux kernel image

The 3.16 packages are still on the file system, but are not installed anymore. They can now be purged. But important is, that the 3.13 kernel is now active.

Let's check if the mailq command now works again in the container:

root@container:~# mailq
Mail queue is empty

Yes, it's working again!

And this is why my personal preference is to use Debian systems for stability. But to use the new LXC features Ubuntu is currently the better choice. Like in life, it's always about compromise.


What are the tasks of a DevOp engineer?
Wednesday - May 27th 2015 - by - (0 comments)

I was asked that question today and these are the tasks I understand to be my duty as a DevOp engineer.

- systems administration
- security
- operations and monitoring
- (re-) engineering of infrastructure and system environments
- automation
- scripting/coding
- understanding applications from developers
- make developers understand security and systems
- being the bridge between code and system
- quality assurance
- use Agile methods to build and deploy new systems (we're talking about Continuous Deployments here)
- use centralized/programmable configuration management (Ansible, chef, puppet, ..)
- troubleshoot incidents (performance issues, hack attacks, etc)
- write and review documentations

As mentioned above, a DevOp is the bridge between the code and the systems. This image (found on http://stackify.com/defining-the-ops-in-devops/) shows that part very well:



Go to Homepage home
Linux Howtos how to's
Nagios Plugins nagios plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

8232 Days
until Death of Computers