For a couple of weeks now, a shared web server was continuously listed in the Spamhaus blacklist. After checking the mail logs on that server, no particular spamming could be identified.
I've already seen spamming scripts in the past, which created their own smtp connection (with fsockopen or whatever function), therefore bypassing the local mailserver. That was probably what was happening here too, but the difficult thing was to catch the spammer in the act! After multiple re-listings in the blacklist, I decided to launch a small script which writes the current smtp connections seen with lsof to a logfile. And this is the goal of this script:
#!/bin/sh
##########################
# lsofsmtp.sh
# purpose: output lsof in a logfile to verify it later
echo $(date) >> /var/log/netstatout.log
/usr/sbin/lsof -i -n | grep smtp >> /var/log/netstatout.log
echo "---------------------" >> /var/log/netstatout.log
exit 0
I started a tail -f on the logfile and of course during the whole day nothing appeared.
cat /var/log/netstatout.log
Mon May 5 17:29:01 CEST 2014
master 3213 root 11u IPv4 8863 0t0 TCP 127.0.0.1:smtp (LISTEN)
---------------------
Mon May 5 17:30:01 CEST 2014
master 3213 root 11u IPv4 8863 0t0 TCP 127.0.0.1:smtp (LISTEN)
---------------------
Mon May 5 17:31:01 CEST 2014
master 3213 root 11u IPv4 8863 0t0 TCP 127.0.0.1:smtp (LISTEN)
---------------------
[...]
---------------------
Sat May 10 05:04:01 CEST 2014
master 3213 root 11u IPv4 8863 0t0 TCP 127.0.0.1:smtp (LISTEN)
[...]
This went on and on, so I eventually forgot about the running script... until the server was blacklisted again.
I immediately verified the logfile and finally, on May 12th, the trap worked:
Mon May 12 15:38:01 CEST 2014
master 3213 root 11u IPv4 8863 0t0 TCP 127.0.0.1:smtp (LISTEN)
perl 15208 httpd 510u IPv4 60129149 0t0 TCP 192.168.55.55:56861->63.250.192.45:smtp (ESTABLISHED)
perl 15209 httpd 510u IPv4 60129254 0t0 TCP 192.168.55.55:52481->63.250.192.45:smtp (ESTABLISHED)
perl 15211 httpd 510u IPv4 60129038 0t0 TCP 192.168.55.55:49858->17.172.34.71:smtp (ESTABLISHED)
perl 15212 httpd 510u IPv4 60129390 0t0 TCP 192.168.55.55:33054->98.138.112.35:smtp (SYN_SENT)
perl 15213 httpd 510u IPv4 60129313 0t0 TCP 192.168.55.55:44641->65.54.188.110:smtp (ESTABLISHED)
perl 15214 httpd 510u IPv4 60129331 0t0 TCP 192.168.55.55:41362->74.125.136.27:smtp (ESTABLISHED)
perl 15215 httpd 510u IPv4 60129326 0t0 TCP 192.168.55.55:46223->64.12.88.131:smtp (ESTABLISHED)
perl 15216 httpd 510u IPv4 60129206 0t0 TCP 192.168.55.55:35967->106.10.166.52:smtp (ESTABLISHED)
perl 15217 httpd 510u IPv4 60129375 0t0 TCP 192.168.55.55:42439->65.54.188.110:smtp (ESTABLISHED)
perl 15218 httpd 510u IPv4 60129282 0t0 TCP 192.168.55.55:49330->200.94.181.6:smtp (ESTABLISHED)
No big surprise - a perl script was doing the direct smtp connections, as I expected. As I couldn't use the mail logs to identify the source, I needed to analyze all web server access logs for this particular minute (more or less).
First I created a list of all log files being changed in the last 120 minutes:
find /var/www/*/logs/ -mmin -120 > /tmp/loglist
Then I grepped through all the found logs for one 15:37 to see what has caused the smtp connections:
for logfile in $(cat /tmp/loglist | grep -v referrer | grep -v error | grep -v agent); do echo $logfile; grep "12/May/2014:15:37" $logfile; echo "------------------------------------------------------"; done
Of course there were a few hundred entries, but each virtual host separated by a long line of dashes (making it optically easier to go through). It took me a few seconds until I found the following entries:
46.165.204.8 - - [12/May/2014:15:37:26 +0200] "GET /images.php HTTP/1.1" 200 22 "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"
46.165.204.8 - - [12/May/2014:15:37:26 +0200] "GET /images.php HTTP/1.1" 200 3 "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"
46.165.204.8 - - [12/May/2014:15:37:26 +0200] "GET /images.php HTTP/1.1" 200 3 "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"
46.165.204.8 - - [12/May/2014:15:37:27 +0200] "GET /images.php HTTP/1.1" 200 3 "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"
46.165.204.8 - - [12/May/2014:15:37:28 +0200] "GET /images.php HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"
Additionally to the name "images.php" another information immediately told me that this must be it: Googlebot's IP starts with 66.249... and not with a 46 ipv4 address...
I checked out images.php, a file already uploaded months ago, and I was surprised. Very surprised. It was completely scrambled with nonsense and started like this:
<?php for($o=0,$e='&\'()*+,-.:]^_`{|,,,|-((.(*,|)`)&(_(*,+)`(
.. and ended like this:
$d='';@ord($e[$o]);$o++){if($o<16){$h[$e[$o]]=$o;}else{$d.=@chr(($h[$e[$o]]<<4)+($h[$e[++$o]]));}}eval($d); ?>
After some investigation it turned out, that this uploaded file was the same as this one: https://gist.github.com/laacz/5387180
It would've been interesting to find out how this file was uploaded but unfortunately this web server only kept a few weeks of rotated logs.
The affected web site was using a non-updated CMS (WebSite X5 Evolution) which most likely had a vulnerability and therefore an invitation for a hacker.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder