Another day another hacked website

Yesterday morning, phone rings to notify my of a new sms. Someone could not access his website on some server that I am root/administer.
I tried to ping the server and got 1 reply every 10-15 packets so my initial thought was that the hosting provider had fucked up. I pinged other machines in the “neighborhood”, they replied just fine. So the problem lied in my server. I got console access through IPMI, you know…the ones with the cipher zero bug, and I managed to login. An apache2 process was constantly using 100% of a core and the machine sent gazillion packets towards a certain destination.

Since I wanted to investigate what exactly this process did, I put an iptables entry in my OUTPUT chain to block packets towards that destination. The machine became responsive again, though the apache process still ran at 100%. Since I run my vhosts using apache2 mpm_itk module, I knew through the apache2 PIDs’ username which site had been hacked. I grepped the logs for any POST, but I couldn’t see anything. Unfortunately the logs only go back 2 days (NOT my policy! and a very bad one actually…but anyway).

strace -p PID did not yield anything interesting, just the process trying to create sockets to send packets towards the destination.

socket(PF_NETLINK, SOCK_RAW, 0) = 417
bind(417, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(417, {sa_family=AF_NETLINK, pid=11398, groups=00000000}, [12]) = 0
sendto(417, "\24\0\0\0\26\0\1\3\233\323\354Q\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 588
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\233\323\354Q\206,\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(417) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 417
fcntl(417, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(417, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(417, {sa_family=AF_INET, sin_port=htons(4883), sin_addr=inet_addr("X.Y.Z.W")}, 16) = 0
fcntl(417, F_SETFL, O_RDWR) = 0
sendto(417, "\207\25\312P\322t\0#\317}jf\2(W\374\375\232h\213\220\31\355\277)\320[\255\273\276\221\374"..., 8192, MSG_DONTWAIT, NULL, 0) = -1 EPERM (Operation not permitted)
close(417) = 0

lsof -n -p PID output had hundreds of open log files and a few connections. Grepping out the logs I noticed one that was quite interesting, it went towards another server at port 5555.
apache2 11398 XXXXXXX 416u IPv4 831501972 0t0 TCP A.B.C.D:59210->B.C.D.E:5555 (CLOSE_WAIT)

I run tcpdump there, and of course it was an irc connection. I started capturing everything.

lsof also revealed this:
apache2 11398 XXXXXXX cwd DIR 8,7 4096 2474373 /var/www/vhosts/XXXXXXX/httpdocs/libraries/phpgacl
which I could have have also seen it doing ls /proc/PID/cwd/ …but anyway.

Looking inside that dir I found a file named gacl_db.php. It was base64 encoded. Well actually it was multiple times base64 encoded and obfsuscated by using character substitutions, so I had to de-obfuscate it. It was quite easy using php and some bash scripting.

This is the original base64 encoded/obfuscated file: Original gacl_db.php
This is the final result: Deobfuscated gacl_db.php
(I have removed the irc server details from the deobfuscated file, it’s still there in the original file for whoever wants it though)

It’s just an IRC bot containing a perl reverse shell as well. It has commands to flood other servers, and that’s what my server was doing.

I joined the IRC server and at that time there were more than 90 bots inside. Right now that I’m writing this blog post there are less than 50. Every bot joining the channel outputs a text like this:

[uname!]: FreeBSD a.b.c.d 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #10: Fri Sep 30 14:45:56 MSK 2011 root@a.b.c.d:/path/to/to/to/sth pl#27 amd64 (safe: off)
[vuln!]: http://www.a-vhost-name.TLD/libraries/phpgacl/gacl_db.php
[uname!]: Linux x.y.z.w 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64 (safe: off)
[vuln!]: http://www.another-vhost-name.another-TLD/libraries/phpgacl/gacl_db.php

So if you run servers or websites, do a locate gacl_db.php.

Since all the bot/servers entering post a [vuln!] message about phpgacl, my guess is that the original vulnerability that allowed the attacker to gain access is right there. I haven’t had time to look into it yet, but I’ve warned my clients to remove this library from their websites as a precaution. You should probably do the same.

Stopping Plesk Panel attacks with OSSEC

During the past few weeks I’ve noticed increased brute forcing activity on various servers that I manage and run Plesk Panel. Most of the entries look like this:

189.205.227.115 - - [30/Jan/2011:07:14:19 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:19 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:19 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:21 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:21 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:23 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [30/Jan/2011:07:14:23 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852

The side effect of all these attacks is increased server load.

Since I already have ossec monitoring these servers the solution was quite simple. I just added a couple more rules to ossec in order to stop these attacks.

Two steps are necessary to stop these attacks:
1) Add plesk panel https log to monitor list in /var/ossec/etc/ossec.conf

  <localfile>
    <log_format>apache</log_format>
    <location>/opt/psa/admin/logs/httpsd_access_log</location>
  </localfile> 

  <localfile>
    <log_format>apache</log_format>
    <location>/opt/psa/admin/logs/httpsd_error_log</location>
  </localfile>

2) Create some custom rules to block (and notify me) of these attacks.

<rule id="100144" level="1">
    <if_sid>31100</if_sid>
    <id>200</id>
    <url>/login_up.php3</url>
    <description>Plesk Login.</description>
  </rule>

<rule id="100145" level="12" frequency="3" timeframe="60">
    <if_matched_sid>100144</if_matched_sid>
    <same_source_ip />
    <description>Attack on plesk panel.</description>
    <group>attack,</group>
  </rule>

That’s it. Ossec now monitors these files and blocks through iptables any attacks with active-response.

Example notification mail:

Received From: foo->/opt/psa/admin/logs/httpsd_access_log
Rule: 100146 fired (level 12) -> "Attack on plesk."
Portion of the log(s):

189.205.227.115 - - [02/Feb/2011:20:19:56 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [02/Feb/2011:20:19:55 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852
189.205.227.115 - - [02/Feb/2011:20:19:54 +0100] "GET /login_up.php3?passwd=setup&login_locale=default&login_name=admin HTTP/1.1" 200 5852