my past 2 articles for LinuxInside

Following my first article on the Greek Linux Magazine called LinuxInside about IPv6, I uploaded my past 2 articles for it. Both articles are in Greek of course.

The first one is about debugging network connectivity issues using the command line on Linux. It was published on the 2nd issue of LinuxInside.
Εντοπίζοντας ένα πρόβλημα δικτύωσης

The other one is an introduction to Zsh. It was published on the 3rd issue of LinuxInside.
Κατακτήστε το Z shell

If you haven’t read the magazine already, feel free to download those pdfs and read them.
All my presentations/articles can be found at: Articles/Presentations

0x375 – 0x07 – Security Considerations for a brave new (IPv6) World

I finally had the chance to present something at the Thessaloniki Tech Talk Sessions also known as 0x375. The people over there have done a great job, and I truly mean that, bringing tech people together. Almost once a month 2 speakers can present a tech topic they like at an open auditorium inside the Aristotle University of Thessaloniki. On those events people from Thessaloniki, but also from nearby cities, gather and have a great time, not only during the presentations but afterwards as well. I won’t spoil the events that take place during the tech talks, because you should definitely go if you are curious, but I can tell you that it’s not uncommon for as many as 15 to 20 people to go for beers after the talks!

So, the past Friday (25/11/2011), me and @apoikos traveled from Athens to Thessaloniki to present at 0x375. My presentation was about some security concepts on IPv6 networks, how old attacks of the IPv4 world transform to new ones on the IPv6 world and about some new ones that will appear on local networks sooner or later. I also had prepared some small live demos, but as always it’s very hard to succeed in a live demo if you don’t quite control the environment. At least some of the stuff I wanted to show were successful, and I’m happy with those. (Thanks to Nuclear for booting his OS X guinea pig)

Some apologies…When giving a presentation on IPv6, in an event that has no other introductory IPv6 presentations, I always face the same problem, most people are not very well aware of how different this protocol is from IPv4. When I ask the audience how well do they know IPv6, most people are embarrassed to say they have never actually used it, so the audience stays very, VERY silent. This means that I have to put around 15-20 slides to make a “quick introduction to IPv6”, and this unfortunately takes usually over 30′ of presentation time. Some techy/advanced people might be bored with this, but there’s no other way to overcome this “issue”. If you go straight to the point and start discussing about ND ICMPv6 messages most people won’t be able to keep up…so I’m sorry if I made some of the audience get bored by my first slides. I promise that my next talk on 0x375, cause there will surely be a next one, will be less boring for you 🙂

Thank you all for coming there, I hope you enjoyed it as much as I did!

You can find the slides and my live demo notes here:
0x375 – 0x07 – kargig – Security Considerations for a brave new (IPv6) World (pdf)
0x375 – 0x07 – kargig – Security Considerations for a brave new (IPv6) World – live demo notes (txt)

P.S. I’ve started collecting some interesting (for me) presentations regarding IPv6 topics at void.gr/kargig/ipv6/. Check them out if you like.

GrRBL goes django

I’ve had this thought for some time now, I needed a nice interface for GrRBL so that it would make it easier for others, read more, people to contribute. Many people have been, politely, complaining about lack of features, policy and so on.

Right now most people use either the submission form or they bounce their emails to grrbl [at] void [dot] gr. Then their emails get manually processed, filtered and if everything goes well the “useful” parts of their email end up in the DNS RBL or the email address blacklist. This process is not automated at all, entries are manually added to a database, and is therefore quite time consuming. What’s worse is that people who are listed don’t have an ‘easy’ way to opt-out, apart from emailing us. The algorithm of adding someone to these lists is also not well-defined. The main rule that is followed is that an IP or email address is added to these lists when at least 3 people have submitted them on different days.

Hopefully this is about to change soon (I don’t know how soon, but soon!). During the past month I’ve been trying to code an interface in django, even though I had no prior experience in it. It’s mostly a self educating process and I like it very much. This django application will be generic enough to cover submissions and listings for IPs, emails and possibly URLs.

  • Short term goals:
  • Anonymous users will only get to see details about an IP they search for. People though will be able to register and add their own entries to a database. These registered users will be able to see the complete listings. Each user will belong to a group and every group will have a different weight which will depend on his ‘expertise’ (I know this is broad, but read on). For example, the group of the individual users will certainly have less weight than the group of the postmasters of Greek ISPs (yeap there are some who regularly contribute). Using their weights users will be able to vote on each entry that’s inside the database. Upon a certain score these entries will be flagged as eligible to be on the blacklist. Listed people will be able to opt-out but this process will be moderated by the superusers, that means that spammers like the infamous sofokleous10 will never get a chance to opt-out even for a single second.
    Most of this functionality is already coded and is working quite well.

  • Mid term goals:
  • Various export formats will be supported (BIND/RBLDNSD, Spamassassin/Postifix/Exim/sendmail/etc). Selective/custom export of entries will be provided. Users will be able to select if they want to export/use a strict blacklist, that is hosts that are scored very high, a moderate one and a very broad/risky one. Levels have yet to be defined. An API will be published so that entries can be re-used in other applications (json format ?)

  • Long term goals:
  • A method/interface that someone would copy/paste their email and it would automagically parse it, provide the user with the discovered malicious entries (IP, emails, URLs) and propose him to add them to the database. Maybe automate this even further so that they are added on a separate moderated queue without user interaction, that would be suitable for submitting entries via email plugins for clients such as mutt/thunderbird/etc.

  • The code:
  • The django application code resides in github for now: https://github.com/kargig/grrbl_django. Everyone is welcome to submit ideas (as issues) and code! Feel free to download, test and provide feedback.

  • Greek Adblock Plus Filter
  • Since the code is very flexible I am thinking whether Greek Adblock Plus Filter can also be benefited by this voting system. It probably can, so expect some changes to that list as well. One interface to rule them all.

    Many thanks go to @apoikos who has been helping me a lot with the tons of questions I still have on django stuff.

    Notes on HP raid controllers

    Lately I had to deal with some HP raid controllers and I’ve gathered some notes on them. I’ll post them here so I won’t forget about them.

    First of all, don’t even think on using them without a battery pack. Seriously DON’T. The performance degradation is humongous. Without a battery pack the controllers were giving me 1/20th of the results with a battery pack. If you want to quickly test them, try iozone using the following options: iozone -t4 -I

    Installing hpacucli is a also a must if you want to monitor or configure the controllers from within your OS. Be sure to add the repositories from HWraid to your system and then issue: aptitude install hpacucli (you are using Debian, arent’ you?). That reminds me that I am using those repositories on so many systems I manage that I must send a donation to the people at hwraid to thank them.

    Below are some commands using hpacucli that I used.
    # Show everything about your raid controllers
    # hpacucli controller all show config detail

    Cache Board Present: True
    Cache Status: OK
    Accelerator Ratio: 25% Read / 75% Write
    Drive Write Cache: Enabled
    Total Cache Size: 512 MB
    Battery Pack Count: 1
    Battery Status: OK
    SATA NCQ Supported: True

    What you must take notice here is the Accelerator Ratio, Drive Write Cache and Battery Pack Count.
    if you have a battery pack installed but your Drive Write Cache is still shown as “Disabled”, you can enable it using the command:
    # hpacucli controller slot=X modify dwc=enable
    You’ll know what to put instead of “slot=X” from the output of the previous command (show config detail).

    To modify Accelerator Ratio (read/write):
    # hpacucli controller slot=X modify cacheratio=25/75

    To enable Array Acceleration for one of your logical drives use:
    # hpacucli controller slot=X logicaldrive Y modify aa=enable

    If you happen to face the following error while opening hpacucli, don’t worry. You don’t need to reboot your machine as I’ve seen in various blogs.

    Error: Another instance of ACU is already running (possibly a service). Please
    terminate the ACU application before running the ACU CLI. Press ENTER to
    exit.

    What you need to do is delete the shared IPC that hpacucli left when it got killed for some reason.
    To see all your ipcs:

    # ipcs
    ------ Shared Memory Segments --------
    key        shmid      owner      perms      bytes      nattch     status      
    
    ------ Semaphore Arrays --------
    key        semid      owner      perms      nsems     
    0xffffffff 32768      root       0          1         
    
    ------ Message Queues --------
    key        msqid      owner      perms      used-bytes   messages    
    

    Then use ipcrm to remove the array with the semid you want:
    # ipcrm -s 32768

    and try to start hpacucli again.

    References:
    1. http://www.datadisk.co.uk/html_docs/redhat/hpacucli.htm
    2. http://people.freebsd.org/~jcagle/hpacucli-readme

    Block Greek ads with Internet Explorer 9+

    Greek Adblock Plus filter list has more than 3500 regular unique (as per IP) subscribers daily. The majority of them uses some Mozilla-based browser (Firefox/Iceweasel). An increasing number of users has started using Chrome along with the Chrome Adblock Extension and Opera. Thanks to Panagiotis Dimopoulos, Greek Adblock Plus filter has now been converted to a Tracking Protection List for Internet Explorer 9+.

    To load the protection list visit the url: Greek Adblock Tracking Protection List for Internet Explorer 9+

    For more details about using Greek Adblock Plus filter on various browsers please visit: Greek Adblock Plus Filter.

    Don’t forget to send in comments and URLs to block. If you are using Firefox’s addon, please use the “Report Issue on this page…” feature of it by right clicking on the status icon of adblock.

    Why vacation auto-reply messages can sometimes be bad

    Say that a user has an email account at the company he works for. Before going on vacation he activates his cool “vacation auto-reply” feature that adds

    Out of Office – I will be back from holidays at the end of July.

    on the top and then quotes the email he was sent.

    During his vacation, he receives a call and he is told he has to urgently sent an email about some financial updates. He rushes to an internet cafe and sends the email. He makes a mistake though and mistypes one of the email addresses of the recipients. Instead of sending the email to “user@domain.com” he sends it at “usar@domain.com”.

    His company’s SMTP server though receives the following error message from the remote SMTP server while trying to deliver the email:

    <usar@domain.com>: host mx.domain.com[1.2.3.4] said: 550 5.1.1
       <usar@domain.com>... User unknown (in reply to RCPT TO command)

    This means that his SMTP server will then send an email to him informing him about the error and quoting parts if not all of the email he had previously sent. The email will likely appear to be from “postmaster@company.com” or “do-not-reply@company.com” or something similar.
    It will look like this:

    This is the mail system at host mail.company.com.
    
    I'm sorry to have to inform you that your message could not
    be delivered to one or more recipients. It's attached below.
    
    For further assistance, please send mail to postmaster.
    
    If you do so, please include this problem report. You can
    delete your own text from the attached returned message.
    
                      The mail system
    
    <usar@domain.com>: host mx.domain.com[1.2.3.4] said: 550 5.1.1
       <usar@domain.com>... User unknown (in reply to RCPT TO command)
    Reporting-MTA: dns; mail.company.com
    X-Postfix-Queue-ID: AE4812AE328
    X-Postfix-Sender: rfc822; employee1@company.com
    Arrival-Date: Thu,  5 May 2011 20:05:27 +0200 (CEST)
    
    Final-Recipient: rfc822; usar@domain.com
    Original-Recipient: rfc822;usar@domain.com
    Action: failed
    Status: 5.1.1
    Remote-MTA: dns; mx.domain.com
    Diagnostic-Code: smtp; 550 5.1.1 <usar@domain.com>... User unknown
    
    From: Loyal Employee <employee1@company.com>
    Date: July 5, 2011 9:05:29 PM GMT+03:00
    To: User User <usar@domain.com>
    Subject: Re: Financial updates
    
    Financial data goes here
    

    But the user has still his vacation auto-reply turned on, so when the automatic postmaster’s email reaches his mailbox, the system will automatically reply back to the “postmaster@company.com” quoting the previous email and adding his auto-reply message:

    Out of Office – I will be back from holidays at the end of July.

    So the postmaster@company.com currently has all the financial details that he shouldn’t!

    Apart from the fact that the user was sending financial data to somebody else in a clear text email instead of an encrypted one, the second biggest mistake that the user has made was that he has enabled vacation auto-replies that quote the email he was previously sent. That’s very very wrong. If you don’t want sensitive stuff ending at the postmaster’s inbox avoid quoting previous emails in your auto-replies by all means.

    Based on a true story 🙂

    please use indexes in your database tables

    I don’t know how many times I’ve heard a sysadmin swearing at a web/application developer that has “forgotten” to add proper index to their database tables. Please dear developers, add those indexes, it’s not that hard!

    Today I had to do it on a table for a xoops module. A single query could keep mysql very very busy…how busy ? the query could take more than 2 minutes, unless it was already cached! I’ll post the way I solved it as an example. It is actually usually quite easy to solve these kind of issues.

    1) How to spot the problem
    Your application is slow and your server’s “top” shows mysql running constantly at full load.

    2) Find the problematic query. Login to mysql and run a ‘show processlist;’

    
    mysql> show processlist;
    +------+--------------+-----------+--------------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+
    | Id   | User         | Host      | db                 | Command | Time | State          | Info                                                                                                 |
    +------+--------------+-----------+--------------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+
    | 1792 | admin        | localhost | aaa                | Sleep   |    0 |                | NULL                                                                                                 | 
    | 4790 | user         | localhost | bbb                | Query   |    0 | Sorting result | SELECT * FROM foobar WHERE (com_rootid = '22797' AND com_id >= '22797') ORDER BY com_id              | 
    | 4791 | user         | localhost | bbb                | Query   |    0 | Sorting result | SELECT * FROM foobar WHERE (com_rootid = '22797' AND com_id >= '22797') ORDER BY com_id              | 
    | 4817 | admin        | localhost | NULL               | Query   |    0 | NULL           | show processlist                                                                                     | 
    +------+--------------+-----------+--------------------+---------+------+----------------+------------------------------------------------------------------------------------------------------+
    4 rows in set (0.00 sec)
    

    3) Take a closer look at the structure of the table

    
    mysql> describe foobar;
    +--------------+-----------------------+------+-----+---------+----------------+
    | Field        | Type                  | Null | Key | Default | Extra          |
    +--------------+-----------------------+------+-----+---------+----------------+
    | com_id       | mediumint(8) unsigned | NO   | PRI | NULL    | auto_increment | 
    | com_pid      | mediumint(8) unsigned | NO   | MUL | 0       |                | 
    | com_rootid   | mediumint(8) unsigned | NO   |     | 0       |                | 
    | com_modid    | smallint(5) unsigned  | NO   |     | 0       |                | 
    | com_itemid   | mediumint(8) unsigned | NO   | MUL | 0       |                | 
    | com_icon     | varchar(25)           | NO   |     |         |                | 
    | com_created  | int(10) unsigned      | NO   |     | 0       |                | 
    | com_modified | int(10) unsigned      | NO   |     | 0       |                | 
    | com_uid      | mediumint(8) unsigned | NO   | MUL | 0       |                | 
    | com_ip       | varchar(15)           | NO   |     |         |                | 
    | com_title    | varchar(255)          | NO   | MUL |         |                | 
    | com_text     | text                  | NO   |     | NULL    |                | 
    | com_sig      | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | com_status   | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | com_exparams | varchar(255)          | NO   |     |         |                | 
    | dohtml       | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | dosmiley     | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | doxcode      | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | doimage      | tinyint(1) unsigned   | NO   |     | 0       |                | 
    | dobr         | tinyint(1) unsigned   | NO   |     | 0       |                | 
    +--------------+-----------------------+------+-----+---------+----------------+
    20 rows in set (0.01 sec)
    

    4) run EXPLAIN on the problematic query

    
    mysql> explain SELECT * FROM foobar WHERE (com_rootid = '17536' AND com_id >= '17536') ORDER BY com_id;
    +----+-------------+--------------------+------+-----------------------+------+---------+------+-------+-----------------------------+
    | id | select_type | table              | type | possible_keys         | key  | key_len | ref  | rows  | Extra                       |
    +----+-------------+--------------------+------+-----------------------+------+---------+------+-------+-----------------------------+
    |  1 | SIMPLE      | foobar             | ALL  | PRIMARY               | NULL | NULL    | NULL | 18271 | Using where; Using filesort | 
    +----+-------------+--------------------+------+-----------------------+------+---------+------+-------+-----------------------------+
    1 row in set (0.02 sec)
    

    If “type” is “ALL”, like the case above, then you have a problem. This is the worst “type” you could have. The query above scans the whole table and has to go through 18271 rows and then has to do a filesort on these rows.

    5) Look at indexes, if there are any….

    
    mysql> show index from foobar;
    +--------------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
    | Table              | Non_unique | Key_name   | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
    +--------------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
    | foobar             |          0 | PRIMARY    |            1 | com_id      | A         |       18272 |     NULL | NULL   |      | BTREE      |         | 
    | foobar             |          1 | com_pid    |            1 | com_pid     | A         |           1 |     NULL | NULL   |      | BTREE      |         | 
    | foobar             |          1 | com_itemid |            1 | com_itemid  | A         |          18 |     NULL | NULL   |      | BTREE      |         | 
    | foobar             |          1 | com_uid    |            1 | com_uid     | A         |           6 |     NULL | NULL   |      | BTREE      |         | 
    | foobar             |          1 | com_title  |            1 | com_title   | A         |       18272 |       40 | NULL   |      | BTREE      |         | 
    +--------------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
    5 rows in set (0.00 sec)
    

    6)Create necessary new index

    
    ALTER TABLE  `DB`.`foobar` ADD INDEX `com_id_rootid` (  `com_id` ,  `com_rootid` )
    

    7) Explain the query again

    
    mysql> explain SELECT * FROM foobar WHERE (com_rootid = '17536' AND com_id >= '17536') ORDER BY com_id;
    +----+-------------+--------------------+------+------------------------+---------------+---------+-------+------+-----------------------------+
    | id | select_type | table              | type | possible_keys          | key           | key_len | ref   | rows | Extra                       |
    +----+-------------+--------------------+------+------------------------+---------------+---------+-------+------+-----------------------------+
    |  1 | SIMPLE      | foobar             | ref  | PRIMARY,com_id_rootid  | com_id_rootid | 3       | const |    1 | Using where; Using filesort | 
    +----+-------------+--------------------+------+------------------------+---------------+---------+-------+------+-----------------------------+
    1 row in set (0.00 sec)
    

    now the query is of type “ref” and it only fetches 1 row!
    If the query could be optimized even more by the developer and he/she could replace ‘>=’ with ‘=’ then it would make us even happier

    
    mysql> explain SELECT * FROM foobar WHERE (com_rootid = '17536' AND com_id = '17536') ORDER BY com_id;
    +----+-------------+--------------------+-------+-----------------------+---------+---------+-------+------+-------+
    | id | select_type | table              | type  | possible_keys         | key     | key_len | ref   | rows | Extra |
    +----+-------------+--------------------+-------+-----------------------+---------+---------+-------+------+-------+
    |  1 | SIMPLE      | foobar             | const | PRIMARY,com_id_rootid | PRIMARY | 3       | const |    1 |       | 
    +----+-------------+--------------------+-------+-----------------------+---------+---------+-------+------+-------+
    1 row in set (0.00 sec)
    

    the type would now be ‘const’, which is the best type we could have.

    The problem got fixed with just one command! Now the queries take milliseconds to finish and the web application is as fast as ever. The server load dropped from 2.5 to 0.4 with just this tiny addition to the indexes.

    CPU load drops dramatically after adding the proper index to the table

    CPU Load

    So dear developers, please run EXPLAIN to your queries before you submit your applications and if you find queries with type=ALL try to add indexes! It saves all of us a loooooot of trouble!

    For anyone who wants to read a nice blog post about EXPLAIN and what each part of the output means, I recommend that he/she reads this: MySQL Explain – Reference

    Resolving OSSEC active response iptables issues

    The past few days some of my servers are having difficult times due to the increase of spam by some botnet(s). From around 600-700 emails per day for unknown addresses/recipients on local domains, this number reached a peak of 8.000 emails 2 days ago. In order to reduce further botnet attempts I’m having ossec to engage, which in turn tries to firewall hosts.

    That worked quite ok for a while but then I’ve started seeing errors in the active-response.log like the ones below:

    Unable to run (iptables returning != 3): 1 – /var/ossec/active-response/bin/firewall-drop.sh delete – 91.121.21.8 1310919172.51029 31106
    Unable to run (iptables returning != 1): 1 – /var/ossec/active-response/bin/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
    Unable to run (iptables returning != 1): 2 – /var/ossec/active-response/bin/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
    Unable to run (iptables returning != 1): 3 – /var/ossec/active-response/bin/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
    Unable to run (iptables returning != 1): 4 – /var/ossec/active-response/bin/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
    Unable to run (iptables returning != 1): 5 – /var/ossec/active-response/bin/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
    Unable to run (iptables returning != 4): 1 – /var/ossec/active-response/bin/firewall-drop.sh add – 115.242.188.157 1310969220.1045522 3302

    Obviously iptables is busy doing something else at the time, adding or deleting some other rule, so the loop inside firewall-drop.sh sometimes fails. That was a bit worrying, I had to fix ossec so one way or another so that iptables rules would eventually be applied. I’ve faced the same issue with iptables in the past, trying to simultaneously add multiple (>5) iptables rules at exactly the same time is very error prone, there’s no way to tell which of those rules will be applied. In order to circumvent the issue, I added locking to the active response script.

    Whenever it comes to locking with shell scripts I am using a set of four functions inside a file that I source when I need to. I place this file usually inside /usr/local/bin/ under the lock.sh filename.

    lockme () {
        if [ -z "$1" ];then
            echo " o Use an argument to lock"
            return 1
        fi
        if [ -z "$2" ];then
            PID=$$
        else
            PID=$2
        fi
        LOCK_PID_FILE=/var/lock/$1
        if [ -f $LOCK_PID_FILE ];then
            sleep 1
            echo " o Lock file found"
            if [ ! -d /proc/`cat $LOCK_PID_FILE 2>/dev/null` ];then
                echo " o Stale lock file ignoring..."
                rm -f $LOCK_PID_FILE
            else
                return 1
            fi  
        fi  
        #temp file
        echo -n $PID > $LOCK_PID_FILE.$PID
        ln -s $LOCK_PID_FILE.$PID $LOCK_PID_FILE && return 0
        rm -f $LOCK_PID_FILE.$PID
        return 1
    }
    
    lockme_wait () {
        if [ -z "$1" ];then
            echo " o Use an argument to lock"
            return 1
        fi  
        if [ -z "$2" ];then
            PID=$$
        else
            PID=$2
        fi  
        while [ 1 ];do
            lockme $1 $PID && break
            sleep 4
        done
        return 0
    }
    
    unlockme () {
        if [ -z "$1" ];then
            echo " o Use an argument to unlock"
            return 1
        fi
        #remove pid file
        rm -f /var/lock/$1.`cat /var/lock/$1 2>/dev/null`
        rm -f /var/lock/$1
        return 0
    }   
    
    kill_locked () {
        if [ -z "$1" ];then
            echo " o Use an argument to kill_locked"
            return 1
        fi
        if [ -e /var/lock/$1 ]; then
            kill `cat /var/lock/$1 2>/dev/null`
        fi
        rm -f /var/lock/$1.`cat /var/lock/$1 2>/dev/null`
        rm -f /var/lock/$1
    }
    

    You can also use %s/var\/lock/tmp/g if you prefer having the locks on the /tmp which is usually ramfs, partition.

    Afterwards I edited /var/ossec/active-response/bin/firewall-drop.sh to just add 3 lines. (I only edited the relevant Linux section of the script, since I haven’t tested, or don’t even know if it’s needed on the BSD, SunOS sections, I left those unedited):

  • Add . /usr/bin/lock.sh right after the “# Checking for an IP” section (around line 45)
  • Right after “# Executing and exiting” add lockme_wait active-response (around line 75)
  • Right after the second while loop finishes, after “done” and before “exit 0” add unlockme active-response (around line 110)
  • That’s it…just 3 lines added and the errors have completely stopped since then.

    P.S. Yes, I could have used lockfile-progs to achieve the same result, but I (also) use lock.sh file in embedded systems when needed, and it’s far more portable and easy.

    World IPv6 Day – The Future is Forever

    It’s time!

    Tomorrow is the World IPv6 Day and in order to celebrate it in Athens, we are having an IPv6 Party at hackerspace.gr!
    I’ll do a small introductory presentation about the basics of IPv6 Protocol and how’s Linux doing with it. After the presentation there will be an open discussion regarding IPv6 … drinking beer.

    Everyone’s invited! Be there!

    WORLD IPV6 DAY is 8 June 2011 – The Future is Forever

    End Users IPv6 Training by Hellenic IPv6 Task force

    Today the Hellenic IPv6 Task force organized a training day for end users. Most of the presentations were entry level, since they were targeting end users, but most of them were very interesting. I was happily surprised by Cosmote‘s presentation on their mobile IPv6 tests using native IPv6 and NAT64.

    My presentation was about what is architecturally different inside an IPv6-enabled CPE. I tried to present topics like WAN addressing, LAN addressing, auto-configuration (SLAAC), DHCPv6 as well as some migration/tunneling techniques. There were lots of comments, a lot of interesting questions and I was happy to “provoke” an interesting exchange of ideas/perceptions on IPv4 NAT mainly due to the previous to last slide of my presentation 🙂

    At the end of the event there was a draw and people got 4 IPv6-enabled CPEs produced by the company I work for (Gennet), 4 DSL accounts by OTE ready to use for OTE’s pilot IPv6 program and 5 IPv6 related books generously provided by Cisco.

    My presentation: Defining an IPv6-ready CPE
    All presentations: IPv6 End Users Training Day (2011)

    On Friday(s) 27/05 and 03/06 I’ll be giving two more IPv6 presentations at University of Pireus: [oss-unipi] Event #26: Introduction to TCP/IP and IPv6. The first one will be introductory and the second one will be more technical and maybe workshop based. In the same day(s) there will also be in-depth technical IPv4 presentations by Apollon Oikonomopoulos. Be there if you like technical presentations!

    Fosswar 2011 – How we did it

    Intro
    As said in my previous post about Fosscomm 2011, during the conference there was a wargame consisting of 5 challenges. We played in a team consisting mainly of the following members: manji/manjiki, trelo_mpifteki, mickflemm and me. Along with us was maisk mainly acting as our manager (!!) shouting at us when we did something he did not like. He was of course a great help (sometimes :P).

    Few words about the team
    As most people who regularly read this blog already know, I consider myself a sysadmin even though I currently work at a company that produces Linux based xDSL routers where I do most of the things required about networking and system interaction like QoS, VoIP, IPv6, firewall, etc. My coding is confined among the realms of scripting languages. Manji is also a sysadmin who has lately started to mess with VoIP. Trelo_mpifteki is mostly a java developer and so is maisk. mickflemm is a coder and a very good one, frequently messing with Linux kernel’s wireless driver support. Obviously, we are certainly not the typical wargame players…

    The challenges
    Be sure to download the challenges from: http://patras.fosscomm.gr/fosswar/

    As soon as the wargame was on we split the challenges among us. Since mickflemm was the only coder he started to mess with challenge number 5 (reverse engineering). Me and trelo_mpifteki started to look at challenge number 1 (networking) while manji started poking at challenge number 3 (networking).

    Challenge 1:
    Type: Networking
    Players: kargig, trelo_mpifteki
    The first challenge said just this:

    Connect to the elite port and find the secret message.

    Everyone knows that elite = 31337
    So we just did a nc X.Y.Z.W 31337 (where X.Y.Z.W is the IP address of the server) and the following message appeared:

    IP protocol = 1
    Timestamp
    id = 1337
    seq = 0xCAFE
    orig = 0xDEAD

    As it is easily understood one must create a packet, the problem is what kind of packet. And then was when I did a huge mistake stalling my team for more than 10 mins by insisting that Protocol 1 is IP. So we started trying to create an IP packet using scapy. Obviously we hit lots of problems because IP’s header clearly misses most of the above options, especially timestamp. Our next idea was to create a TCP packet in order to embed some of the options. This also lead to a dead end. The I looked again at my /etc/protocols file and I saw that Protocol 1 is ICMP. Timestamp is type 13, code 0 and the rest are just options. So our solution was this:

    >>> packet_2= IP(dst='X.Y.Z.W')/ICMP(type=13, id=1337, seq=0xcafe, ts_ori=0xdead)
    >>> send(packet_2)

    Using Wireshark we captured the response packet which looked like this (this setup is on my box atm):

    One can see the message: feedadeadface in it.

    Hint:
    If you need to compile listenicmp.c yourself you have to do something like this:

     # aptitude install libpcap-dev
    # gcc -o listenicmp listenicmp.c -lpcap

    Challenge 2:
    Type: Steganography
    Players: kargig,trelo_mpifteki,maisk
    For this challenge the organizers gave us a hint what we needed to find was close to the end of the image.jpg and after “BAADF00D”.

    # hd image.jpg | tail -n 4
    000152a0  d4 4d 77 22 b9 9a 68 ba  ad f0 0d 78 9c 0b c9 c8  |.Mw"..h....x....|
    000152b0  2c 56 00 a2 44 85 e2 d4  e4 a2 d4 12 85 dc d4 e2  |,V..D...........|
    000152c0  e2 c4 f4 54 3d 00 72 da  08 ef                    |...T=.r...|
    000152ca
    

    so we found “ba ad f0 0d” and the next characters were “78 9c”. We googled those and came up with the result that it was the magic of zlib compression. So what we had to do was get the rest of the file after “ba ad f0 0d” and then uncompress that. To get the rest of the file we found the size of the file and subtracted the bytes (31) that were of interest to us:

    # du -b image.jpg
    86730  image.jpg
    # split -b 86699 image.jpg koko
    # mv kokoab final
    

    Then we run python to decompress “final” file.

    >>> import zlib
    >>> ourfile=file('final')
    >>> ourfile
    <open file 'final', mode 'r' at 0xb7473020>
    >>> chunk=ourfile.read()
    >>> chunk
    'x\x9c\x0b\xc9\xc8,V\x00\xa2D\x85\xe2\xd4\xe4\xa2\xd4\x12\x85\xdc\xd4\xe2\xe2\xc4\xf4T=\x00r\xda\x08\xef'
    >>> zlib.decompress(chunk)
    'This is a secret message.'
    

    Challenge 3:
    Type: Networking
    Players: manji,maisk,kargig

    While me and trelo_mpifteki were trying to solve challenges 1 and 2, manji was looking into challenge 3 pcap file for weird things. Once again we were given a hint that we needed to look at strange headers. Finding strange headers in a 800+ kb pcap file is not an easy task though. At a point manji was looking for very big sequence numbers…and then we got another hint, that we shouldn’t look at those big numbers at all. When me and trelo_mpifteki finished the other two challenges we started looking at Wireshark.
    We applied the following filter:
    (ip.host == 64.22.109.100) && ((tcp.seq == 0) || (tcp.seq ==1))
    And the we had the following results in front of us:

    Since we knew that the message was sent to “64.22.109.100” we needed to look at packets originating from “192.168.1.3”. The first thing I noticed were packets with strange TTLs, they were going up and down..so I made a guess that the secret message could be hidden there. Transforming those TTL values to ASCII was a dead end. Then we started to look closer at every packet that 192.168.1.3 sent to 64.22.109.100 and we grouped them by destination port. There was clearly something going on with destination port 58900. A careful eye will also notice that packets towards port 58900 don’t have an MSS set while others mostly do. So, we expanded our filter with packets that also had destination port 58900:
    ((ip.host == 64.22.109.100) && ((tcp.seq == 0) || (tcp.seq ==1))) && (tcp.dstport == 58900)
    and came up with this:

    If you look closely at the selected packet from Wireshark, you’ll see that the sequence number while set to 0 (zero) contains the letter ‘r’ inside it. The next packet contained the letter ‘o’ and the next one the letter ‘t’. Writing all these letters down we had this sequence:

    r
    o
    t

    q
    r
    n
    q
    o
    r
    r
    s

    That was a rot13 encrypted message! with google’s help we found a rot13 decryptor. The decrypted message was:
    D E A D B E E F

    That’s it! we had 3 out of 5 while no other team had more than 2. So we had wooooon! Congrats to everyone on our team!

    Fosscomm 2011 – My review

    Generic comments
    I just got back home from Fosscomm 2011 and I must admit it has been one of the best organized events of this kind I’ve seen in Greece ever. The single most important fact was that presentations and workshops were always _on time_. They started on time, they finished on time. The organizers had to face even a power cut by the national energy company but they still managed not to fall behind on schedule. My only remark would be about the selection of the presentations that took place in the big room (called BA). Most of them gathered far less people than other presentations which took place in smaller rooms (B3 for example) and those rooms got extremely crowded from time to time. Maybe the organizers thought that generic open source presentations would attract more people than the technical ones but, unfortunately for them, and fortunately for “the greater good”, they were very wrong. This doesn’t reduce their achievement though. Another thing I would like to see on the next Fosscomm is less material given out to participants and instead spend this money on paying for travel expenses of people coming to speak on Fosscomm from abroad. Giving one (or even more) of the phones that HTC kindly provided to the voted by the participants best talk/presentation/workshop would also be very nice. My sincere congratulations to the organizing committee.

    My IPv6 workshop
    On my application to Fosscomm I had asked for one presentation and a separate workshop. This unfortunately wasn’t accepted, probably due to the large amount of other presentations/workshops, so I was given the first workshop on the very first day of Fosscomm, about “Using IPv6”- on Linux. Since I only had one hour, my original plans were to do a quick 15 min introductory presentation on IPv6 and then a 45 min hands-on lab. Since University of Patras could not provide IPv6 connectivity to the Lab, my other option was to have some remote VMs that would have upstream IPv6 connectivity and people on the Lab would ssh to. These VMs were kindly provided to us by Grnet and I have to publicly thank them one more time. My planning was bad though, people had far more questions about introductory IPv6 issues than I expected and the intro presentation was not finished until more than 35 mins had passed. This lead to the unfortunate result that the workshop could not be completed as I had planned. I am glad though that almost everyone logged into the VMs and had the chance to at least set up an IPv6 IP manually as well as an IPv6 default route. They also had their first ping6s. Some got even further by setting up ip6tables rules…What I definitely wanted to have shown during the workshop, and I didn’t have enough time to, was auto-configuration (SLAAC) which I consider to be one of the most intriguing features of IPv6. Next time I am doing either a presentation or a hands-on workshop, definitely not both in one session. Lesson learned.

    Presentation/Workshop material:
    Quick Intro to IPv6
    Using IPv6 on Linux (workshop notes)

    Talks I attended
    All the talks I attended were very interesting, I probably did a very good job picking them 🙂 The ones I attended were:
    Saturday
    a) “A unified user account manager using LDAP/KRB5/CIFS” by Giorgos Mamalakis, Chariton Karamitas
    b) “Network Exploitation with Ncrack” by Fotis Chatzis
    c) “Wargames” by Patroklos Argyroudis, Fotis Chatzis, Chariton Karamitas
    Sunday
    a) “Intro to Arduino” by Pierros Papadeas, Kostas Antonakoglou
    b) “Φωνή VoIPόντως εν τη ερήμω” by Effie Mouzeli
    c) “Bright side of the Darknets” by Athanasios Kostopoulos
    d) “Automated Testing Framework” by Giorgos Keramidas
    e) “Performing Digital Forensics with Open Source tools” by Dimitrios Glynos
    f) “Patras Heap Massacre” by Chariton Karamitas, Patroklos Argyroudis

    I consider the most well presented one being: “Performing Digital Forensics with Open Source tools” while the one with the best slides was definitely: “Patras Heap Massacre”. If you weren’t there to see them, definitely try and find at least the slides if not the video.

    Wargames
    I had never participated before in any wargames because I considered that to be out of my league. That’s the reason I had not even registered for Fosscomm’s wargame called “fosswar”. I was curious though about what it’s like, so me and 3-4 more friends decided to stand in line and enter the room among the other people. The Lab room was crowded, really crowded. At first I thought that one reason for this would be the prize for the winner, an Android phone donated by HTC. I was later proven wrong though. Fosswar started and the organizers presented us with the 5 challenges. Two of those had to do something with networking…so I said to my self that I would be lucky even if I understood what one of those asked. Another challenge had to do with steganography, another one with exploiting a vulnerability in C and the last one was about reverse engineering. While this game could only have one winner, since me and my friends were not actually interested about the prize we decided to work as a team and see whether we can solve anything. Me and a friend looked at the first challenge, another 1 was looking at the third while 2 more were each looking at the fourth and fifth. During the wargame HTC asked for the names of people that wanted to take part in the competition, there nobody actually wanted to give his name. Most of them were there “just for the kicks”. That was even more exciting! HTC then told us that if nobody wanted to give his name the phone would be given on a lottery…so 10-15 people decided to give our names so the phone would “stay” at the fosswar. After solving the first challenge we got so excited that we just had to try the others, we couldn’t give up. So, after 3 hours of thinking and coding, the result was that our team managed to solve 3 of the five challenges, the details on what the challenges were and how we dealt with them will follow in a separate blog post, and we ended up being the winners because no other team had solved more than two. Yes, we had won! We had managed to solve the two networking challenges and the steganography one. That was soooo unexpected!

    Pictures
    My pictures from Fosscomm 2011 are at: https://picasaweb.google.com/kargig/2011050720110508Fosscomm2011#

    My conclusion
    Fosscomm this year definitely showed a trend. People don’t need generic presentations about open source any more, they know what it is, they believe in it, they use it. People really ask for technical presentations, and we need more of them.

    Whoever couldn’t manage to attend Fosscomm 2011, should definitely attend Fosscomm 2012. I am certainly looking forward to it!

    IPv6 workshop at Fosscomm 2011

    This weekend, 7-8 May 2011, I’ll be doing a workshop about “IPv6 usage on Linux” at Fosscomm 2011 conference. Fosscomm is the biggest Greek Open Source conference with lots of interesting presentations, workshops and events. In the spirit of enabling IPv6 wherever possible, I am very happy to have managed to enable IPv6 connectivity for the website of Fosscomm this year!

    During the workshop I will try to show the necessary tools to use, debug network problems and configure various services for IPv6. My goal is to explain to participants the abolishment of NAT, introduce the concepts of SLAAC and DHCPv6 and show them how to configure their networks to effectively use IPv6. If there’s enough time some issues related to IPv6 security will also be discussed. Target audience is people who are not afraid of the command line, have basic networking skills and are eager to learn things regarding IPv6!

    Unfortunately it was impossible to have IPv6 upstream connectivity at the LAB, so participants will be testing everything on an IPv6 enabled remote infrastructure. Due to the limited number of LAB computers and Virtual Machines available, only 35, there is a registration form for participants to fill in. Please take the time to fill in the form any specific topics that you would like the workshop to pay attention to.

    The workshop is taking place on Saturday 07/05/2011 and it will start _exactly_ at 10:45 in order to take full advantage of the time given to us by Fosscomm’s organizers.

    Workshop is supported by the Greek IPv6 Taskforce and there miiiight be IPv6-related material distributed to the participants.

    Please don’t forget to fill in the registration form!!!

    Block Spam with russian encoding using spamassassin

    Lately the amount of spam with russian encoding/charset that I was getting had increased significantly. Spamassassin’s configuration options “ok_locales” and “ok_languages” were not enough because I didn’t want to whitelist some language, but I just wanted to blacklist some.

    So the solution for my problems was the addition of the following lines in the configuration of spamassassin:

    header LOCAL_CHARSET_RUSSIAN Subject:raw =~/\=\?koi8-r\?/i
    score LOCAL_CHARSET_RUSSIAN 7
    describe LOCAL_CHARSET_RUSSIAN Contains russian charset that is not acceptable

    If you want to add even more charsets:

    header LOCAL_CHARSET_BLOCKED Subject:raw =~/\=\?(koi8-r|windows-1251)\?/i
    score LOCAL_CHARSET_BLOCKED 5
    describe LOCAL_CHARSET_BLOCKED Contains charsets that are not acceptable

    Be VERY careful with these rules if you place them in the global config (/etc/spamassassin/local.cf) because if any of your users are getting emails in russian those emails will be probably marked as spam!!

    ΟΣΕ και μυστική φράση κλειδί

    Χρησιμοποιώντας πριν 2 μέρες το νέο σύστημα ηλεκτρονικής κράτησης εισητηρίων του ΟΣΕ αντιμετώπισα το εξής παράδοξο. Μετά την επικύρωση της πιστωτικής μου κάρτας και αφού είχα δώσει και τα ονόματα των συνεπιβατών μου, το σύστημα μου ζητούσε να δώσω μια “μυστική φράση-κλειδί” που θα την ήξερα μόνο εγώ και ο ελεγκτής των εισητηρίων. Αδυνατούσα να καταλάβω που χρειάζεται αυτό και έτσι την επόμενη μέρα το απόγευμα (στις 18:00) έστειλα ένα email στην διεύθυνση που αναφέρει το website ζητώντας πληροφορίες για το κλειδί.

    Το email μου:

    Καλησπέρα

    καταρχήν συγχαρητήρια για το σύστημα online κράτησης εισητηρίων. Ήταν κάτι που χρειαζόταν εδώ και χρόνια.

    Έχω μια απορία όμως, ποιά είναι η λογική πίσω από την χρήση “μυστικής φράσης-κλειδί” ? Εφόσον το εισητήριό μου αναφέρει επάνω τον κωδικό του, έχει το ονομά μου και θα έχω και εγώ ταυτότητα επάνω μου…τι επιπλέον προσθέτει στην ασφάλεια του ΟΣΕ αυτή η φράση ?

    Με τιμή,

    Στις 21:50 (!!!!) την ίδια μέρα πήρα την εξής απάντηση:

    Καλησπέρα,

    Η μυστική φράση έχει να κάνει με τις περιπτώσεις όπου θέλετε να κάνετε δώρο το εισιτήριο ή το βγάζετε για λογαριασμό κάποιου τρίτου, ο οποίος προφανώς δεν θα φέρει μαζί του την πιστωτική κάρτα με την οποία εκδόθηκε ο τίτλος. Και δεδομένου ότι Ελλάδα είμαστε και δεν είναι όλοι οργανωμένοι να κουβαλάνε μαζί τους ταυτότητες κλπ, εισάγαμε την ιδέα του μυστικού κωδικού σαν ένα επιπρόσθετο μέτρο για να μπορείτε να εξασφαλίσετε το εισιτήριο σας και να αποδείξετε ότι είστε ο πραγματικός του κάτοχος.

    Κατ’ αυτό τον τρόπο μπορείτε να εξασφαλίσετε την αγορά σας σε περίπτωση που κάποιος αντιγράψει το εισιτήριο σας ή το τυπώσει (πράγμα καθόλου δύσκολο *) και ανεβεί στο τρένο προσποιούμενος ότι είστε εσείς. Στην περίπτωση αυτή αν και οι δύο δεν φέρετε ταυτότητα ή πιστωτική κάρτα, μόνο ο μυστικός κωδικός μπορεί να ξεδιαλύνει την κατάσταση, ο οποίος κωδικός βρίσκεται τυπωμένος μόνο στην λίστα που έχει ο ελεγκτής.

    * το παρόν σύστημα έχει σχεδιαστεί με την λογική ότι κάποιοι και θα τυπώσουν εισιτήρια 3ων πέρα από τα δικά τους και θα προσπαθήσουν να τα χρησιμοποιήσουν, και ότι κάποιοι άλλοι θα τα φωτοτυπήσουν / Photoshop-άρουν. Οπότε το “τί πληροφορία είναι μοιρασμένη σε ποιο έγγραφο” είναι πολύ προσεκτικά σχεδιασμένο.

    Σε κάθε περίπτωση, η εισαγωγή του “μυστικού κλειδιού” είναι προαιρετική για την αγορά του εισιτηρίου.

    Με εκτίμηση,

    Δεν θα σχολιάσω την εξήγηση που πήρα, αλλά το γεγονός πως κάποιος από το δημόσιο τομέα μου απάντησε σε email την ίδια μέρα και μάλιστα βράδυ. Εκπληκτικό!