Into.the.Void.

Around a year ago I wrote a blog post about how me and @zmousm scaled our streaming infrastructure at GRNET NOC so that we could cope with a sudden demand on the streaming service that we provide to the Greek Parliament. That setup was re-used again in January 2013 (Lagarde-list discussion) where we managed to surpass our previous record of 1.66Gbps reaching 1.79Gbps. We knew that the previous solution could definitely be improved though. Wowza does not seem to scale very well in our environment(*cough* java *cough*), so we modified our setup quite a bit.

What we did was take the original Wowza streamer, and ‘hide’ it behind two different categories of ‘proxy-servers’ that clients communicate with. The first category is made of three varnish proxies sitting at two different datacenters. All clients that fetch HTTP streams communicate only with the varnish proxies and not with the original streamer. Varnish uses very few resources and scales wonderfully. Then we added an nginx-rtmp server to offload RTMP clients from the original streamer. Now all RTMP clients communicate with nginx-rtmp only. We’ve also notified website owners that prefer using our RTMP stream to serve it through their own (flash) applications to switch to the nginx-rtmp endpoint. This means that the original Wowza streamer now mainly serves the three varnish proxies and the nginx-rtmp server as ‘clients’, and since the VM now has far less load, the stream it provides to the ‘proxy-servers’ doesn’t get ‘chopped’ from time to time, as it did previously when it served hundreds of clients.
While each wowza streamer previously needed 6Gb of RAM to serve around 500-600Mbit of traffic, varnish needs <1Gb and can easily serve 900Mbit. Our nginx-rtmp server also uses <1Gb of RAM. So we’re actually using fewer resources to serve more (happier) clients!

This setup gives us a lot of flexibility and extensibility. We can easily scale it horizontally when we want to just by adding more varnish or nginx-rtmp servers.

With this setup we were able to achieve 3.55Gbps and serve more than 6000 clients last Sunday (10/11/2013), that’s double of our previous record!

Here are the graphs:

Filed by kargig at 10:10 under Greek,Internet,Networking
Tags: Greek, grnet, HTTP, nginx, nginx-rtmp, parliament, rtmp, scaling, streaming, varnish, wowza
12 Comments | 14,204 views

Some months ago I started playing with the idea of creating a world map that would have every Tor node on it. Obviously I wan’t the first one…I soon discovered Moritz Bartl’s post on the same topic. Luckilly he had his code posted on Github so I could fork it and add features that I wanted. The original python script parsed the consensus and the misrodescriptors, put Tor nodes into some classes and created a KML file with some description on each node.

Some differences
I changed some parts of the python script to better suit my needs.
a. Create a separate kml files for each Tor node class.
b. Add new classes: Bad, Authority and Named.
c. Pay more attention on requesting every external URL over HTTPS.
d. Generate HTML code that displays those KMLs on a Google Maps overlay.
e. Add some small randomization to each nodes’s coordinates so that nodes in the same city don’t overlap.

You can find a complete changelog at kargig/tormap GitHub repo.

And here’s the outcome: World city map of Tor nodes at https://tormap.void.gr/
One of my main goals was to have selectable classes of nodes that will appear on the map.

To produce the map overlay, a cron script runs every hour, which is also the period it takes for Tor Authority nodes to produce a new consensus, and creates some static files which are then served by nginx.

I’m not a web developer/designer and I don’t really know any javascript. So please, feel free to fork my code and make it look better, run faster and add your own features. I’ll happily accept patches/pull requests!

Extras
On kargig/tormap repo you will also find a handy script, ‘runme.sh’, that downloads all necessary files that need to be parsed by the python script.

Missplaced nodes on the map
Well, blame MaxMind’s GeoIP City database for that. But I think it’s kinda funny to see Tor nodes in Siberia and in the middle of the sea though (look at the West coast of Africa), heh. For those wondering, these nodes are gathered there because their geoip Lat,Long is set to 0,0.
Really though, what’s “Ben’s Cat Shaque” diplayed there next to all those nodes in the west coast of Africa? Anyone has some clue ?

Conspriracy people
I’m sure that people who love conspriracy theories will start posting about those ‘Bad’ Tor nodes in Iran and Syria. Why do you think these are there ? What does it mean ? Let the flames begin!

Future TODO
a. OpenStreetMap
I have started working on an OpenStreetMap implementation of the above using OpenLayers. The biggest hurdle is that OSM does not provide a server that serves map tiles over HTTPS. Makes me wonder…is that actually so difficult ?
b. More stats
I would like to add small graphs on how the number of nodes in each class evolves.

Other Tor mapping efforts
https://b.kentbackman.com/2010/10/04/view-tor-exit-nodes-in-google-earth/
http://freehaven.net/~ioerror/maps/v3-tormap.html

Don’t forget, you can always help Tor by running a node/bridge or sending some money to Tor or EFF!

Filed by kargig at 01:55 under Encryption,Internet,Linux,Networking,Privacy
Tags: Authority, Bad, consensus, debian, geoip, geoipcity, Google maps, html, javascript, map, mapping, maxmind, nginx, OpenStreemMap, python, tor, tormap, tormap.void.gr
1 Comment | 22,589 views