{"id":463,"date":"2009-05-02T17:00:32","date_gmt":"2009-05-02T14:00:32","guid":{"rendered":"http:\/\/www.void.gr\/kargig\/blog\/?p=463"},"modified":"2009-05-02T17:00:32","modified_gmt":"2009-05-02T14:00:32","slug":"some-statistics-on-linux-greek-users-mailing-list-and-forumhelluggr","status":"publish","type":"post","link":"https:\/\/www.void.gr\/kargig\/blog\/2009\/05\/02\/some-statistics-on-linux-greek-users-mailing-list-and-forumhelluggr\/","title":{"rendered":"Some statistics on linux-greek-users mailing list and forum.hellug.gr"},"content":{"rendered":"

Out of boredom I decided to parse the Linux-Greek-Users (LGU) archives and create some graphs. Then I wrote a few more oneliners to deduct some numbers out of the archives. These numbers may or may not mean anything to someone, it’s entirely up to the reader.. Since the archives contain some amount of spam (not too much though) one must take that into consideration as well while reading the numbers I extracted below…<\/p>\n

First thing I did was to download the index file containing the links to the monthly archives since February 1997:
\nwget http:\/\/lists.hellug.gr\/pipermail\/linux-greek-users\/<\/code><\/p>\n

Then download each month’s archive:
\nfor i in `grep date index.html | cut -d\"\\\"\" -f2`; do foo=`echo $i|cut -d\"\/\" -f1`; wget http:\/\/lists.hellug.gr\/pipermail\/linux-greek-users\/$i -O $foo-date.html ; done<\/code>
\n
\nAfter a while I had the full archives at my disk…
\nThen it was time for the first metric.
\nHow many posts does each month have ?<\/strong>
\nThis was quite easy because mailman contains this information inside each month’s archive:
\nfor i in *-date.html; do count=`grep -i \"Messages:\" $i | cut -d\">\" -f 3|cut -d\"< \" -f1`; echo \"$i $count\" >> count.txt;done<\/code>
\nThat command gave me an output such as:<\/p>\n

1999-May-date.html 985
\n1999-November-date.html 1441
\n1999-October-date.html 1148
\n1999-September-date.html 1369
\n2000-April-date.html 690
\n2000-August-date.html 354
\n2000-December-date.html 444
\n2000-February-date.html 833
\n…<\/p><\/blockquote>\n

It didn’t take me a long time before I had this data entered inside OOcalc. Time for the first graph:
\nLGU posts per month:
\n\"lgu-posts_per_month\"<\/a><\/p>\n

And the average monthly posts per year:
\n\"lgu-monthly_average_posts_per_year\"<\/a><\/p>\n

One can certainly argue there’s a tendency here. 2008 had the fewer posts than any other year…<\/p>\n

What’s more interesting is who are actually writing on this list<\/strong>. To extract that information made the following oneliners:
\nFirst of all I converted all html files to utf8 because the archives are kept at iso-8859-7 encoding and I am using a unicode terminal:
\nfor i in *-date.html; do iconv -f iso-8859-7 -t utf-8 $i -o $i.utf; done<\/code>
\nThen I could find all the authors for each monthly archive:
\nfor i in *-date.html.utf; do grep \"\" $i | cut -d\">\" -f 2 >$i-authors.txt; done<\/i><\/code>
\nThen count each author’s posts:
\nfor i in *-date.html.utf-authors.txt; do sort $i | uniq -c | sort -n >$i-sorted-count; done<\/code>
\nEasiest thing to do is to count how many people post per month<\/strong>. Are the users of the mailing list increasing or decreasing ?
\nfor i in *-sorted-count; do wc -l $i >> posters-per-month; done<\/code>
\nThat gave a nice output like that:<\/p>\n

55 1997-April-date.html.utf-authors.txt-sorted-count
\n49 1997-August-date.html.utf-authors.txt-sorted-count
\n88 1997-December-date.html.utf-authors.txt-sorted-count
\n62 1997-February-date.html.utf-authors.txt-sorted-count
\n41 1997-July-date.html.utf-authors.txt-sorted-count
\n60 1997-June-date.html.utf-authors.txt-sorted-count
\n62 1997-March-date.html.utf-authors.txt-sorted-count
\n69 1997-May-date.html.utf-authors.txt-sorted-count
\n…\n<\/p><\/blockquote>\n

I inserted that to OOcalc again and here is the output:
\n\"lgu-posters_per_month1\"<\/a><\/p>\n

The decline in the number of posters is clearly shown.<\/p>\n

Another interesting statistic would be the months with the most and fewer posters respectively:
\nMonths with most posters<\/strong>:
\nsort -n posters-per-month | tail -n10<\/code><\/p>\n

166 1999-May-date.html.utf-authors.txt-sorted-count
\n167 2001-October-date.html.utf-authors.txt-sorted-count
\n168 1999-December-date.html.utf-authors.txt-sorted-count
\n173 1999-October-date.html.utf-authors.txt-sorted-count
\n173 2001-June-date.html.utf-authors.txt-sorted-count
\n174 2001-May-date.html.utf-authors.txt-sorted-count
\n176 1999-September-date.html.utf-authors.txt-sorted-count
\n184 1999-November-date.html.utf-authors.txt-sorted-count
\n192 2000-March-date.html.utf-authors.txt-sorted-count
\n222 2004-November-date.html.utf-authors.txt-sorted-count<\/p><\/blockquote>\n

Months with fewer posters<\/strong>:
\nsort -n posters-per-month | head -n10<\/code><\/p>\n

16 2008-August-date.html.utf-authors.txt-sorted-count
\n32 2008-December-date.html.utf-authors.txt-sorted-count
\n36 2008-July-date.html.utf-authors.txt-sorted-count
\n41 1997-July-date.html.utf-authors.txt-sorted-count
\n44 2008-May-date.html.utf-authors.txt-sorted-count
\n49 1997-August-date.html.utf-authors.txt-sorted-count
\n51 2008-September-date.html.utf-authors.txt-sorted-count
\n51 2009-March-date.html.utf-authors.txt-sorted-count
\n53 2003-August-date.html.utf-authors.txt-sorted-count
\n53 2009-February-date.html.utf-authors.txt-sorted-count<\/p><\/blockquote>\n

Then I got interested in finding out who are the top posters throughout the archives<\/strong>. I didn’t want to write anything complex in order to sum every user’s post and I decided to find out who are the top5 posters in each month and the see whose names are repeated over and over in the top5.<\/p>\n

Using the following oneliner I was able to find the top5 posters of each month:
\nfor i in *-count; do tail -n5 $i >> top5-of-each-month.txt; done<\/code><\/p>\n

And then using a bit of perl I was able to extract the posters appearing most times, here’s the top 20 of them:<\/p>\n

8 Alexios Chouchoulas
\n 9 fs
\n 9 Spiros Bolis
\n 10 Alexandros Papadopoulos
\n 10 Giannis Stoilis
\n 10 Harris Kosmidhs
\n 10 Vasilis Vasaitis
\n 11 Giannis Papadopoulos
\n 11 Michael Iatrou
\n 11 Panos Katsaloulis
\n 11 \u0386\u03b3\u03b3\u03b5\u03bb\u03bf\u03c2 \u039f\u03b9\u03ba\u03bf\u03bd\u03bf\u03bc\u03cc\u03c0\u03bf\u03c5\u03bb\u03bf\u03c2
\n 16 George Notaras
\n 17 Nick Demou
\n 24 George Daflidis-Kotsis
\n 24 Michalis Kabrianis
\n 33 I.Ioannou
\n 41 DJ Art
\n 52 V13
\n 74 Christos Ricudis
\n 83 Giorgos Keramidas<\/p><\/blockquote>\n

The number appearing before the name is the number of months the poster has been inside the top5 posters for a month. That means that Giorgos Keramidas is in the top5 posters for 83 months out of 147 months of archives that I parsed. Pretty impressive!<\/p>\n

Then I wanted to create some graphs about http:\/\/forum.hellug.gr<\/a> as well. HELLUG’s forum has less than 2 years of life so the graphs cannot really be compared to the ones from LGU. I just put them here for completeness.<\/p>\n

Forum.hellug.gr – Posts per month<\/strong>:
\n\"forumhelluggr-posts_per_month\"<\/a><\/p>\n

Forum.hellug.gr – New members per month<\/strong>:
\n\"forumhelluggr-new_members_per_month\"<\/a><\/p>\n

I won’t do any further comments…I’d be glad to see yours though \ud83d\ude42<\/p>\n

P.S. I know that I could have used some form of database to store and process the results of those commands but I wanted to keep it as simple as possible. <\/p>\n","protected":false},"excerpt":{"rendered":"

Out of boredom I decided to parse the Linux-Greek-Users (LGU) archives and create some graphs. Then I wrote a few more oneliners to deduct some numbers out of the archives. These numbers may or may not mean anything to someone, it’s entirely up to the reader.. Since the archives contain some amount of spam (not […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"footnotes":""},"categories":[3],"tags":[65,68,63,66,595,67,47,64],"class_list":["post-463","post","type-post","status-publish","format-standard","hentry","category-linux","tag-forum","tag-forumhelluggr","tag-hellug","tag-lgu","tag-linux","tag-linux-greek-users","tag-oneliner","tag-statistics"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t