{"id":2154,"date":"2020-04-12T19:30:17","date_gmt":"2020-04-12T16:30:17","guid":{"rendered":"http:\/\/www.void.gr\/kargig\/blog\/?p=2154"},"modified":"2020-04-25T21:14:47","modified_gmt":"2020-04-25T18:14:47","slug":"graphing-covid-19-related-data-for-greece-using-elastic-elasticsearch-service","status":"publish","type":"post","link":"https:\/\/www.void.gr\/kargig\/blog\/2020\/04\/12\/graphing-covid-19-related-data-for-greece-using-elastic-elasticsearch-service\/","title":{"rendered":"Graphing Covid-19 related data for Greece using Elastic’s ElasticSearch Service"},"content":{"rendered":"\n

TL;DR<\/h2>\n\n\n\n

https:\/\/covid19.void.gr<\/a><\/p>\n\n\n\n

<\/p>\n\n\n\n

Intro<\/h2>\n\n\n\n

A few weeks ago I started converting the Covid-19 daily reports from Greek National Public Health Organization (EODY)<\/a> to json documents, and publishing them to Github in a repository called covid19-gr-json<\/a>. Reasoning behind this effort was that EODY does not publish their data in a reusable format, one that “machines” can easily read and parse. There are many websites that have created their own Covid-19 graphs but most of them have very basic stats (cases\/day, deaths\/day) and they state that their data sources are coming from media sources, without letting you access and re-use their original data. I haven’t been able to locate any other open data set regarding Covid-19 data for Greece that is not based on media sources but from official data coming out of EODY.<\/p>\n\n\n\n

Report frequency and data consistency<\/h2>\n\n\n\n

After a few days of trying to convert these daily reports to json, EODY stopped publishing them. 3 days later EODY published another report with a different set of metrics than the previous reports. This new report did not have a per region analysis of cases and EODY also stopped publishing data for the number of hospitalized people or how many people have recovered. 4 more days passed without a report. Then EODY started publishing daily reports again, we’re now at 9 days in a row, but the metrics are again different than the previous ones published. There’s now a drill down by age and gender regarding cases, deaths and people in intensive care (IC). Still no mention of how many people have recovered though or how many are hospitalized but are not in IC. The result is that json files in the repository cannot be consistent since the original sources are not consistent. Such is life.<\/p>\n\n\n\n

Visualizing data<\/h2>\n\n\n\n

A few days had passed since I started converting pdf reports to json and I wanted to see how easy it is to visualize some of it. Since data were in json format, the easiest thing for me to do was to store them to an Elasticsearch cluster and create visualizations using Kibana. Working at Elastic has its perks, I created a new Elastic stack deployment in ElasticSearch Service (ESS)<\/a>, and posted the json documents to it. I had to make a tough decision though, which graph tool from Kibana to use to visualize data. I started using line graphs<\/a> but soon switched to Time Series Visual Builder (TSVB)<\/a> as it’s easier to use and allows to visualize multiple metrics\/time series in the same graph. I believe it’s producing way more beautiful visualizations anyway. After creating some visualizations I also created a dashboard, took some screenshots and shared them with friends.<\/p>\n\n\n\n

Allowing access to dashboards<\/h2>\n\n\n\n

As people took notice of the json files in the repo they started pinging me to create some visualizations from them. I created a Kibana readonly user and shared the password with a few of them. That worked well for a couple of days, but sharing a password for a readonly view is a bit meeeh. I wondered if I could create a password-less version of it and post the URL publicly. Publishing a URL in the form of https:\/\/abc123fe48752c2da3b19.eu-central-1.aws.cloud.es.io:9243<\/a> is not exactly memorable either. I googled if it’s possible to post credentials to Kibana, and indeed it is. So nginx to the rescue!<\/p>\n\n\n\n

Reverse proxy to ESS<\/h2>\n\n\n\n

I first tried to use a URL within void.gr<\/code>, eg void.gr\/covid19<\/code> to point directly to Kibana hosted on ESS. Because of the various location<\/code> rules already used in my void.gr<\/code> nginx config this quickly became more complicated than it had to, so I created a new subdomain for it, covid19.void.gr<\/code>. Here’s the reverse proxy config:<\/p>\n\n\n\n

    location \/ {\n      proxy_pass https:\/\/abc123fe48752997c2da3b19.eu-central-1.aws.cloud.es.io:9243;\n      proxy_http_version 1.1;\n      proxy_set_header Upgrade $http_upgrade;\n      proxy_set_header Connection 'upgrade';\n      proxy_set_header Host $proxy_host;\n      proxy_cache_bypass $http_upgrade;\n      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP\n      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;\n      proxy_set_header  X-Forwarded-Proto $scheme;\n      auth_basic "Basic Auth";\n      proxy_set_header Authorization "Basic bXl1c2VyOm15cGFzcw=="; # base64-encoded username:password to pass in header\n      proxy_set_header x-forwarded-user $remote_user;\n      proxy_hide_header kbn-license-sig;\n      proxy_hide_header kbn-xpack-sig;\n      proxy_hide_header kbn-name;\n      proxy_hide_header x-found-handling-server;\n      proxy_hide_header x-found-handling-instance;\n      proxy_hide_header x-found-handling-cluster;\n      proxy_hide_header x-cloud-request-id;\n    }<\/code2><\/pre><\/pre>\n\n\n\n
Authorization header<\/h3>\n\n\n\n
Authentication is based on the base64 encoded output of username:password<\/code> combination which is to be posted to Kibana.  Create it via:<\/p>\n\n\n\n
$ echo -n myuser:mypass | base64\nbXl1c2VyOm15cGFzcw==<\/code2><\/pre><\/pre>\n\n\n\n
Adding these 2 config options is enough for nginx reverse proxy to authenticate to Kibana:<\/p>\n\n\n\n
auth_basic "Basic Auth";\nproxy_set_header Authorization "Basic bXl1c2VyOm15cGFzcw==";<\/code2><\/pre><\/pre>\n\n\n\n
Header removal<\/h3>\n\n\n\n
While not strictly necessary, I think it’s a good practice to remove non-essential headers being sent back to the browser via the reverse proxy.<\/p>\n\n\n\n
End Result<\/h2>\n\n\n\n
One Let’s Encrypt certificate later and tada! https:\/\/covid19.void.gr<\/a> is live!<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Contribution<\/h2>\n\n\n\n
Do you want to contribute either by converting EODY’s reports to json or by improving the above visualizations dashboards ? leave a comment, send an email<\/a> or ping me via twitter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
TL;DR https:\/\/covid19.void.gr Intro A few weeks ago I started converting the Covid-19 daily reports from Greek National Public Health Organization (EODY) to json documents, and publishing them to Github in a repository called covid19-gr-json. Reasoning behind this effort was that EODY does not publish their data in a reusable format, one that “machines” can easily […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"footnotes":""},"categories":[1,11],"tags":[625,624,628,630,627,631,626,632,629],"class_list":["post-2154","post","type-post","status-publish","format-standard","hentry","category-general","category-greek","tag-covid-19","tag-covid19","tag-data","tag-elastic","tag-eody","tag-ess","tag-greece","tag-tsvb","tag-visualizations"],"aioseo_notices":[],"views":11389,"_links":{"self":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/2154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/comments?post=2154"}],"version-history":[{"count":30,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/2154\/revisions"}],"predecessor-version":[{"id":2190,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/2154\/revisions\/2190"}],"wp:attachment":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/media?parent=2154"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/categories?post=2154"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/tags?post=2154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}