resolv.conf options rotate and discovery of ISP DNS issue

Lately I somehow bumped on the manpage of resolv.conf. While reading it I saw the following really nice option:

rotate               sets  RES_ROTATE  in _res.options, which causes round robin selection of name‐
                     servers from among those listed.  This has the effect of spreading  the  query
                     load  among  all  listed servers, rather than having all clients try the first
                     listed server first every time.

Since then my /etc/resolv.conf on both Gentoo and Debian looks like that:
nameserver 194.177.210.10
nameserver 194.177.210.210
nameserver 194.177.210.211
options rotate

(I prefer using GrNET‘s DNS servers than any others in Greece, especially for my laptop configuration. Since they allow recursion I can use them to avoid lousy DNS services provided by lousy DSL routers regardless of the ISP I am currently using, when I am “mobile” with my laptop.)

While using the following config I issued a ping command on a teminal and a tcpdump command on another to see what was actually happening. The result looked like this:
root@lola:~# tcpdump -ni eth1 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
11:20:46.405694 IP 192.168.1.65.55154 > 194.177.210.210.53: 39212+ A? ntua.gr. (25)
11:20:46.444266 IP 194.177.210.210.53 > 192.168.1.65.55154: 39212* 1/5/8 A 147.102.222.210 (319)
11:20:46.484490 IP 192.168.1.65.56152 > 194.177.210.211.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.584171 IP 194.177.210.211.53 > 192.168.1.65.56152: 50452 ServFail 0/0/0 (46)
11:20:46.584449 IP 192.168.1.65.58597 > 194.177.210.10.53: 50452+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:46.624179 IP 194.177.210.10.53 > 192.168.1.65.58597: 50452 1/7/6 (357)
11:20:47.484420 IP 192.168.1.65.32818 > 194.177.210.10.53: 33179+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:47.524176 IP 194.177.210.10.53 > 192.168.1.65.32818: 33179 1/7/6 (357)
11:20:48.484483 IP 192.168.1.65.57670 > 194.177.210.210.53: 21949+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:48.524184 IP 194.177.210.210.53 > 192.168.1.65.57670: 21949 1/3/6 (271)
11:20:49.487610 IP 192.168.1.65.48966 > 194.177.210.211.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.534204 IP 194.177.210.211.53 > 192.168.1.65.48966: 8619 ServFail 0/0/0 (46)
11:20:49.534429 IP 192.168.1.65.49421 > 194.177.210.10.53: 8619+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:49.574138 IP 194.177.210.10.53 > 192.168.1.65.49421: 8619 1/7/6 (357)
11:20:50.494537 IP 192.168.1.65.52525 > 194.177.210.10.53: 3415+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:50.534145 IP 194.177.210.10.53 > 192.168.1.65.52525: 3415 1/7/6 (357)
11:20:51.494552 IP 192.168.1.65.40400 > 194.177.210.210.53: 4504+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:51.534205 IP 194.177.210.210.53 > 192.168.1.65.40400: 4504 1/3/6 (271)
11:20:52.494554 IP 192.168.1.65.42385 > 194.177.210.211.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.544197 IP 194.177.210.211.53 > 192.168.1.65.42385: 48450 ServFail 0/0/0 (46)
11:20:52.544409 IP 192.168.1.65.43773 > 194.177.210.10.53: 48450+ PTR? 210.222.102.147.in-addr.arpa. (46)
11:20:52.584232 IP 194.177.210.10.53 > 192.168.1.65.43773: 48450 1/7/6 (357)

People who are used to reading tcpdump output will immediately point out the ServFail entries of the log. Server 194.177.210.211 refused to provide proper results for the PTR query of 210.222.102.147.in-addr.arpa.

Further investigation of the problem:

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.210
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 66841 IN  PTR achilles.noc.ntua.gr.

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.211
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR

root@lola:~# dig ptr 210.222.102.147.in-addr.arpa @194.177.210.10
;; QUESTION SECTION:
;210.222.102.147.in-addr.arpa.  IN  PTR
;; ANSWER SECTION:
210.222.102.147.in-addr.arpa. 86115 IN  PTR achilles.noc.ntua.gr.

It was obvious that 2 out of 3 DNS servers responded as they should and the other did not.

What I did was to notify a friend working as an administrator there (GrNET) and let him know of the problem. After some investigation, he later on told me that the problem was related to dnssec issues. Possibly a configuration error on RIPE‘s side. As far as I know they had to temporarily disable dnssec on the 147.102 zone…I am not aware whether they fixed the problem (using dnssec) yet though.

I am really glad they acted as fast as possible regarding the solution of the problem 🙂