The following story is inspired by a recent case I had to troubleshoot at work. I think it is a nice example of troubleshooting Linux networking issues, so I’ve modified\/simplified the setup a bit to be able to reproduce it on a VM. I’ll go through the troubleshooting steps in almost the same way we handled the actual case. Service names, IPs, ports, etc are all different that the real case as the focus should not be the example itself but the process.<\/p>\n\n\n\n
It all started a few days ago when I was asked to help on an “unusual” case. Docker containers on every single host of an installation could not establish connections towards services that listen on the “main” IP of the host they run on, nor can they ping that IP, but the containers have full access to the internet and can connect to the service ports on other hosts in the LAN. As everyone who has done even a tiny bit of support, asking whether something changed recently in the setup is always replied back with a single global truth: “nothing has recently changed, it just stopped working”.
Challenge accepted!
<\/p>\n\n\n\n
For reproduction purposes I’ve used a VM with one ethernet interface, and a docker bridge. In this VM I have injected the same problem as with the real case. Even though the real case case was a bit more complicated, to make following the post somewhat easier, I’ve used only one service listening on the host, an Elasticsearch process, and only one Kibana docker container that needs to communicate with Elasticsearch on the host.<\/p>\n\n\n\n
Host interfaces:\n\n2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000\n link\/ether 06:ce:3b:94:fe:ac brd ff:ff:ff:ff:ff:ff\n inet 172.31.45.100\/20 brd 172.31.47.255 scope global dynamic ens5\n valid_lft 3572sec preferred_lft 3572sec\n inet6 fe80::4ce:3bff:fe94:feac\/64 scope link \n valid_lft forever preferred_lft forever\n3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default \n link\/ether 02:42:e9:38:3d:a8 brd ff:ff:ff:ff:ff:ff\n inet 172.17.0.1\/16 scope global docker0\n valid_lft forever preferred_lft forever\n inet6 fe80::42:e9ff:fe38:3da8\/64 scope link \n valid_lft forever preferred_lft forever<\/code2><\/pre><\/pre>\n\n\n\n Kibana’s config has the following ENV variable set ELASTICSEARCH_HOSTS=http:\/\/172.31.45.100:9200<\/a>, and for simplification purposes let’s assume that this IP could not be changed.<\/p>\n\n\n\n
As originally described, curl from the container towards the service IP:port does not work<\/p>\n\n\n\n
(container) bash-4.2$ curl -v 172.31.45.100:9200\n* About to connect() to 172.31.45.100 port 9200 (#0)\n* Trying 172.31.45.100...<\/code2><\/pre><\/pre>\n\n\n\n it just hangs there without error. There’s no DNS resolution involved here, straight curl towards the IP:port<\/p>\n\n\n\n
Let’s check if the service is actually listening on the host<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# ss -ltnp | grep 9200\nLISTEN 0 128 [::]:9200 [::]:* users:(("java",pid=17892,fd=257))<\/code2><\/pre><\/pre>\n\n\n\n <\/p>\n\n\n\n
The service listens on 9200. Since the service listens on all interfaces, let’s curl from the container towards the service IP:port on the docker0 interface.<\/p>\n\n\n\n
<\/p>\n\n\n\n
bash-4.2$ curl -v 172.17.0.1:9200\n* About to connect() to 172.17.0.1 port 9200 (#0)\n* Trying 172.17.0.1...\n* Connected to 172.17.0.1 (172.17.0.1) port 9200 (#0)\n> GET \/ HTTP\/1.1\n> User-Agent: curl\/7.29.0\n> Host: 172.17.0.1:9200\n> Accept: *\/*\n> \n< HTTP\/1.1 200 OK\n< content-type: application\/json; charset=UTF-8\n< content-length: 524\n< \n{\n "name" : "node1",\n "cluster_name" : "centos7",\n "cluster_uuid" : "d6fBSua6Q9OvSu534roTpA",\n "version" : {\n "number" : "7.7.0",\n "build_flavor" : "default",\n "build_type" : "rpm",\n "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",\n "build_date" : "2020-05-12T02:01:37.602180Z",\n "build_snapshot" : false,\n "lucene_version" : "8.5.1",\n "minimum_wire_compatibility_version" : "6.8.0",\n "minimum_index_compatibility_version" : "6.0.0-beta1"\n },\n "tagline" : "You Know, for Search"\n}<\/code2><\/pre><\/pre>\n\n\n\n that works, so the service is running properly. Curl-ing the service from the host using the host’s IP also works<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# curl http:\/\/172.31.45.100:9200\n{\n "name" : "node1",\n "cluster_name" : "centos7",\n "cluster_uuid" : "d6fBSua6Q9OvSu534roTpA",\n "version" : {\n "number" : "7.7.0",\n "build_flavor" : "default",\n "build_type" : "rpm",\n "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",\n "build_date" : "2020-05-12T02:01:37.602180Z",\n "build_snapshot" : false,\n "lucene_version" : "8.5.1",\n "minimum_wire_compatibility_version" : "6.8.0",\n "minimum_index_compatibility_version" : "6.0.0-beta1"\n },\n "tagline" : "You Know, for Search"\n}<\/code2><\/pre><\/pre>\n\n\n\n Let’s check for internet connectivity from the container<\/p>\n\n\n\n
bash-4.2$ curl -v 1.1.1.1\n* About to connect() to 1.1.1.1 port 80 (#0)\n* Trying 1.1.1.1...\n* Connected to 1.1.1.1 (1.1.1.1) port 80 (#0)\n> GET \/ HTTP\/1.1\n> User-Agent: curl\/7.29.0\n> Host: 1.1.1.1\n> Accept: *\/*\n> \n< HTTP\/1.1 301 Moved Permanently\n< Date: Sun, 31 May 2020 10:10:27 GMT\n< Content-Type: text\/html\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Location: https:\/\/1.1.1.1\/\n< Served-In-Seconds: 0.000\n< CF-Cache-Status: HIT\n< Age: 5334\n< Expires: Sun, 31 May 2020 14:10:27 GMT\n< Cache-Control: public, max-age=14400\n< cf-request-id: 030bcf27a3000018e57f0f5200000001\n< Server: cloudflare\n< CF-RAY: 59bfe7b90b7518e5-FRA\n< \n<html>\n<head><title>301 Moved Permanently<\/title><\/head>\n<body bgcolor="white">\n<center><h1>301 Moved Permanently<\/h1><\/center>\n<hr><center>cloudflare-lb<\/center>\n<\/body>\n<\/html><\/code2><\/pre><\/pre>\n\n\n\n internet connectivity for the container works just fine. Let’s curl to another host in the same LAN on the same service port.<\/p>\n\n\n\n
bash-4.2$ curl -v 172.31.45.101:9200\n* About to connect() to 172.31.45.101 port 9200 (#0)\n* Trying 172.31.45.101...\n* Connected to 172.31.45.101 (172.31.45.101) port 9200 (#0)\n> GET \/ HTTP\/1.1\n> User-Agent: curl\/7.29.0\n> Host: 172.31.45.101:9200\n> Accept: *\/*\n> \n< HTTP\/1.1 200 OK\n< content-type: application\/json; charset=UTF-8\n< content-length: 524\n< \n{\n "name" : "node2",\n "cluster_name" : "centos7",\n "cluster_uuid" : "d6fBSua6Q9OvSu534roTpA",\n "version" : {\n "number" : "7.7.0",\n "build_flavor" : "default",\n "build_type" : "rpm",\n "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",\n "build_date" : "2020-05-12T02:01:37.602180Z",\n "build_snapshot" : false,\n "lucene_version" : "8.5.1",\n "minimum_wire_compatibility_version" : "6.8.0",\n "minimum_index_compatibility_version" : "6.0.0-beta1"\n },\n "tagline" : "You Know, for Search"\n}<\/code2><\/pre><\/pre>\n\n\n\n That also works. Time to use the swiss army knife of network troubleshooting, tcpdump. If you want to find which veth interface a container uses you can either use dockerveth<\/a> or use the following commands to figure it out manually.
Get the iflink of container’s eth0:<\/p>\n\n\n\n[root@ip-172-31-45-100 ~]# docker exec -it <container-name> bash -c 'cat \/sys\/class\/net\/eth0\/iflink'<\/code2><\/pre>\n\n\n\n In this case that would be:<\/p>\n\n\n\n
# docker exec -it kibana bash -c 'cat \/sys\/class\/net\/eth0\/iflink'\n41<\/code2><\/pre><\/pre>\n\n\n\n then find the file name of the ifindex that contains that link number in `\/sys\/class\/net\/veth*\/ifindex` of the host<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# grep -lw 41 \/sys\/class\/net\/veth*\/ifindex\n\/sys\/class\/net\/veth0006ca6\/ifindex<\/code2><\/pre><\/pre>\n\n\n\n `veth0006ca6` is what we need to use. Let’s run tcpdump on it<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# tcpdump -nni veth0006ca6\n10:06:16.745143 IP 172.17.0.2.47166 > 172.31.45.100.9200: Flags [S], seq 1062649548, win 29200, options [mss 1460,sackOK,TS val 1781316 ecr 0,nop,wscale 7], length 0\n10:06:16.749126 IP 172.17.0.2.47168 > 172.31.45.100.9200: Flags [S], seq 4174345004, win 29200, options [mss 1460,sackOK,TS val 1781320 ecr 0,nop,wscale 7], length 0\n10:06:16.749131 IP 172.17.0.2.47170 > 172.31.45.100.9200: Flags [S], seq 1386880792, win 29200, options [mss 1460,sackOK,TS val 1781320 ecr 0,nop,wscale 7], length 0<\/code2><\/pre><\/pre>\n\n\n\n the syn packet is seen going out of the container’s veth interface. So let’s tcpdump on docker0<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# tcpdump -nni docker0\n10:07:07.813153 IP 172.17.0.2.47148 > 172.31.45.100.9200: Flags [S], seq 4114480937, win 29200, options [mss 1460,sackOK,TS val 1396384 ecr 0,nop,wscale 7], length 0\n10:07:07.845141 IP 172.17.0.2.47144 > 172.31.45.100.9200: Flags [S], seq 3273546229, win 29200, options [mss 1460,sackOK,TS val 1412416 ecr 0,nop,wscale 7], length 0\n10:07:07.845147 IP 172.17.0.2.47146 > 172.31.45.100.9200: Flags [S], seq 2062214864, win 29200, options [mss 1460,sackOK,TS val 1412416 ecr 0,nop,wscale 7], length 0<\/code2><\/pre><\/pre>\n\n\n\n the syn packet can also be seen on the docker0 bridge. The syn packet cannot be seen on the interface (ens5) that has the service IP (172.31.45.100) on it, since it doesn’t traverse that link to go outside the host.<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# tcpdump -nni ens5 port 9200 or icmp\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes<\/code2><\/pre><\/pre>\n\n\n\n Let’s check routing entries.<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# ip route ls\ndefault via 172.31.32.1 dev ens5 \n172.17.0.0\/16 dev docker0 proto kernel scope link src 172.17.0.1 \n172.31.32.0\/20 dev ens5 proto kernel scope link src 172.31.45.100 <\/code2><\/pre><\/pre>\n\n\n\n Nothing interesting here at all. Time to check iptables.<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# iptables -nxvL\nChain INPUT (policy ACCEPT 169 packets, 27524 bytes)\n pkts bytes target prot opt in out source destination \n\nChain FORWARD (policy DROP 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n 368 27870 DOCKER-ISOLATION all -- * * 0.0.0.0\/0 0.0.0.0\/0 \n 184 14254 DOCKER all -- * docker0 0.0.0.0\/0 0.0.0.0\/0 \n 184 14254 ACCEPT all -- * docker0 0.0.0.0\/0 0.0.0.0\/0 ctstate RELATED,ESTABLISHED\n 184 13616 ACCEPT all -- docker0 !docker0 0.0.0.0\/0 0.0.0.0\/0 \n 0 0 ACCEPT all -- docker0 docker0 0.0.0.0\/0 0.0.0.0\/0 \n\nChain OUTPUT (policy ACCEPT 109 packets, 10788 bytes)\n pkts bytes target prot opt in out source destination \n\nChain DOCKER (1 references)\n pkts bytes target prot opt in out source destination \n\nChain DOCKER-ISOLATION (1 references)\n pkts bytes target prot opt in out source destination \n 368 27870 RETURN all -- * * 0.0.0.0\/0 0.0.0.0\/0 \n\n[root@ip-172-31-45-100 ~]# iptables -nxvL -t nat\nChain PREROUTING (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n 204 12200 DOCKER all -- * * 0.0.0.0\/0 0.0.0.0\/0 ADDRTYPE match dst-type LOCAL\n\nChain INPUT (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n\nChain OUTPUT (policy ACCEPT 6 packets, 456 bytes)\n pkts bytes target prot opt in out source destination \n 0 0 DOCKER all -- * * 0.0.0.0\/0 !127.0.0.0\/8 ADDRTYPE match dst-type LOCAL\n\nChain POSTROUTING (policy ACCEPT 6 packets, 456 bytes)\n pkts bytes target prot opt in out source destination \n 92 6808 MASQUERADE all -- * !docker0 172.17.0.0\/16 0.0.0.0\/0 \n\nChain DOCKER (2 references)\n pkts bytes target prot opt in out source destination \n 191 11460 RETURN all -- docker0 * 0.0.0.0\/0 0.0.0.0\/0 \n\n[root@ip-172-31-45-100 ~]# iptables -nxvL -t mangle\nChain PREROUTING (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n\nChain INPUT (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n\nChain FORWARD (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n\nChain OUTPUT (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination \n\nChain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)\n pkts bytes target prot opt in out source destination <\/code2><\/pre><\/pre>\n\n\n\n There’s not even a DROP rule at all and all the policies are set to ACCEPT. iptables is definitely not dropping the connection. Even if there was a DROP rule, we would see the packet on tcpdump…so where’s the packet going ?<\/p>\n\n\n\n
Let’s add an extra rule for both FORWARD and INPUT chains just to see if iptables can match these rules as the packets are passing by.<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# iptables -I INPUT -p tcp --dport 9200\n[root@ip-172-31-45-100 ~]# iptables -I FORWARD -p tcp --dport 9200<\/code2><\/pre><\/pre>\n\n\n\n wait for a while and then check the statistics of those 2 rules:<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# iptables -nxvL | grep 9200\n 0 0 tcp -- * * 0.0.0.0\/0 0.0.0.0\/0 tcp dpt:9200\n 0 0 tcp -- * * 0.0.0.0\/0 0.0.0.0\/0 tcp dpt:9200<\/code2><\/pre><\/pre>\n\n\n\n no packets match these 2 rules at all! Time to inspect the container and the docker bridge network.<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# docker network inspect bridge\n[\n {\n "Name": "bridge",\n "Id": "a6290df54ea24d14faa8d003d17802b3f8a4967680bc0c82c1211ab75d1815e2",\n "Created": "2020-05-31T09:40:19.81958733Z",\n "Scope": "local",\n "Driver": "bridge",\n "EnableIPv6": false,\n "IPAM": {\n "Driver": "default",\n "Options": null,\n "Config": [\n {\n "Subnet": "172.17.0.0\/16"\n }\n ]\n },\n "Internal": false,\n "Attachable": false,\n "Containers": {},\n "Options": {\n "com.docker.network.bridge.default_bridge": "true",\n "com.docker.network.bridge.enable_icc": "true",\n "com.docker.network.bridge.enable_ip_masquerade": "true",\n "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",\n "com.docker.network.bridge.name": "docker0",\n "com.docker.network.driver.mtu": "1500"\n },\n "Labels": {}\n }\n]<\/code2><\/pre><\/pre>\n\n\n\n pretty standard options for the bridge network, even `enable_icc` is set to `true`. What about the container though ?<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# docker inspect kibana\n[\n {\n "Id": "2f08cc190b760361d9aa2951b4c9c407561fe35b8dbdc003f3f535719456f460",\n "Created": "2020-05-31T10:24:11.942315341Z",\n "Path": "\/usr\/local\/bin\/dumb-init",\n "Args": [\n "--",\n "\/usr\/local\/bin\/kibana-docker"\n ],\n "State": {\n "Status": "running",\n "Running": true,\n "Paused": false,\n "Restarting": false,\n "OOMKilled": false,\n "Dead": false,\n "Pid": 3735,\n "ExitCode": 0,\n "Error": "",\n "StartedAt": "2020-05-31T10:24:12.329214827Z",\n "FinishedAt": "0001-01-01T00:00:00Z"\n },\n "Image": "sha256:eadc7b3d59dd47b1b56f280732f38d16a4b31947cbc758516adbe1df5472b407",\n "ResolvConfPath": "\/var\/lib\/docker\/containers\/2f08cc190b760361d9aa2951b4c9c407561fe35b8dbdc003f3f535719456f460\/resolv.conf",\n "HostnamePath": "\/var\/lib\/docker\/containers\/2f08cc190b760361d9aa2951b4c9c407561fe35b8dbdc003f3f535719456f460\/hostname",\n "HostsPath": "\/var\/lib\/docker\/containers\/2f08cc190b760361d9aa2951b4c9c407561fe35b8dbdc003f3f535719456f460\/hosts",\n "LogPath": "",\n "Name": "\/kibana",\n "RestartCount": 0,\n "Driver": "overlay2",\n "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c434,c792",\n "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c434,c792",\n "AppArmorProfile": "",\n "ExecIDs": null,\n "HostConfig": {\n "Binds": null,\n "ContainerIDFile": "",\n "LogConfig": {\n "Type": "journald",\n "Config": {}\n },\n "NetworkMode": "bridge",\n "PortBindings": {\n "5601\/tcp": [\n {\n "HostIp": "",\n "HostPort": "5601"\n }\n ]\n },\n "RestartPolicy": {\n "Name": "no",\n "MaximumRetryCount": 0\n },\n "AutoRemove": true,\n "VolumeDriver": "",\n "VolumesFrom": null,\n "CapAdd": null,\n "CapDrop": null,\n "Dns": [],\n "DnsOptions": [],\n "DnsSearch": [],\n "ExtraHosts": null,\n "GroupAdd": null,\n "IpcMode": "",\n "Cgroup": "",\n "Links": null,\n "OomScoreAdj": 0,\n "PidMode": "",\n "Privileged": false,\n "PublishAllPorts": false,\n "ReadonlyRootfs": false,\n "SecurityOpt": null,\n "UTSMode": "",\n "UsernsMode": "",\n "ShmSize": 67108864,\n "Runtime": "docker-runc",\n "ConsoleSize": [\n 0,\n 0\n ],\n "Isolation": "",\n "CpuShares": 0,\n "Memory": 0,\n "NanoCpus": 0,\n "CgroupParent": "",\n "BlkioWeight": 0,\n "BlkioWeightDevice": null,\n "BlkioDeviceReadBps": null,\n "BlkioDeviceWriteBps": null,\n "BlkioDeviceReadIOps": null,\n "BlkioDeviceWriteIOps": null,\n "CpuPeriod": 0,\n "CpuQuota": 0,\n "CpuRealtimePeriod": 0,\n "CpuRealtimeRuntime": 0,\n "CpusetCpus": "",\n "CpusetMems": "",\n "Devices": [],\n "DiskQuota": 0,\n "KernelMemory": 0,\n "MemoryReservation": 0,\n "MemorySwap": 0,\n "MemorySwappiness": -1,\n "OomKillDisable": false,\n "PidsLimit": 0,\n "Ulimits": null,\n "CpuCount": 0,\n "CpuPercent": 0,\n "IOMaximumIOps": 0,\n "IOMaximumBandwidth": 0\n },\n "GraphDriver": {\n "Name": "overlay2",\n "Data": {\n "LowerDir": "\/var\/lib\/docker\/overlay2\/84217deb518fa6b50fb38aab03aa6a819150e0a248cf233bda8091b136c4825a-init\/diff:\/var\/lib\/docker\/overlay2\/bfda0aa2ec51f7047b5694e5daf89735f3021691e6154bc370827c168c4572f0\/diff:\/var\/lib\/docker\/overlay2\/5d32d74a3bb95b8e3377b1c115622f12a817a591936c4ae2da4512bc2e281e4b\/diff:\/var\/lib\/docker\/overlay2\/6482e711a89a90bebd61834aa8bd3463f567684dd3cdbbf2698179b752fdad7b\/diff:\/var\/lib\/docker\/overlay2\/4ae81e6a07956c974d985674c35e113ad3fbd9f4fdde43f4752c0e36a1153e69\/diff:\/var\/lib\/docker\/overlay2\/8330cdd839ec316133d659805f2839d1e65b16fbf7035324f419c2aa8d097925\/diff:\/var\/lib\/docker\/overlay2\/cd377c8c6fb23d050771d55ca15253cc9fa5043c7e49f41a2f73acd25f8e7ca9\/diff:\/var\/lib\/docker\/overlay2\/408c72d7e496be76503bbb01d5248c25be98e2290d71cae83d8d5d09d714f81d\/diff:\/var\/lib\/docker\/overlay2\/20068d51c4dd214db7b2b9d30fe13feb2e8ab35de646c9b652fea255476d396b\/diff:\/var\/lib\/docker\/overlay2\/0fdedfd6dbb551d32a9e826188a74936d0cec56e97c1b917fdd04b0e49a59a70\/diff:\/var\/lib\/docker\/overlay2\/0238ff31fbd60fdeaee1e162c92a1aa46735ec7b17df3b11455c09f18657c30f\/diff:\/var\/lib\/docker\/overlay2\/99a7a64a569e5e524e4139f9cf95bd929744c85c1633bcd8173c9172756c3233\/diff",\n "MergedDir": "\/var\/lib\/docker\/overlay2\/84217deb518fa6b50fb38aab03aa6a819150e0a248cf233bda8091b136c4825a\/merged",\n "UpperDir": "\/var\/lib\/docker\/overlay2\/84217deb518fa6b50fb38aab03aa6a819150e0a248cf233bda8091b136c4825a\/diff",\n "WorkDir": "\/var\/lib\/docker\/overlay2\/84217deb518fa6b50fb38aab03aa6a819150e0a248cf233bda8091b136c4825a\/work"\n }\n },\n "Mounts": [],\n "Config": {\n "Hostname": "2f08cc190b76",\n "Domainname": "",\n "User": "kibana",\n "AttachStdin": false,\n "AttachStdout": true,\n "AttachStderr": true,\n "ExposedPorts": {\n "5601\/tcp": {}\n },\n "Tty": false,\n "OpenStdin": false,\n "StdinOnce": false,\n "Env": [\n "ELASTICSEARCH_HOSTS=http:\/\/172.31.45.100:9200",\n "PATH=\/usr\/share\/kibana\/bin:\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin",\n "ELASTIC_CONTAINER=true"\n ],\n "Cmd": [\n "\/usr\/local\/bin\/kibana-docker"\n ],\n "Image": "docker.elastic.co\/kibana\/kibana:7.7.0",\n "Volumes": null,\n "WorkingDir": "\/usr\/share\/kibana",\n "Entrypoint": [\n "\/usr\/local\/bin\/dumb-init",\n "--"\n ],\n "OnBuild": null,\n "Labels": {\n "license": "Elastic License",\n "org.label-schema.build-date": "2020-05-12T03:25:49.654Z",\n "org.label-schema.license": "Elastic License",\n "org.label-schema.name": "kibana",\n "org.label-schema.schema-version": "1.0",\n "org.label-schema.url": "https:\/\/www.elastic.co\/products\/kibana",\n "org.label-schema.usage": "https:\/\/www.elastic.co\/guide\/en\/kibana\/index.html",\n "org.label-schema.vcs-url": "https:\/\/github.com\/elastic\/kibana",\n "org.label-schema.vendor": "Elastic",\n "org.label-schema.version": "7.7.0",\n "org.opencontainers.image.created": "2020-05-04 00:00:00+01:00",\n "org.opencontainers.image.licenses": "GPL-2.0-only",\n "org.opencontainers.image.title": "CentOS Base Image",\n "org.opencontainers.image.vendor": "CentOS"\n }\n },\n "NetworkSettings": {\n "Bridge": "",\n "SandboxID": "258a4a11f55f7425b837c7d5c0420dd344add081be79da7e33c146501dd8f0ec",\n "HairpinMode": false,\n "LinkLocalIPv6Address": "",\n "LinkLocalIPv6PrefixLen": 0,\n "Ports": {\n "5601\/tcp": [\n {\n "HostIp": "0.0.0.0",\n "HostPort": "5601"\n }\n ]\n },\n "SandboxKey": "\/var\/run\/docker\/netns\/258a4a11f55f",\n "SecondaryIPAddresses": null,\n "SecondaryIPv6Addresses": null,\n "EndpointID": "7126885dddf9ff031f2ff8c3b2cbd14708391dae619020bdb40efe7a849a01c7",\n "Gateway": "172.17.0.1",\n "GlobalIPv6Address": "",\n "GlobalIPv6PrefixLen": 0,\n "IPAddress": "172.17.0.2",\n "IPPrefixLen": 16,\n "IPv6Gateway": "",\n "MacAddress": "02:42:ac:11:00:02",\n "Networks": {\n "bridge": {\n "IPAMConfig": null,\n "Links": null,\n "Aliases": null,\n "NetworkID": "a6290df54ea24d14faa8d003d17802b3f8a4967680bc0c82c1211ab75d1815e2",\n "EndpointID": "7126885dddf9ff031f2ff8c3b2cbd14708391dae619020bdb40efe7a849a01c7",\n "Gateway": "172.17.0.1",\n "IPAddress": "172.17.0.2",\n "IPPrefixLen": 16,\n "IPv6Gateway": "",\n "GlobalIPv6Address": "",\n "GlobalIPv6PrefixLen": 0,\n "MacAddress": "02:42:ac:11:00:02"\n }\n }\n }\n }\n]<\/code2><\/pre><\/pre>\n\n\n\n all looks very normal regarding the docker container. Let’s check sysctl settings in \/etc<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# ls -Fla \/etc\/sysctl.d\/\ntotal 12\ndrwxr-xr-x. 2 root root 28 May 31 09:34 .\/\ndrwxr-xr-x. 84 root root 8192 May 31 10:15 ..\/\nlrwxrwxrwx. 1 root root 14 May 31 09:34 99-sysctl.conf -> ..\/sysctl.conf\n\n[root@ip-172-31-45-100 ~]# cat \/etc\/sysctl.d\/99-sysctl.conf \n\n# sysctl settings are defined through files in\n# \/usr\/lib\/sysctl.d\/, \/run\/sysctl.d\/, and \/etc\/sysctl.d\/.\n#\n# Vendors settings live in \/usr\/lib\/sysctl.d\/.\n# To override a whole file, create a new file with the same in\n# \/etc\/sysctl.d\/ and put new settings there. To override\n# only specific settings, add a file with a lexically later\n# name in \/etc\/sysctl.d\/ and put new settings there.\n#\n# For more information, see sysctl.conf(5) and sysctl.d(5).<\/code2><\/pre><\/pre>\n\n\n\n nothing interesting here as well. What if someone has messed up ip forwarding via other means though ?<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# sysctl -a 2>\/dev\/null| grep forward | grep -v ipv6\nnet.ipv4.conf.all.forwarding = 1\nnet.ipv4.conf.all.mc_forwarding = 0\nnet.ipv4.conf.default.forwarding = 1\nnet.ipv4.conf.default.mc_forwarding = 0\nnet.ipv4.conf.docker0.forwarding = 1\nnet.ipv4.conf.docker0.mc_forwarding = 0\nnet.ipv4.conf.ens5.forwarding = 1\nnet.ipv4.conf.ens5.mc_forwarding = 0\nnet.ipv4.conf.lo.forwarding = 1\nnet.ipv4.conf.lo.mc_forwarding = 0\nnet.ipv4.ip_forward = 1\nnet.ipv4.ip_forward_use_pmtu = 0<\/code2><\/pre><\/pre>\n\n\n\n all looks fine here too. Let’s check some more sysctl settings regarding bridge + iptables<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# sysctl -a 2>\/dev\/null| grep bridge\nnet.bridge.bridge-nf-call-arptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-filter-pppoe-tagged = 0\nnet.bridge.bridge-nf-filter-vlan-tagged = 0\nnet.bridge.bridge-nf-pass-vlan-input-dev = 0<\/code2><\/pre><\/pre>\n\n\n\n everything still looks fine in these configuration settings, but the packets from the container still can’t reach the host.
Next step is to setup a netcat listening service on the host on a different port and try to connect to it via the container. That still doesn’t work, no packets to be seen on ens5.
Could it be ebtables ? No..no way..but what if…<\/p>\n\n\n\n[root@ip-172-31-45-100 ~]# ebtables -L\nBridge table: filter\nBridge chain: INPUT, entries: 0, policy: ACCEPT\nBridge chain: FORWARD, entries: 0, policy: ACCEPT\nBridge chain: OUTPUT, entries: 0, policy: ACCEPT<\/code2><\/pre><\/pre>\n\n\n\n still nothing interesting. Could it be a kernel bug ? is this some custom kernel ?<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# uname -a\nLinux ip-172-31-45-100.eu-central-1.compute.internal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU\/Linux<\/code2><\/pre>\n\n\n\n nope…that’s a vanilla centos7 kernel. Could it be nftables ? On 3.10 kernel and centos7 ?<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# nft list tables\n-bash: nft: command not found<\/code2><\/pre><\/pre>\n\n\n\n Nobody uses nftables yet, right ? Another wild thought, are there any ip rules defined ?<\/p>\n\n\n\n
[root@ip-172-31-45-100 ~]# ip rule ls\n0: from all lookup local \n100: from 172.31.45.100 lookup 1 \n32766: from all lookup main \n32767: from all lookup default <\/code2><\/pre><\/pre>\n\n\n\n bingo, there’s a rule with priority 100 that matches the host’s IP address! What is this ip rule doing there ? Let’s check
routing table 1<\/code> that the lookup of rule 100 points to<\/p>\n\n\n\n[root@ip-172-31-45-100 ~]# ip route ls table 1\ndefault via 172.31.32.1 dev ens5 \n172.31.32.0\/20 dev ens5 scope link <\/code2><\/pre><\/pre>\n\n\n\n at last, here’s the answer!<\/p>\n\n\n\n
There’s an IP rule entry that says that packets with a source IP of the ens5 interface should lookup routing entries only in
routing table 1<\/code>, which is not the main routing table. That routing table knows nothing about the docker network (172.17.0.0\/16). Let’s delete the rule from the host<\/p>\n\n\n\n[root@ip-172-31-45-100 ~]# ip rule del from 172.31.45.100\/32 tab 1 priority 100<\/code2><\/pre>\n\n\n\n and check if the container can contact the service now<\/p>\n\n\n\n
(container) bash-4.2$ curl 172.31.45.100:9200\n{\n "name" : "node1",\n "cluster_name" : "centos7",\n "cluster_uuid" : "d6fBSua6Q9OvSu534roTpA",\n "version" : {\n "number" : "7.7.0",\n "build_flavor" : "default",\n "build_type" : "rpm",\n "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",\n "build_date" : "2020-05-12T02:01:37.602180Z",\n "build_snapshot" : false,\n "lucene_version" : "8.5.1",\n "minimum_wire_compatibility_version" : "6.8.0",\n "minimum_index_compatibility_version" : "6.0.0-beta1"\n },\n "tagline" : "You Know, for Search"\n}<\/code2><\/pre> <\/code2><\/pre>\n\n\n\n Success!<\/p>\n\n\n\n
Where’s the SYN+ACK ?<\/h3>\n\n\n\n
Does the SYN packet reach the listening service ? No…and the reason is
rp_filter<\/code>. Centos7 setsnet.ipv4.conf.default.rp_filter=1<\/code>, so whendocker0<\/code> interface gets created it is set tonet.ipv4.conf.docker0.rp_filter=1<\/code>. <\/p>\n\n\n\nHere’s what
rp_filter<\/code> values mean according to kernel documentation<\/a>:<\/p>\n\n\n\n
- 0 – No source validation.<\/li>
- 1 – Strict mode as defined in RFC3704 Strict Reverse Path Each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded.<\/li>
- 2 – Loose mode as defined in RFC3704 Loose Reverse Path Each incoming packet’s source address is also tested against the FIB and if the source address is not reachable via any interface the packet check will fail.<\/li><\/ul>\n\n\n\n
After reverting the deleted ip rule via
ip rule add from 172.31.45.100\/32 tab 1 priority 100<\/code> and settingsysctl -w net.ipv4.conf.docker0.rp_filter=0<\/code> we can see the SYN+ACK packet going out ofens5<\/code> interface towards the default gateway.<\/p>\n\n\n\n[root@ip-172-31-45-100 ~]# tcpdump -enni ens5 port 9200\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes\n15:44:19.028353 06:ce:3b:94:fe:ac > 06:1b:e5:19:30:12, ethertype IPv4 (0x0800), length 74: 172.31.45.100.9200 > 172.17.0.2.47460: Flags [S.], seq 1431574088, ack 1879007024, win 26847, options [mss 8961,sackOK,TS val 22063599 ecr 22063599,nop,wscale 7], length 0\n15:44:20.029163 06:ce:3b:94:fe:ac > 06:1b:e5:19:30:12, ethertype IPv4 (0x0800), length 74: 172.31.45.100.9200 > 172.17.0.2.47460: Flags [S.], seq 1431574088, ack 1879007024, win 26847, options [mss 8961,sackOK,TS val 22064600 ecr 22063599,nop,wscale 7], length 0\n15:44:21.229126 06:ce:3b:94:fe:ac > 06:1b:e5:19:30:12, ethertype IPv4 (0x0800), length 74: 172.31.45.100.9200 > 172.17.0.2.47460: Flags [S.], seq 1431574088, ack 1879007024, win 26847, options [mss 8961,sackOK,TS val 22065800 ecr 22063599,nop,wscale 7], length 0<\/code2><\/pre><\/pre>\n\n\n\n Finding such discarded packets, called martians, in the logs can be done by enabling
log_martians<\/code> viasysctl -w net.ipv4.conf.all.log_martians=1<\/code>. Example syslog message:<\/p>\n\n\n\nMay 31 16:01:08 ip-172-31-45-100 kernel: IPv4: martian source 172.31.45.100 from 172.17.0.2, on dev docker018\nMay 31 16:01:08 ip-172-31-45-100 kernel: ll header: 00000000: 02 42 e9 38 3d a8 02 42 ac 11 00 02 08 00 .B.8=..B......<\/code2><\/pre><\/pre>\n\n\n\n But why ?<\/h3>\n\n\n\n
Why was the rule there in the original case ? Multihoming was tried, it didn’t work as expected and not all the configs were removed. Grep-ing \/etc for the host’s IP found the following file:<\/p>\n\n\n\n
\/etc\/sysconfig\/network-scripts\/rule-ens5:from 172.31.45.100\/32 tab 1 priority 100<\/pre>\n\n\n\nIn multihoming it’s common that packets reaching a host on interface X should also be replied back from interface X. Part of a method to achieve this is to assign each interface its own routing table<\/a>.<\/p>\n\n\n\n
So when asked to troubleshoot networking issues act like Dr. House<\/a> would, assume the worst.<\/p>\n\n\n\n