K8s and DNS — some practical experience of coredns

Time:2020-5-29

About coredns

Install the new version k8s, coredns has become the default DNS. Before Kube DNS. Coredns is a flexible and extensible DNS server, which can be used as kubernetes cluster DNS. Like kubernetes, the coredns project is hosted by CNCF. But in practical use, we need some attention.

Increase the anti affinity of the application to prevent coredns from scheduling to a host

Coredns requires very small resources, so it is easy to schedule to a host. Coredns is a system component. We should try our best to make coredns distributed and enhance its usability. Therefore, add the following settings in yaml of deployment:

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - coredns
            topologyKey: kubernetes.io/hostname

Here, we use a property of k8s anti affinity.

Choose a reasonable coredns scaling

For many k8s deployments, the default is to deploy two coredns instances, but when our cluster grows larger, the two instances cannot meet the needs. So the scaling of coredns is very important. Remember that coredns should not be elastically scalable with HPA. Frequent scaling will lead to many failures in DNS resolution. Cluster promotional autoscaler component is needed. I usually choose to scale DNS according to the number of nodes. You can choose specific scaling strategies.

How to disable IPv6 with coredns

If the k8s cluster host does not shut down the IPv6 kernel module, the default behavior when the container requests coredns is to initiate IPv4 and IPv6 parsing at the same time.
Because we usually only use IPv4 address, or you do not support IPv6 for the host environment, in the actual business scenario, the IPv6 address returned from AAAA resolution will fail to access. As follows:

2019/09/06 18:12:37 [error] 37#0: *265 connect() to [2404:6800:4003:c03::5f]:443 failed (101: Network is unreachable), client: 100.125.198.131, server: , request: "POST /user/google/signin HTTP/1.1", host: "user.inner.xxx.com"
2019/09/06 18:12:37 [error] 37#0: *265 [lua] http_util.lua:49: http_get(): http request error, url = https://www.googleapis.com/oauth2/v1/userinfo?access_token=ya29.Glt7B5qqIHMVkyJNSmE32jGAo-hkEgIyK2CzMcO0ksrXcCZSMts4VcBoY-uNQmXdEhb8QJQAhVsv-5LxESalKNiD7rJrBgYJgfV-z81No9a_vwW59RgBEvYJMAAr; request headers = null ; request body = ; error = network is unreachable, client: 100.125.198.131, server: , request: "POST /user/google/signin HTTP/1.1", host: "user.inner.xxx.com"

So at this time, if we only configure domain – > IPv4 address resolution in coredns, when coredns receives IPv6 resolution request, it will forward to upstream DNS server for resolution because the configuration cannot be found locally, which will slow down DNS resolution request of container.
Coredns provides a plugin called template. After configuration, it can immediately return an empty result response to all IPv6 requests, avoiding the request to forward to the upstream DNS.
For services, if IPv6 parsing is not successful, it will be degraded to IPv4 a parsing.
The template plug-in is enabled in coredns by default. You only need to add the following configuration in the configuration file:

template ANY AAAA {
    rcode NXDOMAIN
}

Coredns configure stub domain and upstream nameserver

In the actual scenario, we often have our own internal DNS server, for example, our consul domain server is located at 10.150.0.1, and all consul names have suffixes consul.local 。 To configure it in coredns, the Cluster Administrator creates the following configuration in coredns configmap:

consul.local:53 {
        errors
        cache 30
        forward . 10.150.0.1
    }