[deconstructing cloud Nativity] getting to know kubernetes service

Time:2020-9-28

Editor’s note: cloud origin is one of the core technology directions pursued by Netease Hangzhou Research Institute (Netease Hangzhou Research Institute). As the technical standard of cloud native industry and the cornerstone of cloud original ecology, the open source container platform kubernetes inevitably has its complexity in design. Based on the summary of senior engineers of Netease Hangzhou Research Institute, kubernetes series introduces the principle and application of kubernetes from multiple angles and levels, How to solve the actual needs in production and avoid risks, hope to communicate with readers and make progress together.

This article is authorized by the author, please do not reprint without permission.

Author: Li Lanqing, senior engineer of Netease Hangzhou Research Institute cloud computing technology center

Why introduce service

As we all know, the life cycle of pod is unstable and may die day by day, which means that the IP of pod is not fixed.

For example, we deploy the nginx service with three copies of deployment, and each pod is assigned an IP address. Due to the unstable life cycle of the pod, the pod may be deleted and rebuilt, and the IP address of the pod will change. There is also a scenario where we may scale nginx deployment from three to five or to two. When we need to access the nginx service mentioned above, it is difficult for the client to configure and manage the IP address of nginx service.

As a result, the kubernetes community is abstractedserviceThis resource object or logical concept.

What is service

Service is one of the core resource objects in kubernetes. Each service in kubernetes is actually the “microservice” we often refer to.

Service defines the entry address of a service, which associates the pod of the back end through the label selector. A service will be automatically assigned a clusterip. The service life cycle is stable, and its clusterip will not change. Users can access the back-end pod by accessing the service’s clusterip. Therefore, no matter how the back-end pod is expanded or shrunk, how to delete and rebuild, the client does not need to care.

[deconstructing cloud Nativity] getting to know kubernetes service

(1) Create a three copy nginx deployment:
nginx.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
# kubectl create -f nginx.yaml
deployment.extensions/nginx created

# kubectl get pods -o wide
nginx-5c7588df-5dmmp                                            1/1     Running       0          57s     10.120.49.230   pubt2-k8s-for-iaas4.dg.163.org   <none>           <none>
nginx-5c7588df-gb2d8                                            1/1     Running       0          57s     10.120.49.152   pubt2-k8s-for-iaas4.dg.163.org   <none>           <none>
nginx-5c7588df-gdngk                                            1/1     Running       0          57s     10.120.49.23    pubt2-k8s-for-iaas4.dg.163.org   <none>           <none>

(2) Create a service and associate the nginx pod with the label selector:

svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
# kubectl create -f svc.yaml
service/nginx created

# kubectl get svc nginx -o wide
NAME    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
nginx   ClusterIP   10.178.4.2   <none>        80/TCP    23s   app=nginx

(3) Accessing service address on k8s node

# curl 10.178.4.2:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Implementation principle

There are several key fields in service:

  • spec.selector: associate the pod belonging to the service through this field
  • spec.clusterIP: k8s automatically assigned virtual IP address
  • spec.ports: defines the listening port and destination port. Users can access theClusterip: listening portTo access the back-end pod

When a user creates a service, the Kube controller manager will automatically create an endpoint resource with the same name as the service

# kubectl get endpoints nginx
NAME    ENDPOINTS                                           AGE
nginx   10.120.49.152:80,10.120.49.23:80,10.120.49.230:80   12m

In the endpoints resource, a list of pods associated with the service is saved. This list is automatically maintained by the Kube controller manager. When a pod is added or deleted, the list will be automatically refreshed.

For example, we deleted one of the pods:

# kubectl delete pods nginx-5c7588df-5dmmp
pod "nginx-5c7588df-5dmmp" deleted

# kubectl get pods
nginx-5c7588df-ctcml                                            1/1     Running     0          6s
nginx-5c7588df-gb2d8                                            1/1     Running     0          18m
nginx-5c7588df-gdngk                                            1/1     Running     0          18m

As you can see, Kube controller manager immediately added a new pod. Then let’s take a look at the endpoints resource. The back-end pod list is also updated automatically

# kubectl get endpoints nginx
NAME    ENDPOINTS                                          AGE
nginx   10.120.49.152:80,10.120.49.23:80,10.120.49.73:80   16m

Well, when users go to visitclusterip:portHow is the traffic load balanced to the back-end pod?

K8s runs one on each nodekube-proxyComponents,kube-proxyCan watch the service and endpoints resources, and realize the service load balancing by configuring iptables rules (now IPVS is also supported, but not in the scope of this article).

You can look at the iptables rules of nginx service on any k8s node

# iptables -t nat -L PREROUTING
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */

# iptables -t nat -L KUBE-SERVICES
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-SVC-4N57TFCL4MD7ZTDA  tcp  --  anywhere             10.178.4.2           /* default/nginx: cluster IP */ tcp dpt:http

# iptables -t nat -L KUBE-SVC-4N57TFCL4MD7ZTDA
Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references)
target     prot opt source               destination
KUBE-SEP-AHN4ALGUQHWJZNII  all  --  anywhere             anywhere             statistic mode random probability 0.33332999982
KUBE-SEP-BDD6UBFFJ4G2PJDO  all  --  anywhere             anywhere             statistic mode random probability 0.50000000000
KUBE-SEP-UR2OSKI3P5GEGC2Q  all  --  anywhere             anywhere

# iptables -t nat -L KUBE-SEP-AHN4ALGUQHWJZNII
Chain KUBE-SEP-AHN4ALGUQHWJZNII (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.120.49.152        anywhere
DNAT       tcp  --  anywhere             anywhere             tcp to:10.120.49.152:80

When users accessclusterip:portIptables will be balanced to the back-end pod through the iptables DNAT balanced load.

service ClusterIP

The clusterip of service is a virtual IP, which is not attached to any network devices, but only exists in the iptables rules. The load balancing of accessing clusterip is realized through DNAT.

When a user creates a service, k8s will automatically allocate an idle IP setting from the service network segment to.spec.clusterIPField. Of course, k8s also supports users to specify clusterip when creating SVC.

The network segment of service is through the command line parameter of Kube apiserver--service-cluster-ip-rangeConfiguration, not allowed to change.

The service network segment cannot conflict with the computer room network, docker network segment and container network segment. Otherwise, the network may be blocked.

The clusterip of service is the virtual IP in the k8s cluster. Different k8s clusters can use the same service network segment. Outside the k8s cluster, the service clusterip cannot be accessed.

Service domain name

Kubernetes has its own domain name resolution service. For example, we can visit the domain namenginx.default.svc.cluster.localTo access the nginx service mentioned above:

$ curl nginx.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

The domain name format is:${ServiceName}.${Namespace}.svc.${ClusterDomain}. where the default value of ${clusterdomain} iscluster.localYou can use the command line parameters of kubelet----cluster-domainConfigure.

headless service

When the service IP is not required, it can be specified when the service is createdspec.clusterIP: NoneThis service is called headless service. Since there is no service IP assigned, Kube proxy will not process this service.

DNS resolution of this service:

  • When selectors are defined in the service: endpoints controller will create corresponding endpoints. The a record in DNS will resolve the SVC address to the address of these pods
  • When selector: endpoints is not defined in the service, the controller will not create endpoints. DNS does this:
  • First, CNAME is transferred to the externalname defined in the service
  • If the externalname is not defined, all endpoints sharing the name with the service will be searched, and the a record will be resolved to the addresses of these endpoints

Different types of services

Service supports a variety of different types, includingClusterIPNodePortLoadBalancer, by fieldspec.typeConfigure.

ClusterIP service

Default type. For the clusterip service, k8s will automatically assign a virtual clusterip that can only be reached within the cluster and cannot be accessed outside the k8s cluster.

NodePort service

In addition to automatically assigning a clusterip to nodeport service, k8s will also automatically assign a nodeport port. Clients outside the cluster can access the IP of any node and add nodeport to balance the load to the back-end pod.

The port range of nodeport can be set through the command line parameter of Kube apiserver--service-node-port-rangeConfiguration, the default value is30000-32767, our current configuration is30000-34999

However, which node IP is accessed by the client is also a problem to be considered, and high availability needs to be considered. Moreover, nodeport will cause an extra hop when accessing back-end services, and it may make SNAT and fail to see the source IP.

Another thing to note is that,service-node-port-rangeCannot conflict with several port ranges:

  • Linux’snet.ipv4.ip_local_port_range, which can be configured as35000-60999
  • The port must be less than 30000
  • The ports of other ordinary services also need to be less than 30000

LoadBalancer service

The loadbalancer service needs to connect to the NLB service of the cloud service provider. When a user creates a service of type loadbalancer,cloud-controller-managerThe NLB API is called to automatically create the LB instance, and the pod of the service back-end is hung to the LB instance back-end.

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: LoadBalancer
$ kubectl get svc nginx
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
nginx     LoadBalancer   10.178.8.216   10.194.73.147   80:32514/TCP   3s

Session persistence in service

Users can configurespec.serviceAffinity=ClientIPTo realize the session keeping function based on client IP. This field defaults to none.

You can also use the appropriate settingsservice.spec.sessionAffinityConfig.clientIP.timeoutSecondsTo set the maximum session dwell time. (the default is 10800 seconds, or 3 hours)

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: ClusterIP
  sessionAffinity: ClientIP

kubernetes service

After we have deployed a k8s cluster, we found that the system automatically helped us in thedefaultIn the namespace, create akubernetesService:

# kubectl get svc kubernetes -o yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
spec:
  clusterIP: 10.178.4.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

You can seekubernetesThe IP of SVC is--service-cluster-ip-rangeAnd the service is not setspec.selector。 In theory, for SVC without selector, Kube controller manager will not automatically create endpoints resource with the same name.

However, we can see that there are endpoints with the same name, and the addresses of multiple apiserver are also saved in the endpoints resource

# kubectl get ep kubernetes
NAME         ENDPOINTS                         AGE
kubernetes   10.120.0.2:6443,10.120.0.3:6443   137d

Specific is how to achieve, interested can look at the source codek8s.io/kubernetes/pkg/master/reconcilers

Frequently Asked Questions

Question 1: why can’t service clusterip be pinged

Because the service clusterip is a virtual IP within the k8s cluster, it is not attached to any network devices, and only exists in the iptables NAT rules to achieve load balancing.

Question 2: why can’t the service network segment conflict with the docker network segment, container network segment, or computer room segment

If the service network segment conflicts with the above network segment, it is easy to cause the network failure when the container or k8s node accesses the above network segment.

Question 3: why can’t service clusterip be accessed outside the k8s cluster

Service clusterip is a virtual IP that can be reached within the k8s cluster, but not outside the cluster. Different k8s clusters can use the same service network segment.

In other words, there is no Kube proxy component of the k8s cluster on the machine outside the cluster, and no corresponding iptables rules are created. Therefore, service clusterip cannot be accessed outside the cluster.

Question 4: can the service segment be expanded

In principle, this network segment is not allowed to be changed. However, if the allocated network segment is too small due to the early planning, the service network segment can be expanded by comparing the operation and maintenance methods of hack.

Question 5: does the service support seven layer load balancing

Service only supports four layers of load balancing, and seven layers of load balancing needs to use ingress

Reference documents

  1. https://kubernetes.io/docs/concepts/services-networking/service/

Brief introduction to the author

Li Lanqing, a senior system development engineer in the container layout team of Netease Hangzhou Research Institute cloud computing technology center, has many years of kubernetes development and operation and maintenance experience. He has led the research and development of production level core systems such as container network management and container mixing department, and promoted the containerization of e-commerce, music, media, education and other businesses within Netease group.