Kubelet source code analysis — Introduction and startup of kubelet

Time:2022-4-21

Kubelet is a component in k8s cluster. Its role as an agent is distributed on each node, whether it is master or worker. It has many functions and complex logic. The main functions are

  1. Node status synchronization: kublet synchronizes the status of the current node to the API server, and will synchronize the CPU, memory, disk space and other resources of the current node to the API server to provide basic data support for the scheduler when scheduling pod
  2. Start stop and status management of Pod: kubelet will start the pod dispatched to this node by the scheduler, synchronize its status, ensure its operation, and be responsible for resource recovery when the pod is closed

Main modules

Kublet has the following listening ports

  • 10250 kubelet API: used to communicate with API server and access the resources and status of node
  • 4194 cdadvisor: it is used to obtain the environment information of nodes and the indicator information of pod status,
  • 10255 readonly API: obtain pod and node information in read-only form
  • 10248 / healthz: for health examination

Kubelet contains the following modules

PLEG, cdadvisor, container manager, volume manager, evaluation manager, oomwatcher, probemanager, statusmanager, imagegc, containergc, imagemanager, certificatemanager, runtimeclassmanager… As shown in the figure below
avatar

Pick a few to introduce

  • PLEG (POD lifecycle event generator): kubelet’s core module always calls the container runtime to obtain the information of containers / sandboxes of this node, compares it with the pods cache information maintained by itself, generates the corresponding podlifecycle event, and sends it to kubelet syncloop for consumption through eventchannel. Then kubelet syncpod triggers the pod synchronization process, and finally achieves the user’s desired state.
  • Cdadvisor: it collects the monitoring information of this node and container, and provides an interface, which is also used by imagemanager, oomwatcher, containermanager, etc.
  • Oomwatcher: the listener of system oom. It will establish systemoom with the cadvisor module, receive the oom signal from the cadvisor through the watch mode, and generate relevant events.
  • Statusmanager: it is responsible for maintaining the status information and updating the pod status to apiserver. However, it is not responsible for monitoring the change of pod status, but provides corresponding interfaces for other components to call, such as probemanager.
  • Volume manager: responsible for the management of the volume used by the pod on the node node. Volume is associated with the life cycle of the pod. It is responsible for the mount / amount / attach / detach process of volume in the process of creating and deleting the pod
  • Probemanager: it relies on statusmanager, livenessmanager and containerrefmanager to regularly monitor the health status of containers in the pod. Currently, it supports two types of probes: livenessprobe and readiesprobe.

Start command

Kubelet runs on each node in binary. You can see its startup command through PS – EF. Generally, when installing the cluster, it will be hosted in the system service, and the kubelet will be started automatically when the node is started.

get into/usr/lib/systemd/system/kubelet.service.dOpen the file in the directory to view

cd /usr/lib/systemd/system/kubelet.service.d
ls
10-kubeadm.conf
cat 10-kubeadm.conf
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Kubelet startup has four environment variables pieced together into parameters, and the first two environment variables have been provided in lines 1 and 2. According to note kubelet_ CONFIG_ Args is the file generated by “kubedm init” and “kubedm join” at runtime. The path of the file is / var / lib / kubelet / kubedm flags env; The last parameter is kubelet_ EXTRA_ Args is the last means for users to overwrite kubelet. The path of the file is / etc / sysconfig / kubelet. According to the experimental machine, the contents are as follows

cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --cluster-dns=10.96.0.10 --docker-endpoint=unix:///var/run/docker.sock --hostname-override=master1 --network-plugin=cni --pod-infra-container-image=deploy.deepexi.com/kubeadm/pause:3.2 --root-dir=/var/lib/kubelet --v=4"
cat /etc/sysconfig/kubelet
-bash: cd: /etc/sysconfig/kubelet: No such file or directory

The startup command of kubelet finally combined is

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --cluster-dns=10.96.0.10 --docker-endpoint=unix:///var/run/docker.sock --hostname-override=master1 --network-plugin=cni --pod-infra-container-image=deploy.deepexi.com/kubeadm/pause:3.2 --root-dir=/var/lib/kubelet --v=4

It is consistent with the results obtained by PS – ef|grep kubelet

ps -ef|grep kubelet
Root 1302 1 3 September 08? 16:17:27 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet. conf --kubeconfig=/etc/kubernetes/kubelet. conf --config=/var/lib/kubelet/config. yaml --cgroup-driver=systemd --cluster-dns=10.96.0.10 --docker-endpoint= unix:///var/run/docker.sock  --hostname-override=master1 --network-plugin=cni --pod-infra-container-image=deploy. deepexi. com/kubeadm/pause:3.2 --root-dir=/var/lib/kubelet --v=4

Among the above parameters, the config parameter passes a yaml file. When you open it, you will find that there are a considerable number of parameters. Because the length is too long, they will not be displayed here

Kubelet start process

Source code version: 1.19

Kubelet handles startup commands and startup parameters through the cobra framework. It can directly jump from the main function to the cobra run function registration, and roughly do the following things

  1. Initialize startup commands and parameters
  2. Initialize featuregate
  3. Verify command line parameters
  4. Load kubeletconfigfile and verify it, that is, the file passed in by the config parameter mentioned in the “start command” section
  5. Loading uses dynamic configuration, if enabled
  6. Construct kubeletserver and kubeletdeps
  7. Call the run function to run kubelet
    The code is located in / CMD / kubelet / APP / server go
func(cmd *cobra.Command, args []string) {
	//1.  Initialize startup commands and parameters
	if err := cleanFlagSet.Parse(args); err != nil {
	}
	//2.  Initialize featuregate
	if err := utilfeature.DefaultMutableFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
	}
	//3.  Verify command line parameters
	if err := options.ValidateKubeletFlags(kubeletFlags); err != nil {
	}
	//4.  Load kubeletconfigfile and verify it
	if configFile := kubeletFlags.KubeletConfigFile; len(configFile) > 0 {
		kubeletConfig, err = loadConfigFile(configFile)
	}
	if err := kubeletconfigvalidation.ValidateKubeletConfiguration(kubeletConfig); err != nil {
		klog.Fatal(err)
	}

	////5.  Loading parts using dynamic configuration

	//6.  Construct kubeletserver and kubeletdeps
	// construct a KubeletServer from kubeletFlags and kubeletConfig
	kubeletServer := &options.KubeletServer{
		KubeletFlags:         *kubeletFlags,
		KubeletConfiguration: *kubeletConfig,
	}

	// use kubeletServer to construct the default KubeletDeps
	kubeletDeps, err := UnsecuredDependencies(kubeletServer, utilfeature.DefaultFeatureGate)
	if err != nil {
		klog.Fatal(err)
	}

	//7.  Call the run function to run kubelet
	if err := Run(ctx, kubeletServer, kubeletDeps, utilfeature.DefaultFeatureGate); err != nil {
		klog.Fatal(err)
	}

}

Run function

The run function mainly performs some environment checks, preparations and verification operations for kubelet startup

  1. Register the current configuration file in the HTTP server / configz URL
  2. Initialize various clients
  3. Initialize auth, cgrouproot, cadvisor, containermanager
  4. Set oom score for kubelet process
  5. Initialize the runtime server and set the CRI
  6. Call the runkubelet method to perform subsequent startup operations
  7. Start healthz HTTP server
    The code is located in / CMD / kubelet / server / server go
func run(ctx context.Context, s *options.KubeletServer, kubeDeps *kubelet.Dependencies, featureGate featuregate.FeatureGate) (err error) {
	//1 register the current configuration file in the HTTP server / configz URL
	err = initConfigz(&s.KubeletConfiguration)
	//2. Initialize various clients, mainly in non standalone mode. Otherwise, all clients will be empty
	switch {
	case kubeDeps.KubeClient == nil, kubeDeps.EventClient == nil, kubeDeps.HeartbeatClient == nil:
		kubeDeps.KubeClient, err = clientset.NewForConfig(clientConfig)
		kubeDeps.EventClient, err = v1core.NewForConfig(&eventClientConfig)
		kubeDeps.HeartbeatClient, err = clientset.NewForConfig(&heartbeatClientConfig)
	}
	//3 initialize auth, cgrouproot, cadvisor, containermanager
	if kubeDeps.Auth == nil {
		auth, runAuthenticatorCAReload, err := BuildAuth(nodeName, kubeDeps.KubeClient, s.KubeletConfiguration)
	}
	nodeAllocatableRoot := cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupsPerQOS, s.CgroupDriver)
	if kubeDeps.CAdvisorInterface == nil {
		imageFsInfoProvider := cadvisor.NewImageFsInfoProvider(s.ContainerRuntime, s.RemoteRuntimeEndpoint)
	}
	if kubeDeps.ContainerManager == nil {
		kubeDeps.ContainerManager, err = cm.NewContainerManager(...)
	}
	//4.  Set oom score for kubelet process
	if err := oomAdjuster.ApplyOOMScoreAdj(0, int(s.OOMScoreAdj)); err != nil {
	}
	//5.  Initialize the runtime server and set the CRI
	err = kubelet.PreInitRuntimeService(...)
	//6.  Call the runkubelet method to perform subsequent startup operations
	if err := RunKubelet(s, kubeDeps, s.RunOnce); err != nil {
		return err
	}
	//7.  Start healthz HTTP server
	if s.HealthzPort > 0 {
		go wait.Until(func() {
			err := http.ListenAndServe(net.JoinHostPort(s.HealthzBindAddress, strconv.Itoa(int(s.HealthzPort))), mux)
		}, 5*time.Second, wait.NeverStop)	
	}
}

RunKubelet

There are only two cores of the runkubelet function

  1. Initialize kubelet object
  2. Run kubelet and related kubelet APIs
func RunKubelet(kubeServer *options.KubeletServer, kubeDeps *kubelet.Dependencies, runOnce bool) error {
	k, err := createAndInitKubelet(...)
	if runOnce {
	}else{
		startKubelet(k, podCfg, &kubeServer.KubeletConfiguration, kubeDeps, kubeServer.EnableCAdvisorJSONEndpoints, kubeServer.EnableServer)
	}
}
createAndInitKubelet

Createandinitkubelet first constructs the kubelet object. The newmainkubelet function also passes in many parameters. The function contains the initialization operation of the “main module” in the previous article. After the completion of the base note, call the BirthCry method to send a Starting kubelet. to k8s. Event. Then start containergc immediately

func createAndInitKubelet(......) {
        k, err = kubelet.NewMainKubelet(...)

        k.BirthCry()
        k.StartGarbageCollection()
        return k, nil
    }
startKubelet

The startkubelet function calls kubelet’s run method to run kubelet Run contains the start method call of manager mentioned in some “main modules”, which means that the modules of kubelet will run from now on. In addition, it also includes the core loop syncloop of kubelet, which will be called here

After running kubelet, kubelet API, readonly API and other servers also start running here

func startKubelet(...) {
	// start the kubelet
	go k.Run(podCfg.Updates())

	// start the kubelet server
	if enableServer {
		go k.ListenAndServe(net.ParseIP(kubeCfg.Address), uint(kubeCfg.Port), kubeDeps.TLSOptions, kubeDeps.Auth,
			enableCAdvisorJSONEndpoints, kubeCfg.EnableDebuggingHandlers, kubeCfg.EnableContentionProfiling, kubeCfg.EnableSystemLogHandler)

	}
	if kubeCfg.ReadOnlyPort > 0 {
		go k.ListenAndServeReadOnly(net.ParseIP(kubeCfg.Address), uint(kubeCfg.ReadOnlyPort), enableCAdvisorJSONEndpoints)
	}
	if utilfeature.DefaultFeatureGate.Enabled(features.KubeletPodResources) {
		go k.ListenAndServePodResources()
	}
}

At this point, kubelet runs and starts to report its node resources, start and stop pod and state management.

Start the call chain of the process

Review the whole startup process through the following call chain

main                                                                             // cmd/kubelet/kubelet.go
 |--NewKubeletCommand                                                            // cmd/kubelet/app/server.go
   |--Run                                                                        // cmd/kubelet/app/server.go
      |--initForOS                                                               // cmd/kubelet/app/server.go
      |--run                                                                     // cmd/kubelet/app/server.go
        |--initConfigz                                                           // cmd/kubelet/app/server.go
        |--BuildAuth
        |--cm.NodeAllocatableRoot
        |--cadvisor.NewImageFsInfoProvider
        |--NewContainerManager
        |--ApplyOOMScoreAdj
        |--PreInitRuntimeService
        |--RunKubelet                                                            // cmd/kubelet/app/server.go
        | |--k = createAndInitKubelet                                            // cmd/kubelet/app/server.go
        | |  |--NewMainKubelet
        | |  |  |--watch k8s Service
        | |  |  |--watch k8s Node
        | |  |  |--klet := &Kubelet{}
        | |  |  |--init klet fields
        | |  |
        | |  |--k.BirthCry()
        | |  |--k.StartGarbageCollection()
        | |
        | |--startKubelet(k)                                                     // cmd/kubelet/app/server.go
        |    |--go k.Run()                                                       // -> pkg/kubelet/kubelet.go
        |    |  |--go cloudResourceSyncManager.Run()
        |    |  |--initializeModules
        |    |  |--go volumeManager.Run()
        |    |  |--go nodeLeaseController.Run()
        |    |  |--initNetworkUtil() // setup iptables
        |    |  |--go Until(PerformPodKillingWork, 1*time.Second, neverStop)
        |    |  |--statusManager.Start()
        |    |  |--runtimeClassManager.Start
        |    |  |--pleg.Start()
        |    |  |--syncLoop(updates, kl)                                         // pkg/kubelet/kubelet.go
        |    |
        |    |--k.ListenAndServe
        |
        |--go http.ListenAndServe(healthz)

Summary

This article is the beginning of kubelet’s source code journey. First, it briefly introduces kublet’s position in k8s cluster, its functions, included modules and startup commands. Then explore the startup process of kubelet layer by layer by tracing the code call chain, which managers are initialized, which managers are started when starting kubelet, and which ports are monitored.

If you are interested, you can read other articles in my “k8s source code journey” series
Kubelet source code analysis — Introduction and startup of kubelet
Kubelet source code analysis — start pod
Kubelet source code analysis – close pod
Kubelet source code analysis — monitoring pod changes
Scheduler source code analysis – scheduling process
Apiserver source code analysis – startup process
Apiserver source code analysis — processing requests

Reference articles

Analysis of kubelet architecture
Kubelet startup process analysis
Wanzi Changwen: what happened behind the creation of pod in k8s?