Use kubeeye to escort your k8s cluster

Time:2022-5-23

Author: kaliarch (Xue Lei), product leader of a cloud MSP service provider, familiar with enterprise level high availability / high concurrency architecture, including hybrid Cloud Architecture and remote disaster, proficient in enterprise Devops transformation and optimization, familiar with shell / Python / go and other development languages, familiar with kubernetes, docker, cloud native, micro service architecture, etc.

preface

Kubeeye is a kubernetes security and configuration problem detection tool, which is used for configuration detection of business applications deployed in k8s clusterOPANode deployment for clusterNode-Problem-DetectorAt the same time, in addition to the built-in predefined rules according to most common scenarios in the industry, the system also supports user-defined rules for cluster detection.

framework

Kubeeye obtains cluster diagnosis data by calling kubernetes API and matching the rules of matching keywords in resources and container syntax. See the architecture diagram for details.

The node detection needs to be installed on the host of the detected node.

characteristic

characteristic

  • Kubeeye reviews your workload yaml specification against industry best practices to help you stabilize your cluster.
  • Kubeeye can find the problems of your cluster control plane, including Kube apiserver / Kube Controller Manager / etcd, etc.
  • Kubeeye can help you detect various node problems, including memory / CPU / disk pressure, unexpected kernel error logs, etc.

Inspection items

Yes / no Inspection items describe level
PrivilegeEscalationAllowed Allow privilege escalation urgent
CanImpersonateUser Role / clusterrole has permission to disguise as other users warning
CanDeleteResources Role / clusterrole has permission to delete kubernetes resources warning
CanModifyWorkloads Role / clusterrole has permission to modify kubernetes resources warning
NoCPULimits The resource has no CPU usage limit set urgent
NoCPURequests No reserved CPU is set for the resource urgent
HighRiskCapabilities High risk functions are enabled, such as all / sys_ ADMIN/NET_ ADMIN urgent
HostIPCAllowed The host IPC is turned on urgent
HostNetworkAllowed The host network is turned on urgent
HostPIDAllowed The host PID is turned on urgent
HostPortAllowed Host port opened urgent
ImagePullPolicyNotAlways The mirror pull policy is not always warning
ImageTagIsLatest The mirror label is latest warning
ImageTagMiss Mirror has no label urgent
InsecureCapabilities Unsafe functions are enabled, such as kill / sys_ CHROOT/CHOWN warning
NoLivenessProbe Survival status check is not set warning
NoMemoryLimits The resource has no memory usage limit set urgent
NoMemoryRequests No reserved memory is set for the resource urgent
NoPriorityClassName Resource scheduling priority is not set notice
PrivilegedAllowed Running resources in privileged mode urgent
NoReadinessProbe Readiness check is not set warning
NotReadOnlyRootFilesystem The root file system is not set to be read-only warning
NotRunAsNonRoot There is no setting to prohibit starting processes as root warning
CertificateExpiredPeriod API server certificate will be checked for expiration date less than 30 days urgent
EventAudit Event check warning
NodeStatus Node status check warning
DockerStatus Docker status check warning
KubeletStatus Kubelet status check warning

deploy

Kubeeye itself is written in golang, and the compiled binary executable file can be used to install relevant components.

install

Binary installation

wget https://github.com/kubesphere/kubeeye/releases/download/v0.3.0/kubeeye-0.3.0-linux-amd64.tar.gz
tar -zxvf kubeeye-0.3.0-linux-amd64.tar.gz
mv kubeeye /usr/bin/

Source code compilation and installation

git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye 
make installke

Install NPD

For the detection of cluster node hosts, kubeeye adoptsNode-problem-Detector, it needs to be installed on the node host node. Kubeeye encapsulates the installation command and can be installed with one click.

⚠ Note: this will install NPD on your cluster, which is only required if you want a detailed node report.

[[email protected] ~]# kubeeye install -e npd
kube-system 	 ConfigMap 	 node-problem-detector-config 	 created
kube-system 	 DaemonSet 	 node-problem-detector 	 created

It mainly creates configmap and node problem detector daemon set of node problem detector config in Kube system namespace.

Running kubeeye in a cluster

In addition to the one-time use of tools, kubeeye is also an operator, which can run inside the cluster for long-term and continuous detection of the cluster.

Deploy kubeeye in kubernetes

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

View kubeeye patrol inspection results

$ kubectl get clusterinsight -o yaml

apiVersion: v1
items:
- apiVersion: kubeeye.kubesphere.io/v1alpha1
  kind: ClusterInsight
  metadata:
    name: clusterinsight-sample
    namespace: default
  spec:
    auditPeriod: 24h
  status:
    auditResults:
      auditResults:
      - resourcesType: Node
        resultInfos:
        - namespace: ""
          resourceInfos:
          - items:
            - level: waring
              message: KubeletHasNoSufficientMemory
              reason: kubelet has no sufficient memory available
            - level: waring
              message: KubeletHasNoSufficientPID
              reason: kubelet has no sufficient PID available
            - level: waring
              message: KubeletHasDiskPressure
              reason: kubelet has disk pressure
            name: kubeeyeNode

test

Command options

[[email protected] ~]# kubeeye -h
KubeEye finds various problems on Kubernetes cluster.

Usage:
  ke [command]

Available Commands:
  audit       audit resources from the cluster
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  install     A brief description of your command
  uninstall   A brief description of your command

Flags:
  -f, --config string         Specify the path of kubeconfig.
  -h, --help                  help for ke
      --kubeconfig string     Paths to a kubeconfig. Only required if out-of-cluster.
      --master --kubeconfig   (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.

It can be seen that kubeeye currently mainly supports two commands, one is install package, such as NPD, and the other is to execute audit to scan the configuration of cluster applications.

audit

[[email protected] ~]# kubeeye audit
KIND         NAMESPACE         NAME                                           MESSAGE
Deployment   dddd              jenkins-1644220286                             [NoCPULimits ImagePullPolicyNotAlways NoMemoryLimits NoPriorityClassName NotReadOnlyRootFilesystem NotRunAsNonRoot]
Deployment   jenkins           jenkins-1644220286                             [NoCPULimits ImagePullPolicyNotAlways NoMemoryLimits NoPriorityClassName NotReadOnlyRootFilesystem NotRunAsNonRoot]
Deployment   smartkm-api-k8s   velero                                         [ImageTagIsLatest NoLivenessProbe NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
DaemonSet    smartkm-api-k8s   restic                                         [ImageTagIsLatest NoLivenessProbe NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
Node                           minikube                                       [KernelHasNoDeadlock FilesystemIsNotReadOnly KubeletHasSufficientMemory KubeletHasNoDiskPressure KubeletHasSufficientPID]
Event        kube-system       node-problem-detector-dmsws.16d844532f662318   [Failed to pull image "k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Event        kube-system       node-problem-detector-dmsws.16d844532f66703e   [Error: ErrImagePull]
Event        kube-system       node-problem-detector-dmsws.16d84453351b8b19   [Error: ImagePullBackOff]

Add custom check rule

We use the command to view the predefined OPA check rules.

kubectl get cm -n kube-system node-problem-detector-config -oyaml

At the same time, you can also create custom inspection rules according to your own business.

  • Create OPA rule storage directory
mkdir opa
  • Add custom OPA rule file

Note: to check the OPA rule set for the workload, the package name must bekubeeye_workloads_regoTo check the OPA rule set by RBAC, the package name must bekubeeye_RBAC_regoThe package name must be the OPA rule set for the check nodekubeeye_nodes_rego

  • The following are the rules for checking the address of the mirror warehouse. Save the following rules to the rule fileimageRegistryRule.rego
package kubeeye_workloads_rego

deny[msg] {
    resource := input
    type := resource.Object.kind
    resourcename := resource.Object.metadata.name
    resourcenamespace := resource.Object.metadata.namespace
    workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"}
    workloadsType[type]

    not workloadsImageRegistryRule(resource)

    msg := {
        "Name": sprintf("%v", [resourcename]),
        "Namespace": sprintf("%v", [resourcenamespace]),
        "Type": sprintf("%v", [type]),
        "Message": "ImageRegistryNotmyregistry"
    }
}

workloadsImageRegistryRule(resource) {
    regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)
}
  • Run kubeeye with additional rules

Tip: kubeeye will read all files in the specified directory.regoEnd of file

kubeeye audit -p ./opa

Troubleshooting

  • NPD installation is abnormal. K8s is used by default gcr. IO, if the installation server cannot connect to the public network, you can use my image warehouse: 1832990 / node problem detector: v0 8.7。
  • Kubeye installation uses the default host$HOME/.kube/configFile. If the k8s config file does not exist, it cannot run normally.

Reference link

This article is composed of blog one article multi posting platformOpenWriterelease!