K8s actual combat (I) | initialization of production level k8s high availability cluster for operation management

Time:2021-9-22

preface

In May, when the flowers bloom in spring, the epidemic situation is basically over, which is worth celebrating. Let’s fight in this beautiful day   K8s high availability load balancing cluster.

Update history

Platform environment

Software information

  • CentOS Linux release 7.7.1908 (Kernel 3.10.0-1062.18.1.el7.x86_64)
  • Docker CE 18.09.9
  • Kubernetes v1.18.2
  • Calico v3.8
  • Keepalived v1.3.5
  • HAproxy v1.5.18

Hardware information

host name ip
master01 192.168.10.12
master02 192.168.10.13
master03 192.168.10.14
work01 192.168.10.15
work02 192.168.10.16
work03 192.168.10.17
VIP 192.168.10.19

Cluster configuration

initialization

Both master and worker execute

# cat >> /etc/hosts << EOF
192.168.10.12    master01
192.168.10.13    master02
192.168.10.14    master03
192.168.10.15    work01
192.168.10.16    work02
192.168.10.17    work03

EOF
#Turn off firewall
systemctl stop firewalld
systemctl disable firewalld

#Turn off SELinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

#Close swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

#Install WGet
yum install wget -y

#Backup
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

#Alibaba cloud Yum source
wget -O /etc/yum.repos.d/CentOS-Base.repo  http://mirrors.aliyun.com/repo/Centos-7.repo#  Get Alibaba cloud EPEL source
wget -O /etc/yum.repos.d/epel.repo  http://mirrors.aliyun.com/repo/epel-7.repo#  Clean up the cache and create a new cache
yum clean all && yum makecache

#Renew
yum update -y

#Synchronization time
timedatectl
timedatectl set-ntp true

Install docker

Both master and worker are installed

#Installing docker CE
#Set up warehouse
#Install required packages
yum install -y yum-utils device-mapper-persistent-data lvm2

#Add docker installation source
yum-config-manager \
    --add-repo \
    http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

#Installing docker CE.
yum install -y containerd.io \
    docker-ce-18.09.9 \
    docker-ce-cli-18.09.9

#Start docker and add startup
systemctl start docker
systemctl enable docker

#Modify the CGroup driver of docker to SYSTEMd
#Modify to domestic source

cat > /etc/docker/daemon.json <<EOF
{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "registry-mirrors":[
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn",
        "https://registry.docker-cn.com"
    ]}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

Install kubedm, kubelet, kubectl

Both master and worker nodes execute

#Configure the yum source of k8s. It is best to use the official source
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg  https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

#Add configuration
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

#Loading
sysctl --system

#Install the latest stable version (v1.18.2) kubelet, kubedm and kubectl on the current date

yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 --disableexcludes=kubernetes

#Start and set kubelet
systemctl start kubelet
systemctl enable kubelet

Implement apiserver load balancing cluster with haproxy

All master nodes execute

yum install haproxy-1.5.18 -y

cat > /etc/haproxy/haproxy.cfg <<EOF
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend  k8s-api
    mode tcp
    option tcplog
    bind *:16443
    default_backend      k8s-api

backend k8s-api
    mode        tcp
    balance     roundrobin
    server master01 192.168.10.12:6443 check
    server master02 192.168.10.13:6443 check
    server master03 192.168.10.14:6443 check
EOF

Start haproxy on all master nodes

systemctl start haproxy
systemctl enable haproxy

Keep alive to implement apiserver high availability cluster

All master nodes execute

yum -y install keepalived psmisc

Configuration of keepalived on master01:

# cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id master01
   
   script_user root
   enable_script_security 
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 10
}

vrrp_instance VI_1 {
    state MASTER 
    interface ens192
    virtual_router_id 50
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.19
    }
    track_script {
        check_haproxy
    }
}
EOF

Configuration of keepalived on master02:

# cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id master02
   
   script_user root
   enable_script_security 
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 10
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens192
    virtual_router_id 50
    priority 98
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.19
    }
    track_script {
        check_haproxy
    }
}
EOF

Configuration of keepalived on master03:

# cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id master03
   
   script_user root
   enable_script_security 
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 10
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens192
    virtual_router_id 50
    priority 96
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.19
    }
    track_script {
        check_haproxy
    }
}
EOF

All master nodes execute

service keepalived start
systemctl enable keepalived

be careful

  • vrrp_ The parameter weight in script must be greater than the difference between the priority of master and backup
  • vrrp_ The parameter nopreempt in instance can prevent the master from automatically failback after recovery

Create k8s cluster

Before initialization, you need to set up hosts resolution
MASTER_ IP   Address of VIP
APISERVER_ NAME   Domain name for VIP

export MASTER_IP=192.168.10.19
export APISERVER_NAME=k8s.api
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts

Execute kubedm init on master01 to initialize

kubeadm init \
 --apiserver-advertise-address 0.0.0.0 \
 --apiserver-bind-port 6443 \
 --cert-dir /etc/kubernetes/pki \
 --control-plane-endpoint k8s.api \
 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
 --kubernetes-version 1.18.2 \
 --pod-network-cidr 192.10.0.0/16 \
 --service-cidr 192.20.0.0/16 \
 --service-dns-domain cluster.local \
 --upload-certs

Note that the first two lines of kubedm join in the saved results are used to add other master and worker nodes to the cluster.

Load environment variables

Executed on master01 for cluster management

If under root

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source .bash_profile

If you are not a root user

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install pod network components

Execute on master01

#Get profile
mkdir calico && cd calico
wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

#Modify profile
vi calico.yaml
#Find 192.168.0.0/16 and modify it to 192.10.0.0/16

#Deploy pod network components
kubectl apply -f calico.yaml

View the status of the pod in real time

watch kubectl get pods --all-namespaces -o wide

Add other master nodes to k8s cluster

Execute on other master nodes

The execution result of kubedm init on master01 contains the instruction information of join

Port changed from 6443 to 16443

export MASTER_IP=192.168.10.19
export APISERVER_NAME=k8s.api
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
kubeadm join k8s.api:16443 --token ztjux9.2tau56zck212j9ra \
    --discovery-token-ca-cert-hash sha256:a2b552266902fb5f6620330fc1a6638a9cdd6fec3408edba1082e6c8389ac517 \
    --control-plane --certificate-key 961494e7d0a9de0219e2b0dc8bdaa9ca334ecf093a6c5f648aa34040ad39b61a
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source .bash_profile

Add all worker nodes to k8s cluster

Execute on the worker node

The execution result of kubedm init on master01 contains the instruction information of join

Port changed from 6443 to 16443

export MASTER_IP=192.168.10.19
export APISERVER_NAME=k8s.api
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
kubeadm join k8s.api:16443 --token ztjux9.2tau56zck212j9ra \
    --discovery-token-ca-cert-hash sha256:a2b552266902fb5f6620330fc1a6638a9cdd6fec3408edba1082e6c8389ac517

Viewing clusters on master01

watch kubectl get nodes -o wide

All are ready, indicating that the cluster installation is successful.

Destructive test

  • Stop the haproxy of master01
  • Shut down the master01 machine

You can see that the VIP can drift to master02

Then you can do the same operation on master02 to observe whether the VIP can float to master03

Conclusion

Today, I built a k8s high availability load balancing cluster, which is my actual operation record.

So have you found a place that can be further optimized.

I’ll give you a bowl of chicken soup

  • The sooner you encounter the problem, the better. If you don’t encounter the problem, it means that your problem is bigger.

Reference articles

https://wsgzao.github.io/post…

https://www.kubernetes.org.cn…

Contact me

WeChat official account: zuolinux_ com

K8s actual combat (I) | initialization of production level k8s high availability cluster for operation management