Etcd cluster of distributed kV storage system

Time:2021-9-20

What is etcd?

Etcd is a highly available distributed key value database, which can be used for service discovery. Etcd adopts raft consistency algorithm and is implemented based on go language. It is characterized by simplicity and ease of use. The so-called simplicity and ease of use refers to simple installation and configuration and the provision of HTTP / HTTPS interface; Security means that etcd supports SSL certificate authentication and peer-to-peer certificate authentication among cluster nodes; Two way certificate authentication between client and server; Reliable means etcd uses raft protocol to realize the availability and consistency of distributed system data; Etcd mainly has two versions V2 and V3; The APIs of V2 and V3 are incompatible with each other, so when we install multiple versions of etcd on the same server, we need to use etcdctl_ API is specified by this environment variable;

Etcd cluster deployment

Environmental preparation

Host name IP address
master01.k8s.org 192.168.0.41
master02.k8s.org 192.168.0.42
master03.k8s.org 192.168.0.43

 

Analysis of hosts file of each host

[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.99 time.test.org time-node
192.168.0.41  master01 master01.k8s.org etcd01 etcd01.k8s.org 
192.168.0.42  master02 master02.k8s.org etcd02 etcd02.k8s.org
192.168.0.43  master03 master03.k8s.org etcd03 etcd03.k8s.org
192.168.0.44  node01 node01.k8s.org
192.168.0.45  node02 node02.k8s.org
192.168.0.46  node03 node03.k8s.org
[[email protected] ~]# 

Turn off the firewalld service of each host

[[email protected] ~]# systemctl stop firewalld
[[email protected] ~]# systemctl disable firewalld
[[email protected] ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[[email protected] ~]# 

Time synchronization between hosts

[[email protected] ~]# grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server time.test.org iburst
# Serve time even if not synchronized to any NTP server.
[[email protected] ~]# systemctl restart chronyd.service 
[[email protected] ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-01-30 15:41:25 CST; 11s ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
  Process: 1411 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 1407 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1409 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─1409 /usr/sbin/chronyd

Jan 30 15:41:25 master01.k8s.org systemd[1]: Stopped NTP client/server.
Jan 30 15:41:25 master01.k8s.org systemd[1]: Starting NTP client/server...
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI...+DEBUG)
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: commandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: generatecommandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: Frequency -25.600 +/- 2.450 ppm read from /var/lib/chrony/drift
Jan 30 15:41:25 master01.k8s.org systemd[1]: Started NTP client/server.
Jan 30 15:41:29 master01.k8s.org chronyd[1409]: Selected source 192.168.0.99
Hint: Some lines were ellipsized, use -l to show in full.
[[email protected] ~]# 

Tip: the cluster can use its own time service, point the server in chrony.conf to the corresponding time server, and then restart chronyd; Of course, you can also use the public time server on the Internet; In short, a service works in a cluster, and its time synchronization is very important;

SSH mutual trust among hosts

[[email protected] ~]# ssh master02
Last login: Sat Jan 30 15:34:33 2021 from master01
[[email protected] ~]# exit
logout
Connection to master02 closed.
[[email protected] ~]# ssh master03
Last login: Sat Jan 30 15:34:37 2021 from master01
[[email protected] ~]# exit
logout
Connection to master03 closed.
[[email protected] ~]# 

Tip: for the configuration of SSH mutual trust, please refer to my blog:https://www.cnblogs.com/qiuhom-1874/p/11783371.html; The main purpose of SSH mutual trust among hosts is to facilitate the synchronization of files among components; After making the above preparations, we can download the etcd binary package for etcd cluster deployment; It should be noted that there are etcd RPM packages in the extras warehouse on centos7. We can use Yum to install them; However, the version in the extras repository is not up-to-date. If you want to use the latest version, you need to download the latest etcd binary package from the official GitHub repository for deployment; There is no special difference between the two deployment methods; If the version requirements are not particularly new, it is recommended to use Yum to install;

Download etcd binary package

[[email protected] ~]#wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
--2021-01-30 15:46:18--  https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
Resolving github.com (github.com)... 52.192.72.89
Connecting to github.com (github.com)|52.192.72.89|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-01-30 15:46:19--  https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.111.154, 185.199.109.154, 185.199.108.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17373058 (17M) [application/octet-stream]
Saving to: ‘etcd-v3.4.14-linux-amd64.tar.gz’

100%[=================================================================================================>] 17,373,058  24.9MB/s   in 0.7s   

2021-01-30 15:46:20 (24.9 MB/s) - ‘etcd-v3.4.14-linux-amd64.tar.gz’ saved [17373058/17373058]
[[email protected] ~]#

Unzip the etcd binary package

[[email protected] ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz
[[email protected] ~]# tar xf etcd-v3.4.14-linux-amd64.tar.gz  -C /usr/local/src/
[[email protected] ~]# cd /usr/local/src/
[[email protected] src]# ls
etcd-v3.4.14-linux-amd64
[[email protected] src]# cd etcd-v3.4.14-linux-amd64/
[[email protected] etcd-v3.4.14-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[[email protected] etcd-v3.4.14-linux-amd64]# 

Soft connect etcd and etcdctl to the path environment variable

[[email protected] etcd-v3.4.14-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[[email protected] etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcd /usr/bin/
[[email protected] etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl /usr/bin/
[[email protected] etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcd
lrwxrwxrwx 1 root root 44 Jan 30 15:59 /usr/bin/etcd -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcd
[[email protected] etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcdctl 
lrwxrwxrwx 1 root root 47 Jan 30 15:59 /usr/bin/etcdctl -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl
[[email protected] etcd-v3.4.14-linux-amd64]# 

Write the etcd.service unit file

[[email protected] ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"
Type=notify

[Install]
WantedBy=multi-user.target
[[email protected] ~]# 

Provide the etcd environment variable loading file / etc / etcd / etcd.conf file

[[email protected] ~]# mkdir /etc/etcd/
[[email protected] ~]# cd /etc/etcd/
[[email protected] etcd]# vim etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="default"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_AUTO_TLS="false"
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
"etcd.conf" [New] 69L, 1686C written                                                                                     
[[email protected] etcd]# 

Change etcd.conf file

Tip: etcd_ DATA_ Dir is used to specify etcd data directory; ETCD_ LISTEN_ PEER_ URLs is used to specify the URL address of cluster node communication listening; ETCD_ LISTEN_ CLIENT_ URLs user specifies the URL address when the client connection is used; ETCD_ Name user specifies the name of the current node etcd instance; ETCD_ INITIAL_ ADVERTISE_ PEER_ URLs is used to specify the URL address of the cluster transaction advertisement; ETCD_ ADVERTISE_ CLIENT_ URLs user specifies the URL address of client transaction notification; ETCD_ INITIAL_ Cluster user specifies cluster members. One member consists of member name = URL address corresponding to inter cluster communication, and multiple members are separated by commas;

Create the / var / lib / etcd directory

[[email protected] etcd]# mkdir /var/lib/etcd/
[[email protected] etcd]# ll -d /var/lib/etcd/
drwxr-xr-x 2 root root 6 Jan 30 16:20 /var/lib/etcd/
[[email protected] etcd]# 

Copy the / usr / bin / etcd and etcdctl binaries on master01 to the / usr / bin / directories of master02 and master03

[[email protected] etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master02:/usr/bin/
etcd                                                                                                     100%   23MB  43.6MB/s   00:00    
etcdctl                                                                                                  100%   17MB  49.1MB/s   00:00    
[[email protected] etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master03:/usr/bin/
etcd                                                                                                     100%   23MB  42.2MB/s   00:00    
etcdctl                                                                                                  100%   17MB  56.8MB/s   00:00    
[[email protected] etcd]# 

Copy etcd.service on master01 to / usr / lib / SYSTEMd / system directory of master02 and master03

[[email protected] etcd]# scp /usr/lib/systemd/system/etcd.service master02:/usr/lib/systemd/system/
etcd.service                                                                                             100%  417   165.1KB/s   00:00    
[[email protected] etcd]# scp /usr/lib/systemd/system/etcd.service master03:/usr/lib/systemd/system/
etcd.service                                                                                             100%  417   175.7KB/s   00:00    
[[email protected] etcd]# 

Create the / etc / etcd / directory and / var / lib / etcd / directory on master02 and master03

[[email protected] etcd]# ssh master02 'mkdir /etc/etcd/ && mkdir /var/lib/etcd'
[[email protected] etcd]# ssh master03 'mkdir /etc/etcd/ && mkdir /var/lib/etcd' 
[[email protected] etcd]# 

Copy the etcd.conf file on master01 to the / etc / etcd / directory of master02 and master03

[[email protected] etcd]# scp /etc/etcd/etcd.conf master02:/etc/etcd/
etcd.conf                                                                                                100% 1749   743.3KB/s   00:00    
[[email protected] etcd]# scp /etc/etcd/etcd.conf master03:/etc/etcd/
etcd.conf                                                                                                100% 1749   824.2KB/s   00:00    
[[email protected] etcd]# 

Modify the / etc / etcd / etcd.conf file on master02

Modify the / etc / etcd / etcd.conf file on master03

The configuration files, related users and directories of the three nodes are ready. Next, we will reload the configuration file of SYSTEMd and load the etcd.service file

[root[email protected] ~]# systemctl daemon-reload
[[email protected] ~]# ssh master02 'systemctl daemon-reload'
[[email protected] ~]# ssh master03 'systemctl daemon-reload' 
[[email protected] ~]# 

Start etcd

[[email protected] ~]# systemctl start etcd
[[email protected] ~]# 

Tip: run the above command on each node to start etcd; The first started node will be blocked because etcd works in a cluster mode. It must have enough votes to work normally. If there are three cluster nodes, at least two nodes can start etcd normally;

Verify: check whether ports 2379 and 2380 of each node are listening?

[[email protected] ~]# ss -tnl
State      Recv-Q Send-Q                                Local Address:Port                                               Peer Address:Port              
LISTEN     0      128                                    192.168.0.41:2379                                                          *:*                  
LISTEN     0      128                                    192.168.0.41:2380                                                          *:*                  
LISTEN     0      128                                               *:22                                                            *:*                  
LISTEN     0      100                                       127.0.0.1:25                                                            *:*                  
LISTEN     0      128                                              :::22                                                           :::*                  
LISTEN     0      100                                             ::1:25                                                           :::*                  
[[email protected] ~]# ssh master02 'ss -tnl'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.42:2379                     *:*                  
LISTEN     0      128    192.168.0.42:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[[email protected] ~]# ssh master03 'ss -tnl' 
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.43:2379                     *:*                  
LISTEN     0      128    192.168.0.43:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[[email protected] ~]# 

Verify: use etcdctl to view the cluster status

[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  endpoint status
192.168.0.41:2379, b8b747c74aaea686, 3.4.14, 4.5 MB, true, false, 13, 2163, 2163, 
192.168.0.42:2379, b3504381e8ba3cb, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, 
192.168.0.43:2379, f572fdfc5cb68406, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, 
[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  member list
b3504381e8ba3cb, started, etcd02, http://etcd02:2380, http://etcd02:2379, false
b8b747c74aaea686, started, etcd01, http://etcd01:2380, http://etcd01:2379, false
f572fdfc5cb68406, started, etcd03, http://etcd03:2380, http://etcd03:2379, false
[[email protected] ~]# 

Tip: if you can list cluster members and view the status of cluster members, it means that the etcd cluster is working normally;

Verification: write data to any node of etcd to see if it can be written normally?

[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  put name "test"
OK
[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  get name 
name
test
[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  del name    
1
[[email protected] ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  get name 
[[email protected] ~]# 

Tip: you can write data to etcd cluster normally by using etcdctl tool;

Generate certificate for etcd cluster

Install git tool on a node

[[email protected] ~]# yum install git -y 

Script tool for cloning generated certificates

[[email protected] ~]# git clone https://github.com/iKubernetes/k8s-certs-generator.git
Cloning into 'k8s-certs-generator'...
remote: Enumerating objects: 58, done.
remote: Total 58 (delta 0), reused 0 (delta 0), pack-reused 58
Unpacking objects: 100% (58/58), done.
[[email protected] ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz  k8s-certs-generator
[[email protected] ~]# cd k8s-certs-generator/
[[email protected] k8s-certs-generator]# ls
etcd-certs-gen.sh  gencerts.sh  k8s-certs-gen.sh  openssl.conf  README.md
[[email protected] k8s-certs-generator]# 

Use the gencerts.sh script to generate the certificates required for etcd

[[email protected] k8s-certs-generator]# sh gencerts.sh -h
Usage: ./gencerts.sh etcd|k8s
[[email protected] k8s-certs-generator]# sh gencerts.sh etcd
Enter Domain Name [ilinux.io]: k8s.org
Generating RSA private key, 4096 bit long modulus
.......++
.................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
.................................................+++
.........................+++
e is 65537 (0x10001)
Generating etcd/pki/peer.csr
Generating RSA private key, 2048 bit long modulus
...........................................................................................................................................+++
...............+++
e is 65537 (0x10001)
Generating etcd/pki/server.csr
Generating RSA private key, 2048 bit long modulus
..............................................................+++
............................+++
e is 65537 (0x10001)
Generating etcd/pki/apiserver-etcd-client.csr
Generating RSA private key, 2048 bit long modulus
............+++
.................................+++
e is 65537 (0x10001)
Generating etcd/pki/client.csr
Generating etcd/pki/peer.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Jan 30 10:46:52 2021 GMT
            Not After : Jan 28 10:46:52 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                FC:BA:D7:73:4E:C7:1D:9D:73:12:E3:60:96:5B:69:58:CE:4F:14:FD
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
                DirName:/CN=etcd-ca
                serial:BE:88:C0:B5:81:5D:6D:D6

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:52 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                1C:BE:22:C0:B7:5F:03:39:5C:E0:FC:47:88:8D:3A:FC:27:FA:0E:BC
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
                DirName:/CN=etcd-ca
                serial:BE:88:C0:B5:81:5D:6D:D6

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/apiserver-etcd-client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4098 (0x1002)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Client
            Netscape Comment: 
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier: 
                FD:52:EA:9F:84:72:35:46:9A:33:71:DE:D0:41:E6:8D:89:C0:62:AE
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4099 (0x1003)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Client
            Netscape Comment: 
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier: 
                6B:31:50:84:00:9E:0F:6E:B8:56:7A:C1:57:82:F4:BB:12:57:52:B2
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[[email protected] k8s-certs-generator]# ls
etcd  etcd-certs-gen.sh  gencerts.sh  k8s-certs-gen.sh  openssl.conf  README.md
[[email protected] k8s-certs-generator]# ls etcd
patches  pki
[[email protected] k8s-certs-generator]# ls etcd/pki/
apiserver-etcd-client.crt  apiserver-etcd-client.key  ca.crt  ca.key  client.crt  client.key  peer.crt  peer.key  server.crt  server.key
[[email protected] k8s-certs-generator]# 

Tip: server.crt and server.key are used for certificates and keys of etcd server; Peer.crt and peer.key are used for certificates and keys required for inter node authentication within the cluster; Client.crt and client.key are used to connect the certificate and key required by the client to the server; Ca.crt and ca.key are the certificates and keys used to authenticate and connect the CA trusted by the server within the cluster;

Copy certificate files to other nodes

[[email protected] k8s-certs-generator]# cp -a etcd/pki/ /etc/etcd/
[[email protected] k8s-certs-generator]# cd /etc/etcd/
[[email protected] etcd]# ls
etcd.conf  pki
[[email protected] etcd]# scp -r pki/ master02:/etc/etcd/
ca.key                                                                                                                  100% 3247     1.8MB/s   00:00    
ca.crt                                                                                                                  100% 1814     1.2MB/s   00:00    
peer.key                                                                                                                100% 1679     1.1MB/s   00:00    
server.key                                                                                                              100% 1679     1.2MB/s   00:00    
apiserver-etcd-client.key                                                                                               100% 1675     1.3MB/s   00:00    
client.key                                                                                                              100% 1675     1.1MB/s   00:00    
peer.crt                                                                                                                100% 1659    75.0KB/s   00:00    
server.crt                                                                                                              100% 1647   917.8KB/s   00:00    
apiserver-etcd-client.crt                                                                                               100% 1570     1.2MB/s   00:00    
client.crt                                                                                                              100% 1570   902.2KB/s   00:00    
[[email protected] etcd]# scp -r pki/ master03:/etc/etcd/
ca.key                                                                                                                  100% 3247     1.1MB/s   00:00    
ca.crt                                                                                                                  100% 1814   695.0KB/s   00:00    
peer.key                                                                                                                100% 1679   621.6KB/s   00:00    
server.key                                                                                                              100% 1679   657.1KB/s   00:00    
apiserver-etcd-client.key                                                                                               100% 1675   950.4KB/s   00:00    
client.key                                                                                                              100% 1675     1.0MB/s   00:00    
peer.crt                                                                                                                100% 1659   916.3KB/s   00:00    
server.crt                                                                                                              100% 1647     1.0MB/s   00:00    
apiserver-etcd-client.crt                                                                                               100% 1570   850.8KB/s   00:00    
client.crt                                                                                                              100% 1570   872.7KB/s   00:00    
[[email protected] etcd]# 

Configure etcd to provide services based on HTTPS protocol

Configure etcd on master01 to enable certificate authentication

[[email protected] etcd]# cat etcd.conf 
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://192.168.0.41:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.41:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="etcd01"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd01:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd01:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
ETCD_CERT_FILE="/etc/etcd/pki/server.crt"
ETCD_KEY_FILE="/etc/etcd/pki/server.key"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/etcd/pki/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
[[email protected] etcd]# 

Tip: etcd_ CERT_ File is used to specify the path of etcd server certificate file; ETCD_ KEY_ File user specifies the key file path corresponding to the server certificate file; ETCD_ CLIENT_ CERT_ Auth user specifies whether to enable client certificate authentication; ETCD_ TRUSTED_ CA_ File user specifies the CA certificate file of client authentication trust;

ETCD_ AUTO_ TLS is used to specify whether to automatically generate certificate files; ETCD_ PEER_ CERT_ File is used to specify the file path of peer-to-peer certificate between clusters; ETCD_ PEER_ KEY_ File is used to specify the key file corresponding to the peer-to-peer certificate between clusters; ETCD_ PEER_ CLIENT_ CERT_ Auth is used to specify whether peer certificate authentication is enabled; ETCD_ PEER_ TRUSTED_ CA_ File is used to specify the CA certificate trusted by peer certificate authentication; ETCD_ PEER_ AUTO_ TLS is used to specify whether to automatically generate peer certificates;

Modify the / etc / etcd / etcd.conf file, etcd01 to etcd01.k8s.org, etcd02 to etcd02.k8s.org, etcd03 to etcd03.k8s.org, and HTTP to HTTPS

Stop the etcd service and delete all files in the / var / lib / etcd / directory

[[email protected] etcd]# systemctl stop etcd
[[email protected] etcd]# rm -rf /var/lib/etcd/*
[[email protected] etcd]# ll /var/lib/etcd/
total 0
[[email protected] etcd]# 

Configure master02 to enable certificate authentication, modify the corresponding HTTP to HTTPS, and modify the corresponding short format name to a name similar to etcd01.k8s.org

Stop the etcd service and delete all files in the / var / lib / etcd / directory

[[email protected] ~]# systemctl stop etcd
[[email protected] ~]# rm -rf /var/lib/etcd/*
[[email protected] ~]# ll /var/lib/etcd/
total 0
[[email protected] ~]# 

Configure master03 to enable certificate authentication, modify the corresponding HTTP to HTTPS, and modify the corresponding short format name to long format name

Stop the etcd service and delete all files under / var / lib / etcd

[[email protected] ~]# systemctl stop etcd
[[email protected] ~]# rm -rf /var/lib/etcd/*
[[email protected] ~]# ll /var/lib/etcd/
total 0
[[email protected] ~]# 

Start etcd on each node

[[email protected] etcd]# systemctl start etcd
[[email protected] etcd]# 

Tip: if etcd on all three nodes can be started normally, there is no problem with our configuration file;

Verify: check whether etcd services of all nodes are started normally and listen to corresponding ports?

[[email protected] etcd]# ss -tnl
State      Recv-Q Send-Q                                Local Address:Port                                               Peer Address:Port              
LISTEN     0      128                                    192.168.0.41:2379                                                          *:*                  
LISTEN     0      128                                    192.168.0.41:2380                                                          *:*                  
LISTEN     0      128                                               *:22                                                            *:*                  
LISTEN     0      100                                       127.0.0.1:25                                                            *:*                  
LISTEN     0      128                                              :::22                                                           :::*                  
LISTEN     0      100                                             ::1:25                                                           :::*                  
[[email protected] etcd]# ssh master02 'ss -tnl'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.42:2379                     *:*                  
LISTEN     0      128    192.168.0.42:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[[email protected] etcd]# ssh master03 'ss -tnl' 
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.43:2379                     *:*                  
LISTEN     0      128    192.168.0.43:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[[email protected] etcd]# 

Verifying: viewing cluster members

[roo[email protected] etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379,https://etcd02.k8s.org:2379,https://etcd03.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" endpoint status
https://etcd01.k8s.org:2379, 61d91b7ed8f88f32, 3.4.14, 20 kB, true, false, 6, 9, 9, 
https://etcd02.k8s.org:2379, ef13441fdfe8af38, 3.4.14, 20 kB, false, false, 6, 9, 9, 
https://etcd03.k8s.org:2379, f11ed09b6567910f, 3.4.14, 20 kB, false, false, 6, 9, 9, 
[[email protected] etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" member list
61d91b7ed8f88f32, started, etcd01.k8s.org, https://etcd01.k8s.org:2380, https://etcd01.k8s.org:2379, false
ef13441fdfe8af38, started, etcd02.k8s.org, https://etcd02.k8s.org:2380, https://etcd02.k8s.org:2379, false
f11ed09b6567910f, started, etcd03.k8s.org, https://etcd03.k8s.org:2380, https://etcd03.k8s.org:2379, false
[[email protected] etcd]# 

Tip: now etcd has SSL authentication enabled. The client must carry the corresponding client certificate, private key file and the CA certificate trusted by the corresponding authentication before it can normally access the etcd cluster; It should be noted that the specified endpoints need to be given in domain name format, and the given IP address cannot pass the authentication normally;