Keep alive to realize high availability of httpd server

Time:2022-5-21

Introduction to keepalived

Keepalived software was originally designed for LVS load balancing software to manage and monitor the status of each service node in LVS cluster system. Later, it added VRRP function that can realize high availability. Therefore, in addition to managing LVS software, keepalived can also be used as high availability solution software for other services (such as nginx, haproxy, mysql, etc.).

Keepalived software mainly realizes the high availability function through VRRP protocol. VRRP is the abbreviation of virtual router redundancy protocol. The purpose of VRRP is to solve the problem of single point of failure of static routing. It can ensure that the whole network can run continuously when individual nodes are down.

Therefore, on the one hand, kept has the function of configuring and managing LVS, and also has the function of health inspection for the nodes under LVS. On the other hand, it can also realize the high availability of system network services.

Keepalived’s official website

Important functions of kept:

Keepalived has three important functions:

  • Manage load balancing software
  • Realize the health check of LVS cluster nodes
  • High availability as a system network service (failover)

Principle of keepalived high availability failover

Failover between keepalived high availability services is realized through VRRP (Virtual Router Redundancy Protocol).

When the keepalived service works normally, the primary master node will continuously send heartbeat messages (in the form of multicast) to the standby node to tell the standby backup node that it is still alive. When the primary master node fails, it cannot send heartbeat messages, so the standby node cannot continue to detect the heartbeat from the primary master node, so it calls its own takeover program, Take over the IP resources and services of the master node. When the primary master node recovers, the standby backup node will release the IP resources and services taken over by itself when the primary node fails and restore to the original standby role.

So, what is VRRP?
VRRP, the full name of which is virtual router redundancy protocol, is called virtual routing redundancy protocol in Chinese. The emergence of VRRP is to solve the single point of failure problem of static routing. VRRP gives the routing task to a VRRP router through a campaign mechanism.

Kept principle

Keepalived high availability architecture diagram
Keep alive to realize high availability of httpd server

Description of the working principle of kept

The communication between keepalived high availability pairs is through VRRP. Therefore, we have learned from VRRP:

  1. VRRP, the full name of which is virtual router redundancy protocol, is called virtual routing redundancy protocol in Chinese. VRRP appears to solve the single point of failure of static routing.
  2. VRRP gives the routing task to a VRRP router through a competitive protocol mechanism.
  3. VRRP uses IP multicast to realize the communication between high availability pairs.
  4. When working, the master node sends out the contract and the standby node receives the package. When the standby node cannot receive the data package sent by the master node, it starts the takeover program to take over the open source of the master node. There can be more than one standby node, which can compete through priority, but generally, there is a pair in the operation and maintenance of the keepalived system.
  5. VRRP uses encryption protocol to encrypt data, but keepalived officials still recommend configuring authentication type and password in clear text.
Next, let’s introduce the working principle of the keepalived service

Keepalived high availability communicates through VRRP. VRRP determines the active and standby through the election mechanism. The priority of the primary is higher than that of the standby. Therefore, when working, the primary will give priority to all resources, and the standby node is in the waiting state. When the primary hangs up, the standby node will take over the resources of the primary node and then provide services instead of the primary node.

Between keepalived services, only the master server will always send VRRP broadcast packets and tell the standby server that it is still alive. At this time, the standby server will not occupy the master. When the master is unavailable, that is, when the standby server cannot monitor the broadcast packets sent by the master, it will start relevant services to take over resources to ensure business continuity The fastest takeover speed can be less than 1 second.

Explanation of keepalived configuration file

Keepalived default profile

The main configuration file of keepalived is / etc / keepalived / keepalived conf

! Configuration File for keepalived

global_ Defs {// global configuration
   notification_ Email {// define the email address of the alarm recipient
     [email protected]
     [email protected]
     [email protected]
   }
   notification_ email_ from Alexandre. [email protected] //Define alarm sender mailbox
   smtp_ Server 192.168.200.1 // email server address
   smtp_ connect_ Timeout 30 // defines the mailbox timeout
   router_ id LVS_ Devel // defines the route identification information, which is unique in the LAN
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_ instance VI_ 1 {// define instance
    State master // specifies the initial state of the keepalived node. The optional value is master | backup
    Interface eth0 // the network card interface bound to the VRRP instance. The user sends the VRRP package
    virtual_ router_ ID 51 // the ID of the virtual route should be the same in the same cluster
    Priority 100 // defines the priority. The active and standby roles are determined by priority. The higher the priority, the higher the priority
    Nopreempt // set no preemption
    advert_ Int 1 // active / standby communication interval
    Authentication {// configure authentication
        auth_ Type pass // authentication method. Here is the password
        auth_ Pass 1111 // the keepalived configuration in the same cluster must be consistent here. It is recommended to use 8-bit random numbers
    }
    virtual_ IPAddress {// configure the VIP address to be used
        192.168.200.16
    }
}

virtual_ Server 192.168.200.16 1358 {// configure virtual server
    delay_ Loop 6 // health check interval
    lb_ Algo RR // LVS scheduling algorithm
    lb_ Kind NAT // LVS mode
    persistence_ Timeout 50 // persistence timeout, in seconds
    Protocol TCP // layer 4 protocol

    sorry_ Server 192.168.200.200 1358 // define the standby server. Use sorry when all RS fails_ Server to respond to the client

    real_ Server 192.168.200.2 1358 {// defines the server that actually processes requests
        Weight 1 // specify a weight for the server. The default value is 1
        HTTP_GET {
            url {
              path /testurl/test. JSP // specify the URL path to check
              Digest 640205b7b0fc66c1ea91c4633fac6334d // summary information
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl3/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            connect_ Timeout 3 // connection timeout
            nb_ get_ Retry 3 // get attempts
            delay_ before_ Retry 3 // how long is the delay before attempting
        }
    }

    real_server 192.168.200.3 1358 {
        weight 1
        HTTP_GET {
            url {
              path /testurl/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334c
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334c
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

Customize master profile

vrrp_ Instance segment configuration
These two are mutually exclusive during setting. Only one configuration can exist

Nopreempt // set to no preemption. The default is preemption. When the high priority machine is restored, it will preempt the low priority machine and become the master. If not, the low priority machine is allowed to continue to become the master, even if the high priority machine has been online. If you want to use this function, the initialization status must be backup.

preempt_ Delay // sets the preemption delay. The unit is seconds, the range is 0 --- 1000, and the default is 0 The number of seconds after the low priority master is found.

vrrp_ Script segment configuration

//Function: add a periodically executed script. The exit status code of the script will be called by all its VRRP instance records.
//Note: at least one VRRP instance calls it and the priority cannot be 0 The priority range is 1-254

vrrp_script <SCRIPT_NAME> {
          ...
    }

//Option Description:
Script "/ path / to / somewhere" // specify the path of the script to execute.
Interval < integer > // specify the interval between script execution. The unit is seconds. The default is 1s.
Timeout < integer > // specify the number of seconds after which the script is considered to have failed to execute.
Weight < - 254 --- 254 > // adjust the priority. The default is 2
Rise < integer > // how many successful executions are considered successful.
Fall < integer > // how many times does the execution fail before it is considered failed.
User < username > [groupname] // the user and group running the script.
init_ Fail // it is assumed that the initial state of the script is failure.

//Weight Description: 
1. If the script is executed successfully (the exit status code is 0) and the weight is greater than 0, the priority increases.
2. If the script execution fails (the exit status code is non-0) and the weight is less than 0, the priority decreases.
3. In other cases, priority remains unchanged.

real_ Server segment configuration

Weight < int > // assign weight to the server. The default is 1
inhibit_ on_ Failure // when the server health check fails, set its weight to 0\
                        //Instead of removing from virtual server
notify_ Up < string > // the script to be executed when the server health check is successful
notify_ // when the health check of the server fails, < string >
Uthreshold < int > // the maximum number of connections to this server
Lthreshold < int > // the minimum number of connections to this server

tcp_ Check segment configuration

connect_ IP < IP address > // the IP address of the connection. The default is the IP address of the real server
connect_ Port < port > // the connected port. The default is the port of the real server
Bindto < IP address > // the address of the interface initiating the connection.
bind_ Port < port > // the source port that initiates the connection.
connect_ Timeout < int > // connection timeout. The default is 5S.
Fwmark < integer > // use fwmark to mark all outgoing inspection packets.
Warmup < int > // specify a random delay, with a maximum of N seconds. Prevents network congestion. If 0, the function is turned off.
Retry < init > // number of retries. The default is once.
delay_ before_ Retry < int > // the default is 1 second. How many seconds to delay before retrying.

example

global_defs {
    router_id LVS_Server
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 150
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass wangqing
    }
    virtual_ipaddress {  
        172.16.12.250 dev ens33
    }
}
virtual_server 172.16.12.250 80 {
    delay_loop 3
    lvs_sched rr
    lvs_method DR
    protocol TCP
    real_server 172.16.12.129 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 172.16.12.130 8080 {
        weight 1
        TCP_CHECK {
            connect_port 8080
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

Keep alive to realize high availability of httpd server

Environmental description

system information host name IP
Redhat8.2 master 192.168.182.141
Redhat8.2 backup 192.168.182.142

The VIP of this high availability service is 192.168.182.100

Configure keepalived on the master

First, turn off the firewall and SELinux
[[email protected] ~]# systemctl disable --now firewalld
[[email protected] ~]# setenforce 0
setenforce: SELinux is disabled

Configure network source

[[email protected] ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo

Install EPEL source

[[email protected] ~]# yum -y install epel-release

Install keepalived

[[email protected] ~]# yum -y install keepalived

View files generated by installation

[[email protected] ~]# rpm -ql keepalived
// etc / kept // configuration directory
/etc/keepalived/keepalived. Conf // main configuration file
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived. Service // service control file

Use the same method to install keepalived on the standby server

Turn off firewall and SELinux
[[email protected] ~]# systemctl disable --now firewalld
[[email protected] ~]# setenforce 0
setenforce: SELinux is disabled

Configure network source

[[email protected] ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo

Install EPEL source

[[email protected] ~]# yum -y install epel-release

Install keepalived

[[email protected] ~]# yum -y install keepalived

Install the httpd service on the active and standby machines respectively

[[email protected] ~]# yum -y install httpd
Start the service and set the startup self startup
[[email protected] ~]# systemctl enable --now httpd
[[email protected] html]# pwd
/var/www/html
[[email protected] html]# cat index.html 
master
Keep alive to realize high availability of httpd server

Operate on backup
[[email protected] ~]# yum -y install httpd
[[email protected] html]# pwd
/var/www/html
[[email protected] html]# cat index.html 
backup

[[email protected] ~]# systemctl enable --now httpd
Keep alive to realize high availability of httpd server

Configure keepalived on the master

[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id lb01
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 121388   
    }
    virtual_ipaddress {
        192.168.182.100
    }
}

virtual_server 192.168.182.100 80 {   
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.182.141 80 {   
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.182.142 80 {   
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}


[[email protected] keepalived]# systemctl enable --now keepalived.service

Configure standby keepalived

[[email protected] keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 121388
    }
    virtual_ipaddress {
        192.168.182.100
    }
}

virtual_server 192.168.182.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.182.141 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.182.142 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

[[email protected] keepalived]# systemctl enable --now keepalived.service

View VIP on master

[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:18:28:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.182.141/24 brd 192.168.182.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.182.100/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe18:287e/64 scope link 
       valid_lft forever preferred_lft forever

View VIP on slave

[[email protected] keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:f6:86:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.182.142/24 brd 192.168.182.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef6:8667/64 scope link 
       valid_lft forever preferred_lft forever

Keepalived monitors the status of the httpd load balancer through scripts

[[email protected] ~]# mkdir /scripts
[[email protected] ~]# cd /scripts/
[[email protected] scripts]# vim check_ht.sh
[[email protected] scripts]# chmod +x check_ht.sh

[[email protected] scripts]# cat check_ht.sh 
#!/bin/bash
httpd_status=$(ps -ef | grep -Ev "grep|$0" | grep -w httpd | wc -l)
if [ $httpd_status -lt 1 ];then
        systemctl stop keepalived
fi

Configure the master keepalived file

[[email protected] scripts]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   router_id lb01
}

vrrp_script httpd_check {                               
    script "/scripts/check_h.sh"
    interval 1
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 121388   
    }
    virtual_ipaddress {
        192.168.182.100
    }
    track_script {
        httpd_check
    }
    notify_master "/scripts/notify.sh master"
    notify_backup "/scripts/notify.sh backup"
}

virtual_server 192.168.182.100 80 {   
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.182.141 80 {   
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.182.142 80 {   
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}


[[email protected] scripts]# systemctl restart keepalived.service

Configure standby keepalived

[[email protected] scripts]# cat notify.sh 
#!/bin/bash
VIP=$2
case "$1" in
    master)
        httpd_status=$(ps -ef | grep -Ev "grep|$0"|grep -w httpd | wc -l)
        if [ $httpd_status -lt 1 ];then
            systemctl start httpd
        fi
        sendmail
    ;;
    backup)
          httpd_status=$(ps -ef | grep -Ev "grep|$0" | grep -w httpd | wc -l)
          if [ $httpd_status -gt 0 ];then
                  systemctl stop httpd
          fi
    ;;
    *)
              echo "Usage:$0 master | backup VIP"
    ;;
esac

Configure the keepalived configuration of backup

[[email protected] ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 121388
    }
    virtual_ipaddress {
        192.168.182.100
    }
    notify_master "/scripts/notify.sh master" 
    notify_backup "/scripts/notify.sh backup"
}

virtual_server 192.168.182.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.182.141 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.182.142 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

[[email protected] scripts]# systemctl restart keepalived.service

Simulate main service downtime

[[email protected] scripts]# systemctl stop httpd.service
Keep alive to realize high availability of httpd server

Shut down the service upgraded to the primary backup machine, and the primary machine returns to the master

[[email protected] scripts]# systemctl stop httpd.service
[[email protected] ~]# systemctl start keepalived
Keep alive to realize high availability of httpd server