Redhat6, Rsync + inotify real time backup

Time:2021-6-10

Rsync is a tool for file synchronization and data transmission in Linux system,

It has four application modes: 1. Local mode 2. Remote shell mode 3. Rsync list mode 4. Server mode

Here we build the server mode, which is based on the C / S mode. We need Rsync to enable a daemons in the background, and this process is permanent on the Rsync server to receive file transfer requests.

 

For example, suppose there is a website, the data will be updated every day, in order to ensure data security, establish a remote disaster recovery system. However, Rsync needs to be triggered before it can be updated. Although it can be triggered regularly by crontab, the Linux daemons, there is a certain time difference between the two triggering actions, which may lead to inconsistent data between the server and the client, and the data cannot be fully recovered in case of disaster.

Here we use the inotify tool, which is an asynchronous file monitoring file system. It monitors all kinds of changes in the file system. When there is any change in the file, Rsync synchronization will be triggered, which just solves the problem of real-time data synchronization.

Node name

purpose

IP

Web page path

server

Content publishing node

193.168.0.20

/webdata

web2

Service node

193.168.0.22

/web2data

web3

Service node

193.168.0.23

/web3data

Inotify is used to monitor the file system, so it must be installed in the content publishing node, and the service node does not need to install, only needs to install Rsync.

1. Configure Rsync on the server node

[[email protected] ~]# tar xzf apt/rsync-3.0.4.tar.gz
[[email protected] ~]# cd rsync-3.0.4/ 
[[email protected] rsync-3.0.4]# ./configure
[[email protected] rsync-3.0.4]# make && make install

1.2. If there is no rsyncd.con configuration file when installing Rsync, we can just build one ourselves

web2Rsyncd.conf of node

[[email protected] ~# vim /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = no
max connections = 10
strict modes = yes
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log

[web2]
path = /web2data
comment = web2 file
ignore errors
read only = no
write only = no
hosts allow = *
list = false
uid = root
gid = root
auth users = web2user
secrets file = /etc/web2.pass

1.3.web3Rsyncd.conf of node

[[email protected] ~# vim /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = no
max connections = 10
strict modes = yes
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log

[web3]
path = /web3data
comment = web3 file
ignore errors
read only = no
write only = no
hosts allow = *
list = false
uid = root
gid = root
auth users = web3user
secrets file = /etc/web3.pass

Auth user is a defined user, which has nothing to do with Linux system users. It is the user name used to connect the module.

Secrets file is a specified file containing “user name: password” format. The user name is the user name defined by auth user

[[email protected] ~]# more /etc/web2.pass 
web2user:admin
[[email protected] ~]# chmod 600 /etc/web2.pass

[[email protected] ~]# more /etc/web3.pass 
web3user:admin
[[email protected] ~]# chmod 600 /etc/web3.pass

1.4. Start Rsync daemons (web2, Web3)

[[email protected] ~]# /usr/local/bin/rsync –daemon
[[email protected] ~]# ps -ef |grep rsync
root      38588      1  0 Mar23 ?        00:00:00 /usr/local/bin/rsync --daemon
[ [email protected]  ~]#Echo "/ usr / local / bin / Rsync -- daemon" > > / etc / RC. Local # add self boot file

2. Configure content publishing point

2.1. Publishing point mainly synchronizes static files to two server nodes in real time. This process is implemented by a shell script

#!/bin/bash
host1=193.168.0.22
host2=193.168.0.23

src=/webdata/
dst1=web2
dst2=web3
user1=web2user
user2=web3user

/usr/local/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f%e' -e close_write,delete,cre
ate,attrib $src \
| while read files
	do
	/usr/local/bin/rsync -vzrtopg --delete --progress --password-file=/etc/server.pass $src [email protected]$h
ost1::$dst1
	/usr/local/bin/rsync -vzrtopg --delete --progress --password-file=/etc/server.pass $src [email protected]$h
ost2::$dst2
	echo "${files} was rsynced" >> /tmp/rsync.log 2>&1
done

2.2. Connect backup module password

[[email protected] ~]# more /etc/server.pass
admin

2.3. Here, for convenience, put the shell script in the webdata directory, named inotify.sh, and give the execution permission to run in the background

[[email protected] ~]# chmod 755 /webdata/inotify.sh
[[email protected] ~]# nohup /webdata/inotify.sh &

2.4. Add the system self startup file

[[email protected] ~]# echo "/webdata/inotify.sh &" >> /etc/rc.local

After all configurations are completed, you can add, delete, and write files to the website publishing node. You will see two server nodes

Follow the publishing node and update the file( If synchronization does not occur for the first time, wait a moment, the content publishing node needs to dock with the server node module. After synchronization occurs, the subsequent updates will be much faster.)

 

  • Sometimes we encounter such a situation: write a large file to the inotify monitoring directory. When it takes a period of time to write the file, inotify will continuously output the updated information of the file, which will continuously trigger Rsync synchronization and occupy a lot of system resources. In this case, the best way is to wait for the file to be completely written before triggering the update, You can modify “- e close”_ write,delete,create,atrrib”