Rsync of Linux file synchronization tool

Time:2021-10-17

Learning background

1. Recently, the company’s projects are using Jenkins for automatic construction. Because Jenkins consumes performance during construction, a separate server is used as the construction server. However, I feel that the cost is too high. I use a single server to build it, and the server configuration cannot be too poor. I wanted to build and use a Jenkins service locally. However, there is a problem that webhooks cannot be used, so we choose the rotation training construction. In fact, it can be solved by intranet penetration. The example is still learning and the experiment is successful. Since the use is not very familiar, I will write an article to share later.

2. In Jenkins architecture, we need to configure a work directory, such as / root / Jenkins / workspace / Daodao_ System, but we need to use nginx to configure the project directory. It is certainly impossible to configure it to the working directory of Jenkins. Through understanding, the Rsync command of Linux can realize file synchronization. File synchronization here refers to synchronizing the files under Jenkins working directory to the directory specified by us (such as the project directory configured by nginx). The Rsync synchronization command has the advantages of high efficiency, low bandwidth consumption, and supports replication links, devices, owners, groups, permissions, etc.

Command introduction

Rsync is the most commonly used command to copy and synchronize files remotely (or locally). With the Rsync command, you can copy and synchronize local data remotely across directories, disks and networks. For example: data backup and mirroring between two Linux hosts. This article introduces the common usage of remote and local file transfer on Linux host. Rsync can be allowed without root account.

Rsync command features

1. Efficiently copy and synchronize data to the opposite end or to the local end.

2. It supports copying links, devices, owners, groups and permissions.

3. Faster than SCP (secure copy). Rsync uses the remote update protocol, which allows only the differences between two sets of files to be transferred. For the first transfer, it copies the entire contents of the file or directory from the source to the destination, but from the next time, it copies only the changed parts to the destination.

4. Rsync consumes less bandwidth because it uses compression and decompression methods to send and receive data at both ends at the same time. HTTP compression technology.

Command installation

The environment of this demo article is CentOS 7.0, which can be used directly. For installation, refer toThis article

Command syntax format

Rsync [options] resource source directory destination directory

-V: detailed mode output

-R: copy data recursively, but do not retain timestamp and permission when transmitting data

-A: archive mode. The archive mode is always recursive copy, and the symbolic link, permission, owner and group timestamp are reserved

-Z: compressed transmission

-h : human-readable

– progress: displays the transmission process

– exclude = pattern specifies the file mode that excludes transfers

– include = pattern specifies the file mode to be transferred

– delete when synchronizing, delete those files in DST but not in SRC

– max size: limit the maximum size of the transferred file

– dry run: displays which files will be transferred and will not actually be transferred

– bwlimit: limit transmission bandwidth

-W: Copy files without incremental detection

Use case

Synchronize a file locally


[[email protected] jenkins]# rsync -zvrh remoting.jar /root/work/

sending incremental file list

created directory /root/work

remoting.jar

sent 796.37K bytes  received 68 bytes  1.59M bytes/sec

total size is 872.44K  speedup is 1.10

View synchronization results


[[email protected] jenkins]# ll /root/work/

total 852

-rw-r--r-- 1 root root 872440 Nov  3 20:03 remoting.jar

Synchronize a directory

[[email protected] jenkins]# rsync -zvrh workspace/ /root/work/

sending incremental file list 

Company server test / tests / unit/

Company server test / tests / unit/ExampleTest.php

Company server test @ TMP/

sent 18.82M bytes  received 7.30K bytes  7.53M bytes/sec

total size is 29.87M  speedup is 1.59

View synchronization results

[[email protected] jenkins]# ll /root/work/

total 868

-rw-r--r--  1 root root 872440 Nov  3 20:03 remoting.jar

Drwxr-xr-x 11 root 4096 Nov 3 20:05 company server test

Drwxr-xr-x 2 root 4096 Nov 3 20:04 company server test @ TMP

Synchronize remote directories


[[email protected]]$ rsync -avz rpmpkgs/ [email protected]:/home/

[email protected]'s password:

sending incremental file list

./

httpd-2.2.3-82.el5.centos.i386.rpm

mod_ssl-2.2.3-82.el5.centos.i386.rpm

nagios-3.5.0.tar.gz

nagios-plugins-1.4.16.tar.gz

sent 4993369 bytes  received 91 bytes  399476.80 bytes/sec

total size is 4991313  speedup is 1.00

summary

Using this command, what attracts me most is that incremental synchronization can be realized instead of full synchronization. This reduces the consumption of bandwidth and speeds up the transmission speed. As for the principle of command synchronization, you can refer toThis article

This work adoptsCC agreement, reprint must indicate the author and the link to this article

Love the official account of “two technical circles of cards”.