Found a data backup and recovery artifact! Open source, free, fast, efficient and secure

brief introduction

Restic is a fast, efficient, and secure backup program. It supports three major operating systems (Linux, MacOS, windows) and some smaller operating systems (FreeBSD, OpenBSD). It is written in go programming language, uses AES-256 to encrypt data, and uses poly1305-aes to authenticate data.

GitHub address:https://github.com/restic/restic

Design principles

Restic is a program that can backup correctly. Its design follows the following principles:

• simple: backup should be a smooth process, otherwise you may want to skip it. Restic should be easy to configure and use so that you can recover it directly in case of data loss. Again, restoring data should not be complicated.
• fast: using restic to back up data should only be limited by network or hard disk bandwidth, so that you can back up files every day. If it takes too much time, no one will back up. Restoring the backup should only transfer the data required by the files to be restored, so the process is very fast.
• Verifiable: recovery is more important than backup, so restic makes it easy to verify that all data can be recovered.
• security: restic uses encryption technology to ensure the confidentiality and integrity of your data. It is assumed that the storage location of the backup data is not a trusted environment (for example, someone such as a system administrator can access your backup space). Restic is designed to protect your data from such attackers.
• Efficient: as the data grows, additional snapshots should only occupy the actual incremental storage. More importantly, before the duplicate data is actually written to the storage backend, it should be de duplicated to save valuable backup space.

install

CentOS

[[email protected] ~]# yum install yum-plugin-copr -y
[[email protected] ~]# yum copr enable copart/restic -y
Loaded plugins: copr, fastestmirror
copr done
[[email protected] ~]# yum install restic -y

If there is an error in the above installation, please execute the following command to solve the source problem

[[email protected] ~]# yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
Loaded plugins: fastestmirror
adding repo from: https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
grabbing file https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo to /etc/yum.repos.d/copart-restic-epel-7.repo
repo saved to /etc/yum.repos.d/copart-restic-epel-7.repo

macOS

# brew
$ brew install restic

# macprots
$ sudo port install restic

For more installation methods, please refer to:https://restic.readthedocs.io…

Configuring the backup repository

The location where the backup is saved is called the repository. The repository can be stored locally or on a remote server or service. The following storage methods are supported:

For automatic backups, restic accepts the repository location restic in the environment variable_ Repository. Restic can also read the repository location restic from the file specified through the –repository-file option or the environment variable_ Repository_ File.

For passwords, there are several options:

• Set the environment variable restic_ Password
• Specify the file path with password restic through the option –password file or environment variable_ Password_ File
• Configure the program restic called when a password is required through the option –password command or environment variable_ Password_ Command

Create local repository

Take creating a local repository as an example

[[email protected] ~]# restic init --repo /restic/backup_dir
enter password for new repository: 
enter password again: 
created restic repository dff64d39c6 at /restic/backup_dir

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
#The prompt is very clear. Let you remember the password entered here. Losing the password means losing the data

Please refer to the official documentation for other repository creation methods:
https://restic.readthedocs.io…

Backup practice

Back up the contents of the directory data to the repository

[[email protected] ~]# restic -r /restic/backup_dir --verbose backup ./data
open repository
enter password for repository: 
repository dff64d39 opened successfully, password is correct
created new cache in /root/.cache/restic
lock repository
load index files
no parent snapshot found, will read all files
start scan on [./data]
start backup on [./data]
scan finished in 1.455s: 2922 files, 71.126 MiB

Files:        2922 new,     0 changed,     0 unmodified
Dirs:           99 new,     0 changed,     0 unmodified
Data Blobs:   2889 new
Tree Blobs:     99 new
Added to the repo: 72.083 MiB

Processed 2922 files, 71.126 MIB in 0:05 \
Snapshot 4d20711e saved \

--Verbose \\output process information

You can also back up a single file

[[email protected] ~]# ls ./data
goInception-linux-amd64-v1.2.3.tar.gz  httpd-2.4.6-95.el7.centos.x86_64.rpm  mingongge.z02
httpd-2.4.46                           mingongge.file                        mingongge.zip
httpd-2.4.46.tar.gz                    mingongge.z01
[[email protected] ~]# restic -r /restic/backup_dir --verbose backup ./data/mingongge.zip
open repository
enter password for repository: 
repository dff64d39 opened successfully, password is correct
lock repository
load index files
no parent snapshot found, will read all files
start scan on [./data/mingongge.zip]
start backup on [./data/mingongge.zip]
scan finished in 0.249s: 1 files, 942.793 KiB

Files:           1 new,     0 changed,     0 unmodified
Dirs:            1 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      2 new
Added to the repo: 750 B

processed 1 files, 942.793 KiB in 0:00
snapshot 3e5b7dea saved

If you execute the backup command in step 1 again, you will find that it does not add any content, but only adds another snapshot for the current data. In fact, restic has the function of scanning files (scanning and comparing files one by one), so it only backs up and stores the same data once.

File detection function

Scanning the entire contents of each file is a waste of resources. Therefore, restic also uses change detection rules based on file metadata to determine whether the file has not changed since the last backup. If so, the file will not be scanned again.

On UNIX (including Linux and MAC), since the file is located in the same location as the file in the previous backup, the following file metadata attributes must match to assume that its contents have not changed:

• Modify timestamp (mtime)
• Metadata change timestamp (CTime)
• file size
• Inode number (internal number used to reference files in the file system)

Therefore, for the above reasons, some parameters are introduced as follows:

--Force \n turn off change detection and rescan all files
--Ignore CTime \mtime matching is required, but CTime is allowed to be different
--Ignore inode \mtime matching is required, but inode number and CTime are allowed to be different

Exclude file parameters

--Exclude \\n specifies to exclude one or more items one or more times
--Iexclude \e is the same as exclude, but the path is ignored
--Exclude caches \
--Exclude file \
--Iexclude file \e is the same as exclude file, but the path is ignored
--Exclude if present foo \
--Exclude larger than size \

For more related functions, please refer to:https://restic.readthedocs.io…

Repository usage

Since the data is backed up to the repository, we also need to use the repository. The following describes the related operations.

List all snapshots of the repository

This function is the same as using LS command on the system at ordinary times to view the displayed functions

[[email protected] ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
4d20711e  2021-06-04 03:40:47  centos7                 /root/data
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
3 snapshots

#You can also use the following parameters to filter the matching view
--path="dir_name"
--host hostname
#Group outputs through the same filters (host, path, label)
--group-by

For more information, please refer to:https://restic.readthedocs.io…

Detect repository data

[[email protected] ~]# restic -r /restic/backup_dir/ check
using temporary cache in /tmp/restic-check-cache-294136679
enter password for repository: 
repository dff64d39 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-294136679
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  3 / 3 snapshots
no errors were found

data recovery

This is the key point. Recovering data is the king’s bombing.

Create an environment that simulates data deletion

[[email protected] ~]# cd data/
[[email protected] data]# ll
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip
[[email protected] data]# rm -rf ./*
[[email protected] data]# ll
total 0

Recover data

Recover erroneously deleted data directly from a snapshot

[[email protected] ~]# restic -r /restic/backup_dir/ restore 4d20711e --target /root/
enter password for repository: 
repository dff64d39 opened successfully, password is correct
restoring <Snapshot 4d20711e of [/root/data] at 2021-06-04 03:40:47.878873654 -0400 EDT by [email protected]> to /root/
[[email protected] ~]# ll /root/data/
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip

The above recovery method specifies to recover data from a snapshot. Sometimes there are many snapshots. If you do not know how to quickly recover from a snapshot, you can use the latest parameter to recover data from the snapshot of the last backup.

[[email protected] ~]# rm -rf /root/data/*
[[email protected] ~]# ll /root/data/
total 0
[[email protected] ~]# restic -r /restic/backup_dir/ restore latest --target /root/
enter password for repository: 
repository dff64d39 opened successfully, password is correct
restoring <Snapshot 94c62288 of [/root/data] at 2021-06-04 03:51:21.232686491 -0400 EDT by [email protected]> to /root/
[[email protected] ~]# ll /root/data/
total 33796
-rw-r--r--  1 root root 13034487 Aug 30  2020 goInception-linux-amd64-v1.2.3.tar.gz
drwxr-sr-x 11 root   40     4096 Dec 24 22:35 httpd-2.4.46
-rw-r--r--  1 root root  9363314 Aug  5  2020 httpd-2.4.46.tar.gz
-rw-r--r--  1 root root  2846172 Oct 14  2020 httpd-2.4.6-95.el7.centos.x86_64.rpm
-rw-r--r--  1 root root        0 Jan 16 11:32 mingongge.file
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z01
-rw-r--r--  1 root root  4194304 Jan 16 16:24 mingongge.z02
-rw-r--r--  1 root root   965420 Jan 16 16:24 mingongge.zip

Introduction to other management

Delete snapshot

Here, the names of snapshots and backup files are the same. If there are too many snapshots, they may need to be deleted regularly. Generally, the backup time for data is basically 30 days, except that the particularly important data may be longer. This depends on the actual situation.

[[email protected] ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
4d20711e  2021-06-04 03:40:47  centos7                 /root/data
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
3 snapshots
[[email protected] ~]# restic -r /restic/backup_dir/ forget 4d20711e
enter password for repository: 
repository dff64d39 opened successfully, password is correct
[0:00] 100.00%  1 / 1 files deleted
[[email protected] ~]# restic -r /restic/backup_dir/ snapshots
enter password for repository: 
repository dff64d39 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
3e5b7dea  2021-06-04 03:46:34  centos7                 /root/data/mingongge.zip
94c62288  2021-06-04 03:51:21  centos7                 /root/data
-------------------------------------------------------------------------------
2 snapshots

#This method is the same as recovery. Both specify the snapshot ID

Deleting in this way only deletes the snapshot, but the referenced file is still in the repository, that is, the size of the repository has not changed. You need to use the prune parameter to clear the data.

[[email protected] ~]# restic -r /restic/backup_dir/ prune
enter password for repository: 
repository dff64d39 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 2 snapshots
[0:00] 100.00%  2 / 2 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  19 / 19 packs processed

to repack:            0 blobs / 0 B
this removes          0 blobs / 0 B
to delete:            0 blobs / 0 B
total prune:          0 blobs / 0 B
remaining:         2990 blobs / 72.175 MiB
unused size after prune: 0 B (0.00% of remaining size)

done

You can also use the following command to perform the above two steps at the same time, which is equivalent to a merge operation

restic forget --keep-last 1 --prune
#--Keep last 1 do not delete the last snapshot

For more information on snapshot deletion, refer to:https://restic.readthedocs.io…

Password management for repositories

We can use the key command to set multiple passwords for the same repository. We can also use add, list, remove, passwd to manage passwords.

• View password

[[email protected] ~]# restic -r /restic/backup_dir/ key list
enter password for repository: 
repository dff64d39 opened successfully, password is correct
 ID        User  Host     Created
---------------------------------------------
*d216779f  root  centos7  2021-06-04 03:28:34
---------------------------------------------

• New password

[[email protected] ~]# restic -r /restic/backup_dir/ key add
enter password for repository: 
repository dff64d39 opened successfully, password is correct
enter new password: 
enter password again: 
saved new key as <Key of [email protected], created on 2021-06-04 04:43:18.024358447 -0400 EDT m=+18.001857421>
[[email protected] ~]# restic -r /restic/backup_dir/ key list
enter password for repository: 
repository dff64d39 opened successfully, password is correct
 ID        User  Host     Created
---------------------------------------------
*d216779f  root  centos7  2021-06-04 03:28:34
 33d0b428  root  centos7  2021-06-04 04:43:18
---------------------------------------------

• delete

  [[email protected] ~]# restic -r /restic/backup_dir/ key remove 33d0b428
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
  removed key 33d0b428cba5c62585f1190432e61d46b88b4a6418c693d09ec47db596eace1f
  [[email protected] ~]# restic -r /restic/backup_dir/ key list
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
   ID        User  Host     Created
  ---------------------------------------------
  *d216779f  root  centos7  2021-06-04 03:28:34
  ---------------------------------------------

• Change password

  [[email protected] ~]# restic -r /restic/backup_dir/ key passwd
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
  enter new password: 
  enter password again: 
  saved new key as <Key of [email protected], created on 2021-06-04 04:51:13.658184739 -0400 EDT m=+27.022974479>
  [[email protected] ~]# restic -r /restic/backup_dir/ key list
  enter password for repository: 
  repository dff64d39 opened successfully, password is correct
   ID        User  Host     Created
  ---------------------------------------------
  *a62168f6  root  centos7  2021-06-04 04:51:13
  ---------------------------------------------
  #The password has been updated, which can be seen from the ID change

Frequently asked questions

https://restic.readthedocs.io…

For more relevant functions and operations, interested readers can install and experience them by themselves. After a round of experience, I feel that this restic is indeed a very powerful, fast, safe and efficient backup artifact. The key is that it is open source and free. It is so fragrant that it is strongly recommended.