Redis series (IV) redis profile and persistence

Time:2021-9-18

It is more or less used in NoSQL development, and it is also a necessary knowledge point for interview. I’ve asked every interview in recent days. However, I feel that the answer is not good, and there are still many knowledge points that need to be sorted out. Here, I’ll comb through several redis notes, and then add the above test questions.

Redis series:

  1. Redis series (I) introduction to redis
  2. Redis series (II) 8 data types of redis
  3. Redis series (III) integration of redis transactions and spring boot
  4. Redis series (IV) redis profile and persistence
  5. Redis series (V) publish subscribe mode, master-slave replication and sentinel mode
  6. Redis series (VI) redis’s cache penetration, cache breakdown and cache avalanche
  7. Redis series (VII) redis interview questions
  8. Redis Command Reference

1. Redis.conf details

Found the profile specified at startup:

[[email protected] ~]# cd /usr/local/bin
[[email protected] bin]# ls
dump.rdb         jemalloc.sh  kconfig           luajit        mcrypt    redis-benchmark  redis-check-rdb  redis-sentinel
jemalloc-config  jeprof       libmcrypt-config  luajit-2.0.4  mdecrypt  redis-check-aof  redis-cli        redis-server
[[email protected] bin]# cd kconfig/
[[email protected] kconfig]# vim redis.conf

1. Unit

# Redis configuration file example.
#
# Note that in order to read the configuration file, Redis must be
# started with the file path as first argument:
#
# ./redis-server /path/to/redis.conf

# Note on units: when memory size is needed, it is possible to specify
# it in the usual form of 1k 5GB 4M and so forth:
#
# 1k => 1000 bytes
# 1kb => 1024 bytes
# 1m => 1000000 bytes
# 1mb => 1024*1024 bytes
# 1g => 1000000000 bytes
# 1gb => 1024*1024*1024 bytes
#
# units are case insensitive so 1GB 1Gb 1gB are all the same.

The unit in the configuration file is not case sensitive.

2. Contain

################################## INCLUDES ###################################

# Include one or more other config files here.  This is useful if you
# have a standard template that goes to all Redis servers but also need
# to customize a few per-server settings.  Include files can include
# other files, so use this wisely.
#
# Notice option "include" won't be rewritten by command "CONFIG REWRITE"
# from admin or Redis Sentinel. Since Redis always uses the last processed
# line as value of a configuration directive, you'd better put includes
# at the beginning of this file to avoid overwriting config change at runtime.
#
# If instead you are interested in using includes to override configuration
# options, it is better to use include as the last line.
#
# include /path/to/local.conf
# include /path/to/other.conf

A configuration file can combine multiple configuration files.

3. Network

bind 127.0.0.1 		#  Bound IP
Protected mode no # protected mode
port 6379 			#  port settings

4. General

daemonize yes 		#  Run as a daemon. The default is no. We need to turn it on as yes
pidfile /var/run/redis_ 6379.pid 		#   If it is started in the background, we need to specify a PID file

#Log level
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel notice
logfile "" 		#  Location of log files
databases 16 	#  The number of databases. The default is 16
Always show logo yes # whether the logo is always displayed

5. Snapshot snapshot

Persistence: the number of operations performed within a specified time will be persisted to the file

Redis is an in memory database. If there is no persistence, the data will be lost after power failure.

################################ SNAPSHOTTING  ################################
#
# Save the DB on disk:
#
#   save  
#
#   Will save the DB if both the given number of seconds and the given
#   number of write operations against the DB occurred.
#
#   In the example below the behaviour will be to save:
#   after 900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000 keys changed
#
#   Note: you can disable saving completely by commenting out all "save" lines.
#
#   It is also possible to remove all the previously configured save
#   points by adding a save directive with a single empty string argument
#   like in the following example:
#
#   save ""
#If at least one key is modified within 900s, the persistence operation is performed
save 900 1

#If at least 10 keys are modified within 300s, the persistence operation is performed
save 300 10
save 60 10000

Stop writes on bgsave error yes # if persistence fails, do you want to continue working
Rdbcompression yes # whether to compress RDB files requires some CPU resources
Rdbchecksum yes # check and verify errors when saving RDB files
Directory where dir. / # RDB files are saved

6. Security security

Redis password can be set. There is no password by default.

[[email protected] bin]# redis-cli -p 6379
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> config get requirepass 		#  Get redis password
1) "requirepass"
2) ""
127.0.0.1:6379 > config set requirepass "123456" # set redis password
OK
127.0.0.1:6379> ping
(error) NOAUTH Authentication required. 		#  It is found that all commands have no permission
127.0.0.1:6379> auth 123456 			#  Login with password
OK
127.0.0.1:6379> config get requirepass
1) "requirepass"
2) "123456"
127.0.0.1:6379>

7. Clients restrictions

################################### CLIENTS ####################################

# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
#
# maxclients 10000 		#  Set the maximum number of clients that can link to redis
# maxmemory  		#  Redis sets the maximum memory capacity
 Maxmemory policy noemotion # memory processing policy after reaching the maximum limit
     -Noeviction: when the memory usage reaches the threshold, all commands that cause memory requests will report errors.
    -All keys LRU: use LRU algorithm to delete keys in all keys until enough memory is free.
    -Volatile LRU: in keys with expiration time set, LRU algorithm is used to delete keys until enough memory is free.
    -All keys random: randomly delete all keys until enough memory is free.
    -Volatile random: delete keys randomly among keys with expiration time set until enough memory is free.
    -Volatile TTL: in the key space with expiration time set, keys with earlier expiration time are removed first.

8. Append only mode AOF configuration

Appendonly no # by default, the AOF mode is not enabled. By default, RDB is used for persistence. In most cases, RDB is fully sufficient

appendfilename "appendonly.aof" 		#  The name of the persistent file
#Appendfsync always # every modification consumes sync performance
Appendfsync everysec # executes sync once per second, which may cause 1s of data loss.
#Appendfsync no # does not execute sync. At this time, the operating system synchronizes the data itself, which is the fastest.

2. Persistence (interview and focus)

Detailed explanation of redis RDB persistence

Interview and work, persistence is the focus.

Redis is an in memory database. If the database state in memory is not saved to disk, the database state in the server will disappear once the server process exits. Therefore, redis provides the persistence function.

1、RDB (Redis DataBase)

What is RDB

Within the specified time interval, write the dataset snapshot in memory to disk, that is, snapshot snapshot. When it is restored, it reads the snapshot file directly into memory.

Redis will separately create (fork) a sub process for persistence. It will first write the data to a temporary file. After the persistence process is completed, redis will use this temporary file to replace the last persistent file. During the whole process, the main process does not perform any IO operations, which ensures extremely high performance. If large-scale data recovery is required and is not very sensitive to the integrity of data recovery, RDB is more efficient than AOF. The only disadvantage of RDB is that the last persistent data may be lost.

In the production environment, this file needs to be remembered

The default persistence method is RDB, which generally does not need to be modified.

The file saved by RDB is dump.rdb:

# The filename where to dump the DB
dbfilename dump.rdb

Test 1:

First, modify the policy of saving snapshots in the configuration file

################################ SNAPSHOTTING  ################################
#
# Save the DB on disk:
#
#   save  
#
#   save ""

# save 900 1
# save 300 10
# save 60 10000
Save 60 5 # as long as the key is modified five times within 60s, the RDB operation will be triggered.

Save Profile:

127.0.0.1:6379> save
OK
127.0.0.1:6379>

Delete the original dump.rdb file:

[[email protected] bin]# ls
dump.rdb         jemalloc.sh  kconfig           luajit        mcrypt    redis-benchmark  redis-check-rdb  redis-sentinel
jemalloc-config  jeprof       libmcrypt-config  luajit-2.0.4  mdecrypt  redis-check-aof  redis-cli        redis-server
[[email protected] bin]# rm -rf dump.rdb 
[[email protected] bin]# ls
jemalloc-config  jeprof   libmcrypt-config  luajit-2.0.4  mdecrypt         redis-check-aof  redis-cli       redis-server
jemalloc.sh      kconfig  luajit            mcrypt        redis-benchmark  redis-check-rdb  redis-sentinel
[[email protected] bin]#

Modify the key 5 times within 60s:

127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> set k2 v2
OK
127.0.0.1:6379> set k3 v3
OK
127.0.0.1:6379> set k4 v4
OK
127.0.0.1:6379> set k5 v5
OK

The dump.rdb file appears again.

[[email protected] bin]# ls
dump.rdb         jemalloc.sh  kconfig           luajit        mcrypt    redis-benchmark  redis-check-rdb  redis-sentinel
jemalloc-config  jeprof       libmcrypt-config  luajit-2.0.4  mdecrypt  redis-check-aof  redis-cli        redis-server
[[email protected] bin]#

Recover data:

Close the redis service and client, and the data will be automatically recovered when entering again:

127.0.0.1:6379> shutdown 		#  Close redis service
not connected> exit
[ [email protected]  bin]# ps -ef|grep redis 		#  Redis has been turned off
root     25989 23576  0 14:27 pts/1    00:00:00 grep --color=auto redis
[ [email protected]  bin]# redis-server kconfig/redis.conf  		#  Turn on the service again
25994:C 02 May 2020 14:28:01.003 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
25994:C 02 May 2020 14:28:01.003 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=25994, just started
25994:C 02 May 2020 14:28:01.003 # Configuration loaded
[ [email protected]  bin]# redis-cli -p 6379 		#  Client connection
127.0.0.1:6379> get k2 						#  Data can be obtained directly, indicating that K2 is persisted.
"v2"
127.0.0.1:6379>

Test 2:

Delete dump.rdb file

[email protected] bin]# rm -rf dump.rdb 
[[email protected] bin]# ls
jemalloc-config  jeprof   libmcrypt-config  luajit-2.0.4  mdecrypt         redis-check-aof  redis-cli       redis-server
jemalloc.sh      kconfig  luajit

Clear all data on the client:

127.0.0.1:6379> flushall
OK

Verify the dump.rdb file again:

[[email protected] bin]# ls
dump.rdb         jemalloc.sh  kconfig           luajit        mcrypt    redis-benchmark  redis-check-rdb  redis-sentinel
jemalloc-config  jeprof       libmcrypt-config  luajit-2.0.4  mdecrypt  redis-check-aof  redis-cli        redis-server
[[email protected] bin]#

The dump.rdb file appears again.

Trigger mechanism

1. If the save rule is satisfied, the RDB rule will be triggered automatically

2. Executing the flush command will also trigger RDB rules

3. Exiting redis will also generate RDB files

The backup automatically generates a dump.rdb file.

How to recover RDB files

1. You only need to put the RDB file in the redis startup directory. When redis starts, dump.rdb will be automatically checked to recover the data in it;

2. View the location where the RDB file is stored, and use the following command in the client.

127.0.0.1:6379> config get dir
1) "dir"
2) "/ usr / local / bin" # if the dump.rdb file exists in this directory, the data in it will be recovered automatically after startup
127.0.0.1:6379>

Advantages and disadvantages of RDB

advantage:

1. Suitable for large-scale data recovery

2. The requirements for data integrity are not high

Disadvantages:

1. The operation needs a certain time interval. If redis goes down unexpectedly, the last modified data will be lost

2. The fork process takes up a certain amount of space.

2、AOF (Append Only File)

Aof (append only file) persistence records each write command in the form of an independent log, and re executes the commands in the AOF file when redis is restarted to recover data. The main function of AOF is to solve the real-time problem of data persistence.

Detailed explanation of redis AOF persistence

Each operation is recorded in the form of a log. All instructions of the process executed by redis are recorded (read operations are not recorded). Only files are appended, but files cannot be overwritten. At the beginning of redis startup, the file will be read to rebuild the data. In other words, if redis restarts, the write instructions will be executed from front to back according to the contents of the log file to complete the data recovery.

Aof saves the appendonly.aof file.

Turn on AOF mode

Change appendonly, which is no by default in the configuration file, to yes and restart the service.

appendonly yes
# The name of the append only file (default: "appendonly.aof")
appendfilename "appendonly.aof"

After restart, you can see the AOF file:

[[email protected] bin]# ls
appendonly.aof  ipsort           jeprof            logs          mcrypt    redis-benchmark  redis-cli       tntrecht
backup.db       jemalloc-config  kconfig           luajit        mdecrypt  redis-check-aof  redis-sentinel
dump.rdb        jemalloc.sh      libmcrypt-config  luajit-2.0.4  pnscan    redis-check-rdb  redis-server

But the file is empty. Use the client to add some data and view it again:

[[email protected] bin]# redis-cli -p 6379
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> set k2 v2
OK
127.0.0.1:6379> set k3 v3
OK
127.0.0.1:6379>
[[email protected] bin]# cat appendonly.aof 
*2
$6
SELECT
$1
0
*3
$3
set
$2
k1
$2
v1
*3
$3
set
$2
k2
$2
v2
*3
$3
set
$2
k3
$2
v3

Automatically repair AOF files

If the AOF file is modified manually, the redis service may not start. For example, here I manually add some commands on the last line of the AOF file:

set
$2
k3
$2
v3
gjjjjjjjjj

Delete dump.rdb file and restart the service:

[[email protected] bin]# rm -rf dump.rdb 
[[email protected] bin]# ls
appendonly.aof   jemalloc.sh  libmcrypt-config  luajit-2.0.4  pnscan           redis-check-rdb  redis-server
ipsort           jeprof       logs              mcrypt        redis-benchmark  redis-cli        tntrecht
jemalloc-config  kconfig      luajit            mdecrypt      redis-check-aof  redis-sentinel
[[email protected] bin]# redis-server kconfig/redis.conf 
13746:C 02 May 2020 16:22:43.345 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
13746:C 02 May 2020 16:22:43.346 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=13746, just started
13746:C 02 May 2020 16:22:43.346 # Configuration loaded
[[email protected] bin]# redis-cli -p 6379
Could not connect to Redis at 127.0.0.1:6379: Connection refused 		#  connection failed
not connected>

If the AOF file is misplaced, the client cannot link it, and the AOF file needs to be repaired. Redis provides toolsredis-check-aof --fix

[[email protected] bin]# redis-check-aof --fix appendonly.aof 
0x              6e: Expected prefix '*', got: 'g'
AOF analyzed: size=122, ok_up_to=110, diff=12
This will shrink the AOF from 122 bytes, with 12 bytes, to 110 bytes
Continue? [y/N]: y
Successfully truncated AOF
[[email protected] bin]#

Restart the service and try the link again successfully.

Advantages and disadvantages of AOF

appendonly yes 		#  The default is No
Appendfilename "appendonly. AOF" # the name of the persistent file
#Appendfsync always # every modification will sync, which will consume performance
Appendfsync everysec # executes sync once every second, and may lose this 1s of data
#Appendfsync no # does not execute sync. At this time, the operating system synchronizes the data itself, which is the fastest

advantage:

1. Each modification is synchronized, and the integrity of the file is better

2. Sync once per second, and you may lose one second of data

3. Never synchronized, most efficient

Disadvantages:

1. Compared with data files, AOF is much larger than RDB, and the repair speed is also slower than RDB

2. Aof also runs slower than RDB, so the default configuration of redis is RDB persistence.

3. Expand

1. RDB persistence can snapshot and store your data within a specified time interval

2. Aof persistence records every write operation to the server. When the server restarts, these commands will be re executed to recover the original data. The AOF command additionally saves each write operation to the end of the file with redis protocol. Redis can also rewrite the background memory of the AOF file, so that the volume of the AOF file will not be too large

3. Only cache. If you only want your data to exist when the server is running, you can also not use any persistence

4. Enable two persistence methods at the same time

  • In this case, when redis restarts, AOF files will be preferentially loaded to recover the original data, because normally, the data set saved by AOF files is more complete than that saved by RDB files.
  • RDB data is not real-time. When the two are used synchronously, the server will only find AOF files when restarting. Do you want to use AOF only? The author suggests no, because RDB is more suitable for backing up databases (AOF is changing and hard to back up), fast restart, and there will be no potential bugs in AOF, which will be kept as a means in case.

5. Performance recommendations

  • Because RDB files are only used for backup purposes, it is recommended to only persist RDB files on slave, and only backup once every 15 minutes is enough. Only save 900 1 is retained.
  • If you enable AOF, the advantage is that in the worst case, only less than two seconds of data will be lost. The startup script is simpler. It can only load your own AOF file. The cost is that first, it brings continuous IO, but at the end of AOF rewriting, the new data generated in the rewriting process will be written to the new file, resulting in almost inevitable blocking. As long as the hard disk is allowed, the frequency of AOF rewriting should be minimized. The default value of the basic size of AOF rewriting is 64M, which is too small and can be set to more than 5g. The default value exceeds 100% of the original size, and the size rewriting can be changed to an appropriate value.
  • If AOF is not enabled, high availability can be achieved only by master slave replay, which can save a lot of IO and reduce the system fluctuation caused by rewriting. The price is that if the master / slave goes down at the same time, more than ten minutes of data will be lost. The startup script also needs to compare the RDB files in the two master / slave and load the newer one. Microblog is this architecture.

The next note will introduce redis’s publish and subscribe mode, master-slave replication and sentinel mode.

Recommended Today

Seven Python code review tools recommended

althoughPythonLanguage is one of the most flexible development languages at present, but developers often abuse its flexibility and even violate relevant standards. So PythoncodeThe following common quality problems often occur: Some unused modules have been imported Function is missing arguments in various calls The appropriate format indentation is missing Missing appropriate spaces before and after […]