The so-called persistence can be simply understood as the process of saving the data in memory to the hard disk for storage. The persistent data can still be accessed after the system is restarted or down, ensuring the security of the data.
RedisThere are two persistence schemes. One is snapshot（
RDB； One is append only mode（
APPEND ONLY MODE）, called AOF. Next, let’s take a look at their use and precautions.
Redis DataBaseThe abbreviation of is
RedisDefault persistence scheme. It can snapshot memory data sets within a specified time interval（
snapshot）Write to disk and restore the snapshot file（
dump.rdb）Read back memory.
Let’s start by picking up the information in the configuration file
In a givenSecondsWithin the database, if theWrite operandWhen the set value is reached, the data is synchronized to the data file. Support multiple conditional cooperation,
RedisThree conditions are provided in the default configuration file:
save 900 1 // 1 change in 900s save 300 10 // 10 changes in 300s save 60 10000 // 10000 changes in 60s
be careful: if you don’t want to use
RDBPlan, you can put
save "" Open the comments on the, and the top three comments out.
bgsaveWhen an error occurs,
RedisWhether to stop executing the write command;
- If yes
yes, when there is a problem with the hard disk,
RedisStop accepting the write operation, so that we can find it in time and avoid a large amount of data loss;
- If yes
bgsaveContinue to execute the write command.
If you have set
RedisThe correct monitoring and persistence of the server, that is, other means are used to discover and control data integrity. You may want to disable this function, so that even if there are problems in disk, permissions, etc,
be careful: if the background save process starts working again,
RedisRe writing will be automatically allowed.
Specifies whether to save to the local databasecompress（
LZFCompressed) data, the default is
yes。 If in order to save
CPUTime, you can turn off this option, but it will cause the database file to become huge.
5Initially, after storing the snapshot, you can also use
CRC64Algorithm for data verification,
CRC64The verification is placed at the end of the file. After opening, save and load
RDBFiles will increase by about
10%If you want to get the maximum performance improvement, you can turn off this function.
RDBThe checksum of the file is zero, which tells the loading code to skip the check.
Specify the file name of the local database and load it automatically after restartMemory, manual execution
saveThe order will take effect immediately.
Big pit, please pay attention：
shutdownAll commands are cleared and submitted to
Specify the local database storage directory.
RedisNeed to save
dump.rdbWhen the file is, it calls system functions
fork(), create a sub process (completely consistent with the main process);
- The subprocess writes the dataset to a temporary file
- When the child process completes the new
RDBWhen writing files,
RDBFile replaces the original
RDBFile and delete the old one
This way of working makes
RedisCan be copied from write time（
copy-on-write）Benefit from the mechanism.
How to trigger RDB snapshot
- The default snapshot configuration in the configuration file;
save(blocking, just save the snapshot and wait for others) or
bgsave(asynchronous) commands, and snapshots can also respond to client commands;
flushallCommand to clear all data in the database, which is of little significance;
shutdownCommand to ensure that the server shuts down normally without losing any data, which is of little significance.
Recover data from RDB files
In actual development, the damage of physical hard disk is generally considered, so we will choose backup
dump.rdbDocuments. To be backed up
dump.rdbCopy files to
redisOf the installation directory
redisService is enough.
RDBIs a very compact file, which is very suitable for data set backup;
RDBIt is a compact single file, which is easy to transfer to another remote data center or Amazon’s S3 (possibly encrypted), which is very suitable for disaster recovery;
RedisThe main process of is not in progress
I/OOperation ensures high performance;
- It is suitable for large-scale data recovery. If the requirements for data integrity and consistency are not high,
RedisIn case of unexpected downtime, you may lose several minutes of data;
forkSub process to save the data set to the hard disk. When the data set is relatively large,
forkThe process is very time-consuming and may lead to
RedisUnable to respond to client requests within some milliseconds. If the data set is huge and
CPUWhen the performance is not very good, this situation will last for 1 second;
forkHowever, the frequency of rewriting log files can be adjusted to improve the durability of data sets.
In order to solve
RDBFrom the problem of losing too much data during downtime
durablePersistence mode of:
Append Only FileThe abbreviation of is not enabled by default.
AOFEach write operation is recorded in the form of log. Only files are allowed to be appended, but files cannot be overwritten. When the server restarts, these commands will be re executed to recover the original data.
Let’s take another look at the in the configuration file
APPEND ONLY MODE:
It is off by default and changed to
yesTurn on persistence.
RDBCan be enabled simultaneously without problems.
The default name of the file, which is created at startup. loadBefore
Synchronization policy: system functions
fsync()Tell the operating system to actually write data on the disk.
RedisThree different modes are supported
appendfsync always // Every time a data change occurs, it will be immediately recorded to the disk. The performance is poor, but the data integrity is good appendfsync everysec // The default recommendation is asynchronous operation, which records every second. If it goes down, data will be lost within 1 second appendfsync no // Out of sync, refresh data only when the operating system needs it
To understand the following configuration contents, first talk about the principle of “log rewriting”:
AOFThe command is appended to the end of the file, so as the number of write commands increases,
AOFThe size of the file will become larger and larger. In order to avoid this situation, a rewriting mechanism is added: you can rewrite the service without interrupting the service client
Override trigger:Through execution
bgrewriteaofCommand to generate a new
AOFFile that contains the data needed to rebuild the current datasetleastCommand.
Redis 2.2To execute this command manually,
Redis 2.4It can be triggered automatically by modifying the configuration file (configuration is covered below).
RedisExecute system functions
fork(), create a sub process (completely consistent with the main process);
- The subprocess starts a new
AOFWrite the contents of the file to the temporary file;
- For all newly executed write commands, the parent process accumulates them into a memory cache and appends these changes to the existing cache
AOFThe end of the file, so that even if there is a shutdown in the middle of rewriting, the existing
AOFDocuments are also safe;
- When the child process finishes rewriting, it sends a signal to the parent process. After receiving the signal, the parent process appends all the data in the memory cache to the new process
AOFEnd of file.
RedisReplace the old file with the new file atomically, and then all commands will be appended directly to the new file
AOFEnd of file.
When we execute the main process at the same timeWrite operationAnd subprocessesrewriteDuring operation, both will operate the disk, and rewriting often involves a large number of disk operations, which will cause the main process to write
aofBlocking occurs when the file is.
To solve this problem,
no-appendfsync-on-rewriteThe parameters are out.
- If this parameter is set to
no, is the safest way. It won’t lose data, but it has to endure the problem of blocking;
- If set to
yes, this is equivalent to
no, which means that no disk operation is performed, but the buffer is written. Therefore, this will not cause blocking (because there is no competing disk), but if at this time
redisHang up and lose data. How much data is lost? stay
linuxUnder the default setting of the operating system, data will be lost for up to 30s.
Therefore, if the application system cannot tolerate delay and can tolerate a small amount of data loss, it is set to
yes； If the application system cannot tolerate data loss, set to
Override percentage. The default is after the last rewrite
aofDouble the file size.
Minimum value of rewrite trigger: 64MB.
auto-aof-rewrite-percentageParameters determine the automatic trigger timing.
RedisThe last time it was rewritten is recorded
AOFThe default configuration is when
AOFFile size is last
rewriteDouble the size after and the file is larger than
Large Internet companies are generally
AOFWhen the file is truncated, that is
AOFThe last command of the file is incomplete. If you start at this time
AOFWhen the data is loaded back into memory, a problem occurs.
- Yes: load a truncated
RedisThe server starts sending logs to notify the user of the event;
- No: the server will abort with an error and refuse to start.
When we know
AOFWhen an error is reported in a file, the following methods can be used to repair the error
AOFCreate a backup of the file;
redis-check-aofCommand, to the original
AOFRepair the file;
redis-check-aof --fix appendonly.aofRepair the command and kill all non-conforming syntax
diff -uCompare the repaired
AOFDocuments and original
AOFBackup of files and check the differences between the two files;
RedisServer, wait for the server to load the repaired
AOFFile and data recovery.
AOFWhen you file a file,
RedisCan be in
AOFUsed in documents
RDBPreamble to speed up rewriting and recovery. When on, overrides the
AOFThe document consists of two different sections:
AOFFile toRedisStart with a string and load prefixed
RDBFile, and then continue loading
- Higher data integrity and consistency,
AOFBy using different strategies, the persistence of can lose up to 1 second of data;
AOFFile is a log file that is only appended and does not need to be written
RedisCan be in
AOFWhen the file volume becomes too large, it is automatically adjusted in the background
AOFRewriting is absolutely safe;
AOFWrite operation of file record
RedisThe format of the protocol is saved, which is easy to read and analyze the documents;
- For the same dataset,
AOFThe volume of the file is usually larger than
RDBVolume of documents;
- Based on the
AOFMay be slower than
In general, every second
fsyncThe performance is still very high while off
RDBAs fast, even under high load. But when dealing with huge write loads,
RDBIt can provide more guaranteed maximum delay time（
Comparison and summary
How to choose which persistence method to use?
Generally speaking, if you want to be comparable
PostgreSQLFor data security, two persistence functions should be used at the same time.
If you are very concerned about data, but can still withstand data loss within a few minutes, you can only use
Because AOF persistence has better real-time performance, that is, less data is lost when the process exits unexpectedly
AOFAt presentmain streamPersistent mode.
Many users only use
AOFPersistence, but we don’t recommend this method: because it is generated regularly
snapshot）Very convenient for database backup, and
RDBRestoring datasets is also faster than
Interaction between AOF and RDB
When the version number is greater than or equal to
BGSAVECannot execute during execution
BGREWRITEAOF。 Conversely, in
BGREWRITEAOFIn the process of execution, it cannot be executed
BGSAVE。 This prevents two
RedisThe background process makes a lot of changes to the disk at the same time
BGSAVEIs executing, and the user explicitly calls
BGREWRITEAOFCommand, the server will reply to the user with a
OKStatus and inform the user
BGREWRITEAOFHas been scheduled for execution: once
BGREWRITEAOF It will officially begin.
RedisWhen starting, if
AOFIf the persistence is turned on, the program will take priority
AOFFile to recover the dataset because
AOF The data saved in the file is usually the most complete.
Backup redis data
- Create a recurring task（
cron job）, one per hour
RDBFiles are backed up to a folder, and one file is saved every day
RDBBackup files to another folder;
- Make sure that the backup of the snapshot has the corresponding date and time information. Each time the periodic task script is executed, use the
findCommand to delete expired snapshots;
- At least once a day, will
RDBBack up outside your data center, or at least back up until you run
RedisOutside the physical machine of the server.
In practical application, because
RDBThe document is only used for backup purposes. It is recommended to use it only in
RDBFiles, and only need 15 minutes to back up once is enough, only retain
save 900 1This rule.
AOFThe advantage is that in the worst case, only less than 2 seconds of data will be lost, and the startup script is relatively simple
AOFJust file it. The first cost is to bring sustained
AOF rewrite In the end
rewriteThe blocking caused by new data written to new files is almost inevitable.
As long as the hard disk license, it should be minimized
AOFOverridden base size default
64MIt’s too small. You can set it to
5Gabove. When the default exceeds 100% of the original size, the rewriting can be changed to an appropriate value.
Master-Slave ReplicationHigh availability is also possible. Can save a lot of money
IO, also reduced
rewriteSystem fluctuation caused by. The price is if
Master/Slave If it is dropped at the same time, more than ten minutes of data will be lost. The startup script should also compare the two
RDBFile, load the newer one.
That’s all for today, if you have different opinions or better
idea, welcome to contact ah q, add ah q, you can join the technical exchange group to participate in the discussion!