Interviewer: you said you are proficient in redis. Have you seen the persistent configuration?

Time:2022-4-19

We have already introducedRedisCommands for five data typesAndBasic configuration of configuration fileToday, let’s uncover it from two aspects: Theory and configurationRedisA lasting mystery veil.

The so-called persistence can be simply understood as the process of saving the data in memory to the hard disk for storage. The persistent data can still be accessed after the system is restarted or down, ensuring the security of the data.

RedisThere are two persistence schemes. One is snapshot(SNAPSHOTTING), abbreviationRDB; One is append only mode(APPEND ONLY MODE), called AOF. Next, let’s take a look at their use and precautions.

RDB

RDBbyRedis DataBaseThe abbreviation of isRedisDefault persistence scheme. It can snapshot memory data sets within a specified time interval(snapshot)Write to disk and restore the snapshot file(dump.rdb)Read back memory.

Let’s start by picking up the information in the configuration fileSNAPSHOTTING:

configuration file

save

In a givenSecondsWithin the database, if theWrite operandWhen the set value is reached, the data is synchronized to the data file. Support multiple conditional cooperation,RedisThree conditions are provided in the default configuration file:

save 900 1 	// 1 change in 900s
save 300 10 	// 10 changes in 300s
save 60 10000 	// 10000 changes in 60s

be careful: if you don’t want to useRDBPlan, you can putsave "" Open the comments on the, and the top three comments out.

stop-writes-on-bgsave-error yes

WhenbgsaveWhen an error occurs,RedisWhether to stop executing the write command;

  • If yesyes, when there is a problem with the hard disk,RedisStop accepting the write operation, so that we can find it in time and avoid a large amount of data loss;
  • If yesno, thenRedisignorebgsaveContinue to execute the write command.

If you have setRedisThe correct monitoring and persistence of the server, that is, other means are used to discover and control data integrity. You may want to disable this function, so that even if there are problems in disk, permissions, etc,RedisStill working.

be careful: if the background save process starts working again,RedisRe writing will be automatically allowed.

rdbcompression yes

Specifies whether to save to the local databasecompressRedisuseLZFCompressed) data, the default isyes。 If in order to saveCPUTime, you can turn off this option, but it will cause the database file to become huge.

rdbchecksum yes

fromRDBedition5Initially, after storing the snapshot, you can also useCRC64Algorithm for data verification,CRC64The verification is placed at the end of the file. After opening, save and loadRDBFiles will increase by about10%If you want to get the maximum performance improvement, you can turn off this function.

DisableChecksumCreatedRDBThe checksum of the file is zero, which tells the loading code to skip the check.

dbfilename dump.rdb

Specify the file name of the local database and load it automatically after restartMemory, manual executionsaveThe order will take effect immediately.

Big pit, please pay attentionflushallshutdownAll commands are cleared and submitted todump.rdb

dir ./

Specify the local database storage directory.

theory

operation mode

  • WhenRedisNeed to savedump.rdbWhen the file is, it calls system functionsfork(), create a sub process (completely consistent with the main process);
  • The subprocess writes the dataset to a temporary fileRDBMedium;
  • When the child process completes the newRDBWhen writing files,RedisNew useRDBFile replaces the originalRDBFile and delete the old oneRDBDocuments.

This way of working makesRedisCan be copied from write time(copy-on-write)Benefit from the mechanism.

How to trigger RDB snapshot

  1. The default snapshot configuration in the configuration file;
  2. commandsave(blocking, just save the snapshot and wait for others) orbgsave(asynchronous) commands, and snapshots can also respond to client commands;
  3. implementflushallCommand to clear all data in the database, which is of little significance;
  4. implementshutdownCommand to ensure that the server shuts down normally without losing any data, which is of little significance.

Recover data from RDB files

In actual development, the damage of physical hard disk is generally considered, so we will choose backupdump.rdbDocuments. To be backed updump.rdbCopy files toredisOf the installation directorybinDirectory, restartredisService is enough.

advantage

  • RDBIs a very compact file, which is very suitable for data set backup;
  • RDBIt is a compact single file, which is easy to transfer to another remote data center or Amazon’s S3 (possibly encrypted), which is very suitable for disaster recovery;
  • RedisThe main process of is not in progressI/OOperation ensures high performance;
  • It is suitable for large-scale data recovery. If the requirements for data integrity and consistency are not high,RDBthanAOFMore efficient.

shortcoming

  • stayRedisIn case of unexpected downtime, you may lose several minutes of data;
  • RDBNeed oftenforkSub process to save the data set to the hard disk. When the data set is relatively large,forkThe process is very time-consuming and may lead toRedisUnable to respond to client requests within some milliseconds. If the data set is huge andCPUWhen the performance is not very good, this situation will last for 1 second;AOFAlso needforkHowever, the frequency of rewriting log files can be adjusted to improve the durability of data sets.

AOF

In order to solveRDBFrom the problem of losing too much data during downtime1.1Version start,RedisAdded adurablePersistence mode of:AOF

AOFyesAppend Only FileThe abbreviation of is not enabled by default.AOFEach write operation is recorded in the form of log. Only files are allowed to be appended, but files cannot be overwritten. When the server restarts, these commands will be re executed to recover the original data.

Let’s take another look at the in the configuration fileAPPEND ONLY MODE:

configuration file

appendonly no

It is off by default and changed toyesTurn on persistence.AOFandRDBCan be enabled simultaneously without problems.

appendfilename “appendonly.aof”

The default name of the file, which is created at startup. loadBeforedump.rdbfile

appendfsync

Synchronization policy: system functionsfsync()Tell the operating system to actually write data on the disk.RedisThree different modes are supported

appendfsync always 	// Every time a data change occurs, it will be immediately recorded to the disk. The performance is poor, but the data integrity is good
appendfsync everysec 	// The default recommendation is asynchronous operation, which records every second. If it goes down, data will be lost within 1 second
appendfsync no 	// Out of sync, refresh data only when the operating system needs it

To understand the following configuration contents, first talk about the principle of “log rewriting”:

rewrite

becauseAOFThe command is appended to the end of the file, so as the number of write commands increases,AOFThe size of the file will become larger and larger. In order to avoid this situation, a rewriting mechanism is added: you can rewrite the service without interrupting the service clientAOFRebuild files(rebuild)。

Override trigger:Through executionbgrewriteaofCommand to generate a newAOFFile that contains the data needed to rebuild the current datasetleastCommand.Redis 2.2To execute this command manually,Redis 2.4It can be triggered automatically by modifying the configuration file (configuration is covered below).

Rewriting principle:

  • RedisExecute system functionsfork(), create a sub process (completely consistent with the main process);
  • The subprocess starts a newAOFWrite the contents of the file to the temporary file;
  • For all newly executed write commands, the parent process accumulates them into a memory cache and appends these changes to the existing cacheAOFThe end of the file, so that even if there is a shutdown in the middle of rewriting, the existingAOFDocuments are also safe;
  • When the child process finishes rewriting, it sends a signal to the parent process. After receiving the signal, the parent process appends all the data in the memory cache to the new processAOFEnd of file.
  • RedisReplace the old file with the new file atomically, and then all commands will be appended directly to the new fileAOFEnd of file.

no-appendfsync-on-rewrite no

When we execute the main process at the same timeWrite operationAnd subprocessesrewriteDuring operation, both will operate the disk, and rewriting often involves a large number of disk operations, which will cause the main process to writeaofBlocking occurs when the file is.

To solve this problem,no-appendfsync-on-rewriteThe parameters are out.

  • If this parameter is set tono, is the safest way. It won’t lose data, but it has to endure the problem of blocking;
  • If set toyes, this is equivalent toappendfsyncSet tono, which means that no disk operation is performed, but the buffer is written. Therefore, this will not cause blocking (because there is no competing disk), but if at this timeredisHang up and lose data. How much data is lost? staylinuxUnder the default setting of the operating system, data will be lost for up to 30s.

Therefore, if the application system cannot tolerate delay and can tolerate a small amount of data loss, it is set toyes; If the application system cannot tolerate data loss, set tono

auto-aof-rewrite-percentage 100

Override percentage. The default is after the last rewriteaofDouble the file size.

auto-aof-rewrite-min-size 64mb

Minimum value of rewrite trigger: 64MB.

according toauto-aof-rewrite-min-sizeandauto-aof-rewrite-percentageParameters determine the automatic trigger timing.RedisThe last time it was rewritten is recordedAOFThe default configuration is whenAOFFile size is lastrewriteDouble the size after and the file is larger than64MTriggered when.

Large Internet companies are generally3Gstart

aof-load-truncated yes

WhenAOFWhen the file is truncated, that isAOFThe last command of the file is incomplete. If you start at this timeRedis, willAOFWhen the data is loaded back into memory, a problem occurs.

  • Yes: load a truncatedAOFRedisThe server starts sending logs to notify the user of the event;
  • No: the server will abort with an error and refuse to start.

When we knowAOFWhen an error is reported in a file, the following methods can be used to repair the errorAOFFile:

  • For existingAOFCreate a backup of the file;

  • useRedisIncidentalredis-check-aofCommand, to the originalAOFRepair the file;

    • redis-check-aof –fix

    • redis-check-aof --fix appendonly.aofRepair the command and kill all non-conforming syntax

  • Optional)Usediff -uCompare the repairedAOF Documents and originalAOFBackup of files and check the differences between the two files;

  • restartRedisServer, wait for the server to load the repairedAOF File and data recovery.

aof-use-rdb-preamble yes

RewritingAOFWhen you file a file,RedisCan be inAOFUsed in documentsRDBPreamble to speed up rewriting and recovery. When on, overrides theAOFThe document consists of two different sections:RDB fileAOF tail

loadRedisWill recognizeAOFFile toRedisStart with a string and load prefixedRDBFile, and then continue loadingAOFTail.

theory

advantage

  • Higher data integrity and consistency,AOFBy using different strategies, the persistence of can lose up to 1 second of data;
  • AOFFile is a log file that is only appended and does not need to be writtenseek
  • RedisCan be inAOF When the file volume becomes too large, it is automatically adjusted in the backgroundAOFRewriting is absolutely safe;
  • AOFWrite operation of file recordRedisThe format of the protocol is saved, which is easy to read and analyze the documents;

shortcoming

  • For the same dataset,AOFThe volume of the file is usually larger thanRDBVolume of documents;
  • Based on thefsyncStrategy,AOFMay be slower thanRDB

In general, every secondfsyncThe performance is still very high while offfsyncCan letAOFSpeed andRDBAs fast, even under high load. But when dealing with huge write loads,RDBIt can provide more guaranteed maximum delay time(latency)。

Comparison and summary

How to choose which persistence method to use?

Generally speaking, if you want to be comparablePostgreSQLFor data security, two persistence functions should be used at the same time.

If you are very concerned about data, but can still withstand data loss within a few minutes, you can only useRDBPersistence.

Because AOF persistence has better real-time performance, that is, less data is lost when the process exits unexpectedlyAOFAt presentmain streamPersistent mode.

Many users only useAOFPersistence, but we don’t recommend this method: because it is generated regularlyRDBSnapshotsnapshot)Very convenient for database backup, andRDBRestoring datasets is also faster thanAOFRecover faster.

Interaction between AOF and RDB

When the version number is greater than or equal to2.4ofRedisIn,BGSAVECannot execute during executionBGREWRITEAOF。 Conversely, inBGREWRITEAOFIn the process of execution, it cannot be executedBGSAVE。 This prevents twoRedisThe background process makes a lot of changes to the disk at the same time I/OOperation.

IfBGSAVEIs executing, and the user explicitly callsBGREWRITEAOFCommand, the server will reply to the user with aOKStatus and inform the userBGREWRITEAOFHas been scheduled for execution: onceBGSAVEAfter execution,BGREWRITEAOF It will officially begin.

WhenRedisWhen starting, ifRDBPersistence andAOFIf the persistence is turned on, the program will take priorityAOFFile to recover the dataset becauseAOF The data saved in the file is usually the most complete.

Backup redis data

  • Create a recurring task(cron job), one per hourRDBFiles are backed up to a folder, and one file is saved every dayRDBBackup files to another folder;
  • Make sure that the backup of the snapshot has the corresponding date and time information. Each time the periodic task script is executed, use thefindCommand to delete expired snapshots;
  • At least once a day, willRDBBack up outside your data center, or at least back up until you runRedisOutside the physical machine of the server.

Performance recommendations

In practical application, becauseRDBThe document is only used for backup purposes. It is recommended to use it only inslaveUpper persistenceRDBFiles, and only need 15 minutes to back up once is enough, only retainsave 900 1This rule.

If onAOFThe advantage is that in the worst case, only less than 2 seconds of data will be lost, and the startup script is relatively simpleloadownAOFJust file it. The first cost is to bring sustainedIO, secondAOF rewrite In the endrewriteThe blocking caused by new data written to new files is almost inevitable.

As long as the hard disk license, it should be minimizedAOF rewriteFrequency,AOFOverridden base size default64MIt’s too small. You can set it to5Gabove. When the default exceeds 100% of the original size, the rewriting can be changed to an appropriate value.

If notAOF, justMaster-Slave ReplicationHigh availability is also possible. Can save a lot of moneyIO, also reducedrewriteSystem fluctuation caused by. The price is ifMaster/Slave If it is dropped at the same time, more than ten minutes of data will be lost. The startup script should also compare the twoMaster/SlaveMediumRDBFile, load the newer one.

That’s all for today, if you have different opinions or betteridea, welcome to contact ah q, add ah q, you can join the technical exchange group to participate in the discussion!

Recommended Today

Cloud Native Virtualization: Building Edge Computing Instances with Kubevirt

With the popularity of Kubernetes, more and more businesses are running on containers, but there are still some business forms that are more suitable for running on virtual machines. How to control virtual machines and containers at the same time has gradually become a mainstream demand in the cloud-native era. Kubevirt gave came up with […]