Redis persistence AOF and RDB

Time:2020-11-24

Redis persistence

  • Redis supports two persistence mechanisms, RDB and AOF. The persistence function can effectively avoid data loss caused by process exit. When the next restart, the previously persistent files can be used to realize data recovery

RDB

  • RDB persistence is a process to generate a snapshot of current process data and save it to the hard disk
  • Manually trigger the corresponding save and bgsave commands respectively:
      1. Save command: block the current redis server until the RDB process is completed. For instances with large memory, it will cause long-term blocking, which is not recommended in online environment
    • redis-cli

Redis persistence AOF and RDB

    • redis-server

Redis persistence AOF and RDB

      1. Bgsave command: redis process executes fork operation to create subprocess. The sub process is responsible for the RDB persistence process, and it will end automatically after completion. Blocking only occurs in the fork phase, which usually takes a short time
    • redis-cli

Redis persistence AOF and RDB

    • redis-server

Redis persistence AOF and RDB

  • All operations related to RDB in redis adopt bgsave mode
  • Persistence mechanism for automatic triggering of RDB:
      1. Using the save related configuration, such as “save m n”, means that bgsave will be automatically triggered when there are n modifications to the dataset within m seconds
      1. If the slave node performs full copy operation, the master node automatically executes bgsave to generate RDB files and sends slave nodes
      1. When you execute the debug reload command to reload redis, the save operation will also be triggered automatically
      1. By default, bgsave is automatically executed if AOF persistence is not enabled when the shutdown command is executed

Bgsave persistence process

Redis persistence AOF and RDB

RDB file processing

  • Save: the RDB file is saved in the directory specified by the dir configuration, and the file name is specified by dbfile name configuration. It can be executed dynamically by executing the run time of config set dir {new dir} and config set dbfile name {new file name}. When the next run, the RDB file will be saved to the new directory.

Redis persistence AOF and RDB

  • In case of bad disk or full disk, you can modify the file path to the available disk path through config set dir {new dir}, and then perform bgsave to switch disks. The same is applicable to AOF persistent files
  • Compression: redis uses lzf algorithm to compress the generated RDB file by default. The compressed file is much smaller than the memory size. It is enabled by default. It can be dynamically modified through the parameter config set rdbcompression {yes | no}. The configuration file is as follows:

Redis persistence AOF and RDB

  • Verification: if redis refuses to start when loading a damaged RDB file, the redis check dump tool provided by redis can be used to detect the RDB file and obtain the corresponding error report

AOF

  • Aof (append only file) persistence: record every write command in the form of independent log, and re execute the command in AOF file to recover data during restart
  • Function: solve the real-time of data persistence
  • To enable the AOF function, you need to set the configuration: appendonly yes, which is not enabled by default. The AOF file name is set by appendinfilename. The default file name is appendonly.aof .

Redis persistence AOF and RDB

  • Aof workflow:

Redis persistence AOF and RDB

  • The content written by AOF command is directly in text protocol format
  • Why does AOF append commands to AOF_ BUF?
    • Redis uses a single thread response command. If every command to write an AOF file is directly appended to the hard disk, then the performance depends entirely on the current hard disk load. Write the buffer AOF first_ Another advantage of buf is that redis can provide multiple buffer synchronization strategies to balance performance and security
  • The AOF buffer synchronization file policy is controlled by the parameter appendfsync
Configurable value explain
always Command write to AOF_ After buf, the system fsync operation is synchronized to the AOF file. After fsync completes, the thread returns.
everysec Command write to AOF_ After buf, the system write operation is invoked and the thread is returned after write completes. The fsync synchronization file operation is called once per second by a dedicated thread
no Command write to AOF_ After buf, the system write operation is invoked, fsync synchronization is not performed on AOF files, and synchronous disk operation is responsible for the operating system. The synchronization cycle is the longest 30 seconds.
  • The system calls write and fsync
    • The write operation triggers the delayed write mechanism. Linux provides page buffer in the kernel to improve the IO performance of hard disk. The write operation returns directly after writing to the system buffer. Synchronous hard disk operation depends on system scheduling mechanism, such as buffer page space full or reaching a specific time period. Before synchronizing files, if the system is down at this time, the data in the buffer will be lost
  • Fsync performs forced hard disk synchronization for a single file operation (such as AOF file). Fsync will block until it is written to the hard disk and returns to ensure data persistence.
  • Aof file rewriting is the process of converting the data in the redis process into a write command and synchronizing it to the new AOF file.
  • Why can the rewritten AOF file be smaller?
      1. Data that has timed out in the process is no longer written to the file
      1. Rewriting is generated directly using in-process data, so that the new AOF file only retains the write command for the final data
      1. Multiple write commands can be combined into one
  • Aof rewriting reduces the file space. In addition, the other purpose is that smaller AOF files can be loaded faster by redis