Kafka log segmentation and message searching

Time:2020-3-26

Kafka is a message middleware (later Kafka gradually turns to kafkastream, a loss processing platform). The final storage of messages is in the log.

Kafka’s message is finally sent with the partition under the topic as the final goal, so Kafka’s log storage is also based on the partition.

Log.dir parameter in configuration fileThis parameter can be configured in Kafka configuration file.

Kafka log segmentation and message searching

The above figure shows three directories, representing the three partitions with the topic name of message store. The data files of the three partitions are stored under the three directories respectively. From the directory, we can see that the creation rules of the partition directory areTopicname partition number。 Let’s pick the 0 partition to see which files are stored below.

Kafka log segmentation and message searching

Logsegement (log segment)

The way to write logs in Kafka is to add them in order. When the log file reaches a certain size, it will be segmented to form a new log file. Here, one log file can be used as a log segment. The introduction of log segments facilitates the query (binary search) and location of Kafka data.

Log segments are divided into active log segments and inactive log segments. Only active log segments (currently, only one partition may exist) can be written and read, while inactive log segments can only be read

Log files have many suffixes, such as. Index,. Log,. Snapshot,. Timeindex

category Effect
.index Offset index file
.log log file
.snapshot Log snapshot
.timeindex Time stamp index file
leader-epoch-checkpoint Checkpoint file for replica synchronization

The name of each file has a fixed format. The length of the file name is 20 bits. It is named after the offset value of the first message in the log. If it is not enough, 0 will be added. Therefore, the offset of the first message in the 00000000000000000038.log is 38.

#This command can view the log file of Kafka, as shown in the following figure
bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000038.log --print-data-log

Kafka log segmentation and message searching
From the log file, we can see the start displacement, end displacement, time stamp of the message set, as well as the specific message displacement, time stamp, header and content (payload) information.

#This command can view the offset index file of Kafka
bin/kafka-dump-log.sh --files  00000000000000000000.index
bin/kafka-dump-log.sh --files  00000000000000000038.index
#This command can view the time stamp index file of Kafka
bin/kafka-dump-log.sh --files  00000000000000000000.timeindex

Kafka log segmentation and message searching
Kafka log segmentation and message searching

Log and index files

Configuration item Default value Explain
log.index.interval.bytes 4096 (4K) Increasing the interval density of index entries will affect the interval density and query efficiency in index files
log.segment.bytes 1073741824 (1G) Log file maximum
log.roll.ms The maximum range allowed by the difference between the maximum timestamp of the message in the current log segment and the timestamp of the current system, in milliseconds dimension
log.roll.hours 168 (7 days) The maximum range, hour dimension allowed by the difference between the maximum timestamp of the message in the current log segment and the timestamp of the current system
log.index.size.max.bytes 10485760 (10MB) Trigger offset index file or timestamp index file segment byte limit

Log segmentation

Log segments have current and past log segments. Kafka will open a new file during log segmentation. The trigger log segmentation mainly has the following conditions:

  • The current log segment log file size exceeds the configured size of log.segment.bytes
  • The difference between the maximum time stamp of the message in the current log segment and the time stamp of the system exceeds the millisecond value configured by log.roll.ms
  • The difference between the maximum time stamp of the message in the current log segment and the time stamp of the current system exceeds the hour value configured by log.roll.hours, and the priority is lower than log.roll.ms
  • The index file and timestamp index file in the current log segment exceed the configured size of log.index.size.max.bytes
  • The difference between the offset of the appended message and the offset in the current log segment is greater than the integer.max? Value, which means that the offset of the message to be appended cannot be converted to the relative offset. The reason is that in the offset index file, the message offset based on baseoffset is represented by 4 bytes.

The size of the index file (log.index.size.max.bytes) will be fixed first when the index file is segmented. When a new segment is created, the index file of the previous segment will be cropped. The size of the file represents the actual data size.

Message lookup

Offset lookup (. Index)

The offset index file consists of a 4-byte relative offset and a 4-byte physical address.
Kafka log segmentation and message searching

Kafka internally maintains a concurrentskiplistmap to save in each log segment. By means of a skip table, it locates to a specific log offset index file. Then, in this file, it finds the position no larger than the position corresponding to the offset to be searched according to the dichotomy method, and then it traverses the log file from the position to find the cancellation corresponding to the offset to be searched Interest.

Timestamp lookup (. Timeindex)

The time stamp index file is composed of 8-byte time stamps and 4 own relative offsets.
Kafka log segmentation and message searching
In time stamp search, first compare the time stamp to be searched with the last record of each time stamp index file. If the time stamp of the last record is less than or equal to 0, compare it with the file modification time, and find the time index file no less than the time stamp to be searched. After finding the corresponding log period timestamp index file, dichotomy finds the offset no larger than the time stamp, and then searches the offset file according to the offset.

The offset index file offset is incremental, but the time stamp of the time stamp index file is not incremental. Unless the broker section sets the log.message.timestamp.type parameter to logappendtime, the time stamp can keep monotonous growth. Because in the new version of Kafka producer, the client is allowed to set the time stamp. If the log.message.timestamp.type parameter is set to createtime, it will cause disorder in the actual stamp index file.

Log cleaning

Log cleanup is covered in the next section.