Kafka is a message middleware (later Kafka gradually turns to kafkastream, a loss processing platform). The final storage of messages is in the log.
Kafka’s message is finally sent with the partition under the topic as the final goal, so Kafka’s log storage is also based on the partition.
Log.dir parameter in configuration fileThis parameter can be configured in Kafka configuration file.
The above figure shows three directories, representing the three partitions with the topic name of message store. The data files of the three partitions are stored under the three directories respectively. From the directory, we can see that the creation rules of the partition directory areTopicname partition number。 Let’s pick the 0 partition to see which files are stored below.
Logsegement (log segment)
The way to write logs in Kafka is to add them in order. When the log file reaches a certain size, it will be segmented to form a new log file. Here, one log file can be used as a log segment. The introduction of log segments facilitates the query (binary search) and location of Kafka data.
Log segments are divided into active log segments and inactive log segments. Only active log segments (currently, only one partition may exist) can be written and read, while inactive log segments can only be read
Log files have many suffixes, such as. Index,. Log,. Snapshot,. Timeindex
|.index||Offset index file|
|.timeindex||Time stamp index file|
|leader-epoch-checkpoint||Checkpoint file for replica synchronization|
The name of each file has a fixed format. The length of the file name is 20 bits. It is named after the offset value of the first message in the log. If it is not enough, 0 will be added. Therefore, the offset of the first message in the 00000000000000000038.log is 38.
#This command can view the log file of Kafka, as shown in the following figure bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000038.log --print-data-log
From the log file, we can see the start displacement, end displacement, time stamp of the message set, as well as the specific message displacement, time stamp, header and content (payload) information.
#This command can view the offset index file of Kafka bin/kafka-dump-log.sh --files 00000000000000000000.index bin/kafka-dump-log.sh --files 00000000000000000038.index #This command can view the time stamp index file of Kafka bin/kafka-dump-log.sh --files 00000000000000000000.timeindex
Log and index files
|Configuration item||Default value||Explain|
|log.index.interval.bytes||4096 (4K)||Increasing the interval density of index entries will affect the interval density and query efficiency in index files|
|log.segment.bytes||1073741824 (1G)||Log file maximum|
|log.roll.ms||The maximum range allowed by the difference between the maximum timestamp of the message in the current log segment and the timestamp of the current system, in milliseconds dimension|
|log.roll.hours||168 (7 days)||The maximum range, hour dimension allowed by the difference between the maximum timestamp of the message in the current log segment and the timestamp of the current system|
|log.index.size.max.bytes||10485760 (10MB)||Trigger offset index file or timestamp index file segment byte limit|
Log segments have current and past log segments. Kafka will open a new file during log segmentation. The trigger log segmentation mainly has the following conditions:
- The current log segment log file size exceeds the configured size of log.segment.bytes
- The difference between the maximum time stamp of the message in the current log segment and the time stamp of the system exceeds the millisecond value configured by log.roll.ms
- The difference between the maximum time stamp of the message in the current log segment and the time stamp of the current system exceeds the hour value configured by log.roll.hours, and the priority is lower than log.roll.ms
- The index file and timestamp index file in the current log segment exceed the configured size of log.index.size.max.bytes
- The difference between the offset of the appended message and the offset in the current log segment is greater than the integer.max? Value, which means that the offset of the message to be appended cannot be converted to the relative offset. The reason is that in the offset index file, the message offset based on baseoffset is represented by 4 bytes.
The size of the index file (log.index.size.max.bytes) will be fixed first when the index file is segmented. When a new segment is created, the index file of the previous segment will be cropped. The size of the file represents the actual data size.
Offset lookup (. Index)
The offset index file consists of a 4-byte relative offset and a 4-byte physical address.
Kafka internally maintains a concurrentskiplistmap to save in each log segment. By means of a skip table, it locates to a specific log offset index file. Then, in this file, it finds the position no larger than the position corresponding to the offset to be searched according to the dichotomy method, and then it traverses the log file from the position to find the cancellation corresponding to the offset to be searched Interest.
Timestamp lookup (. Timeindex)
The time stamp index file is composed of 8-byte time stamps and 4 own relative offsets.
In time stamp search, first compare the time stamp to be searched with the last record of each time stamp index file. If the time stamp of the last record is less than or equal to 0, compare it with the file modification time, and find the time index file no less than the time stamp to be searched. After finding the corresponding log period timestamp index file, dichotomy finds the offset no larger than the time stamp, and then searches the offset file according to the offset.
The offset index file offset is incremental, but the time stamp of the time stamp index file is not incremental. Unless the broker section sets the log.message.timestamp.type parameter to logappendtime, the time stamp can keep monotonous growth. Because in the new version of Kafka producer, the client is allowed to set the time stamp. If the log.message.timestamp.type parameter is set to createtime, it will cause disorder in the actual stamp index file.
Log cleanup is covered in the next section.