Tag：partition
-
Time：2020-1-24
Today, let’s talk about Kafka, a popular message queuing component in big data scenarios. This article will mainly introduce from the theoretical point of view. Kafka is an open source, high throughput, real-time, and persistent streaming message queue, which can handle both online (message) and offline applications (business data and logs). In today’s hot big […]
-
Time：2020-1-23
[TOC] Section I Kafka cluster Before inheriting If you are a developer and are not interested in building Kafka cluster, you can skip this chapter and look at tomorrow’s content directly If you think it’s no harm to know more, please keep reading As a reminder, there are many figures in this chapter Kafka cluster […]
-
Time：2020-1-19
Grape In recent days, I have simply studied Kafka, read some books and looked up some materials. Combined with these, I have briefly summarized some basic knowledge of Kafka, so as to make a recordAccording to the old rules, we have three questions about this article: What is Kafka? What are the advantages and disadvantages? […]
-
Time：2020-1-18
Welcome to WeChat’s public address.Learn java from xiaoha, daily push the dry article in Java field, pay attention to that 100g massive learning and interview resources are attached for free!! Personal website: https://www.exception.site/springboot/spring-boot2-kafka What is Kafka? Kafka is an open-source distributed publish subscribe message middleware and flow processing platform of Apache foundation. It originated from […]
-
Time：2020-1-17
Summary During the operation of kafkaproducer, the cluster meta information is needed, which is stored in cluster Node, topicpartition and partitioninfo Cluster meta information is composed of three types of objectsNode, TopicPartition, PartitionInfo Node source code node represents a node in the cluster Topicpartition source topicpartition represents a partition of topic Partitioninfo source code partitioninfo […]
-
Time：2020-1-16
Summary As mentioned in the previous sectionClusterIt is mainly used to store node information, topicpartitionpartition information and partitioninfo partition details. This section is aboutMetadataWill beClusterEncapsulation provides a unified query entry, and also provides the update operation of cluster information. Class annotation By readingMetadataThe following conclusions can be drawn from the notes: Metadata is shared by […]
-
Time：2020-1-13
Kafka is a real-time data processing framework open-source by Linkin, which has been updated to version 2.3. Different from the general message middleware, Kafka obtains extremely high throughput through data persistence and disk reading and writing, and can carry out real-time flow processing independently of the stream processing platform of storm and spark streaming. Kakfa’s […]
-
Time：2020-1-12
During partition test, it is found that after gdisk partition is used, fdisk cannot be used for partition, even if it is formatted. When fdisk is used to view the hard disk, it is found that the hard disk has become a GPT partition and cannot be partitioned through fdisk. Therefore, GPT partition should be […]
-
Time：2020-1-7
1.1 overall architecture Kafka has a reservoir? Don’t worry, everyone. Let’s go to the previous structure chart. From the above architecture chart, it can be seen that the production process is mainly a producer thread and a sender thread. They get data through batchqueue. Their relationship is one-to-one, so Kafka’s production process is asynchronous. Its […]
-
Time：2020-1-6
The WeChat public is “back-end”, focusing on the sharing of back-end technologies: Java, Golang, WEB framework, distributed middleware, service governance, and so on. Recently, I was studying Kafka and found that there are some differences between its core concepts and rocketmq. Next, I will talk about the differences between Kafka partition and rocketmq queue. Rocketmq […]
-
Time：2020-1-5
Schematic diagram of Kafka zoning Recently, Kafka has been used as a consumption queue. The working principle of partitions on the Internet is that there are many text descriptions, so I drew a sketch and shared it Kafka writes data to a partition randomly. The data in each partition is in order. As shown in […]
-
Time：2020-1-4
At the beginning of the article, readers are happier, not to mention the structure chart.We introduced this architecture chart in the previous article “Kafka producer’s reservoir mechanism”. In the previous article, we introduced the process of message collection in this chart (we became the “reservoir” mechanism). Here we will introduce another part of it, message […]
