Pulsar vs. Kafka (2): segment centric architecture


Original: Sijie Guo

Zhai Jia

In the last article, we explored Apache pulsar’s message model, which unifies high-performance flows and flexible queues. The comparison shows how Apache pulsar and Apache Kafka implement message consumption, confirmation and retention in the messaging model. In this article, we will introduce some system architectures and design concepts behind Apache pulsar, and finally make some comparisons with Apache Kafka’s architecture.

Pulsar’s layered architecture

The most fundamental difference between Apache pulsar and other messaging systems is the layered architecture. Apache pulsar cluster consists of two layers: stateless service layer, which is composed of a group of brokers that receive and deliver messages; And a stateful persistence layer, which is composed of a set of Apache bookkeeper storage nodes named books, which can persistently store messages. The following figure shows a typical deployment of Apache pulsar.

Pulsar vs. Kafka (2): segment centric architecture

The producer & consumer interface is provided in the pulsar client. The application uses the pulsar client to connect to the broker to publish and consume messages.

Pulsar clients do not interact directly with the storage tier Apache bookkeeper. The client also does not have direct zookeeper access. This isolation provides a basis for pulsar to implement a secure multi tenant unified authentication model.

Apache pulsar supports clients in multiple languages, including Java, C + +, python, go and WebSockets.

Apache pulsar also provides a set of Kafka compatible APIs. Users can migrate existing Kafka applications by simply updating dependencies and pointing clients to pulsar clusters, so that existing Kafka applications can be used with Apache pulsar immediately without changing any code.

Broker layer — stateless service layer

Broker clusters form a stateless service layer in Apache pulsar. The service layer is “stateless” because the broker does not actually store any message data locally. Messages about pulsar topics are stored in the distributed log storage system (APACHE bookkeeper). We will discuss bookkeeper more in the next section.

Each topic partition is assigned by pulsar to a broker, which is called the owner of the topic partition. Pulsar producers and consumers connect to the owner broker of the topic partition to send messages to the owner agent and consume messages.

If a broker fails, pulsar will automatically move its own subject partition to one of the remaining available brokers in the cluster. One thing to say here is: because brokers are stateless, when topic migration occurs, pulsar only transfers ownership from one broker to another. In this process, there will be no data replication.

The following figure shows a pulsar cluster with four brokers, in which four topic partitions are distributed in four brokers. Each broker owns and provides message services for a topic partition.

Pulsar vs. Kafka (2): segment centric architecture

Bookkeeper layer – persistent storage layer

Apache bookkeeper is the persistent storage layer of Apache pulsar. Each topic partition in Apache pulsar is essentially a distributed log stored in Apache bookkeeper.

Each distributed log is divided into segments. Each segment, as a ledger in Apache bookkeeper, is evenly distributed and stored in multiple bookies (storage nodes of Apache bookkeeper) in the bookkeeper cluster. The creation opportunities of segments include the following: Based on the configured segment size; Rolling time based on configuration; Or when the owner of the segment is switched.

By means of segment segmentation, the messages in the topic partition can be evenly and evenly distributed among all bookies in the cluster. This means that the size of the subject partition is not only limited by the capacity of one node; Instead, it can scale to the total capacity of the entire bookkeeper cluster.

The following figure illustrates a topic partition divided into x segment segments. Each segment stores 3 copies. All segments are distributed and stored in four bookies.

Pulsar vs. Kafka (2): segment centric architecture

Segment centric storage

The layered architecture of storage services and segment centric storage are two key design concepts of Apache pulsar (using Apache bookkeeper). These two foundations provide pulsar with many important benefits:

  • Unlimited subject partition storage
  • Immediate expansion without data migration
  • Seamless broker failover
  • Seamless cluster expansion
  • Seamless storage (bookie) failover
  • Independent scalability

Now let’s expand to see several benefits.

Unlimited subject partition storage

Since the topic partition is divided into segments and stored in Apache bookkeeper in a distributed manner, the capacity of the topic partition is not limited by the capacity of any single node. On the contrary, the theme partition can be extended to the total capacity of the entire bookkeeper cluster. Just add a bookie node to expand the cluster capacity. This is the key to Apache pulsar’s support for storing unlimited stream data and processing data in an efficient and distributed manner. Distributed log storage using Apache bookkeeper is crucial for unified messaging services and storage.

Immediate expansion without data migration

Since the message service and message storage are divided into two layers, moving the subject partition from one broker to another can be completed almost instantaneously without any data rebalancing (copying the data from one node to another). This feature is crucial for many aspects of high availability, such as cluster expansion; Quick response to broker and bookie failures. I will use examples to explain in more detail below.

  • Seamless broker failover

The following figure illustrates an example of how pulsar handles broker failures. In the example, broker 2 is disconnected for some reason (such as power failure). Pulsar detects that broker 2 is closed and immediately transfers the ownership of topic1-part2 from broker 2 to broker 3. In pulsar, data storage and data service are separated, so when agent 3 takes over the ownership of topic1-part2, it does not need to copy partiton’s data. If new data arrives, it is immediately attached and stored as segment x + 1 in topic1-part2. Segment x + 1 is distributed and stored on bookie1, 2 and 4. Because it does not need to re replicate the data, the ownership transfer occurs immediately without sacrificing the availability of the subject partition.

Pulsar vs. Kafka (2): segment centric architecture

  • Seamless cluster capacity expansion

The following figure illustrates how pulsar handles the capacity expansion of a cluster. When broker 2 writes messages to segment X of topic1-part2, bookie X and bookie y are added to the cluster. Broker 2 immediately finds the newly added bookies X and y. The broker will then try to store the messages of segment x + 1 and X + 2 in the newly added bookie. The newly added bookie is used immediately, and the traffic increases immediately without re copying any data. In addition to rack aware and area aware policies, Apache bookkeeper also provides resource aware placement policies to ensure that traffic is balanced among all storage nodes in the cluster.

Pulsar vs. Kafka (2): segment centric architecture

  • Seamless storage (bookie) failover

The following figure illustrates how pulsar (through Apache bookkeeper) handles the disk failure of bookie. There is a disk failure that destroys segment 4 stored on bookie 2. The Apache bookkeeper background will detect this error and copy and fix it.

Pulsar vs. Kafka (2): segment centric architecture

Replica repair in Apache bookkeeper is a many to many fast repair at the segment (or even entry) level, which is finer than re replicating the entire topic partition and will only copy the necessary data. This means that Apache bookkeeper can read messages in segment 4 from bookie 3 and bookie 4 and repair segment 4 at bookie 1. All replica repairs are performed in the background and are transparent to brokers and applications. Even if an error occurs in the bookie node, by adding a new available bookie to replace the failed bookie, all brokers can continue to accept writes without sacrificing the availability of the topic partition.

Independent scalability

Since the message service layer and the persistent storage layer are separate, Apache pulsar can independently extend the storage layer and the service layer. This independent expansion is more cost-effective:

When you need to support more consumers or producers, you can simply add more brokers. The topic partitions will be migrated in a balanced manner in brokers immediately, and the ownership of some topic partitions will be transferred to new brokers immediately.

When you need more storage space to keep messages longer, you just need to add more bookie. Through intelligent resource awareness and data placement, traffic will automatically switch to the new bookie. Apache pulsar will not involve unnecessary data relocation and will not re copy old data from existing storage nodes to new storage nodes.

Comparison with Kafka

Apache Kafka and Apache pulsar have similar message concepts. The client interacts with the message system through the topic. Each topic can be divided into multiple partitions. However, the fundamental difference between Apache pulsar and Apache Kafka is that Apache Kafka takes partition as the storage center, while Apache pulsar takes segment as the storage center.

Pulsar vs. Kafka (2): segment centric architecture

The figure above shows the difference between zone centric and segment centric systems.

In Apache Kafka, partitions can only be stored on a single node and copied to other nodes, and their capacity is limited by the minimum node capacity. This means that capacity expansion requires rebalancing the partitions, which in turn requires re replicating the entire partition to balance the data and traffic of the newly added agents. Retransmission of data is very expensive and error prone, and consumes network bandwidth and I / O. Maintenance personnel must be very careful when performing this operation to avoid damaging the production system.

The recopy of partition data in Kafka does not only occur on cluster expansion in partition centric systems. Many other things can also trigger data re copying, such as copy failure, disk failure or computer failure. During data re replication, partitions are usually unavailable until data re replication is complete. For example, if you configure a partition to be stored as 3 replicas, if one replica is lost, you must replicate the entire partition before the partition can be used again.

This defect is usually ignored before the user encounters a failure, because in many cases, it is only the reading of cached data in memory in a short time. When the data is saved to disk, users will inevitably encounter more and more problems of data loss and fault recovery, especially when the data needs to be saved for a long time.

On the contrary, in Apache pulsar, partitions are also used as logical units, but segments are used as physical storage units. Partitions are segmented over time and evenly distributed throughout the cluster in order to expand effectively and rapidly.

Pulsar is segment centric, so there is no need to rebalance and copy data when expanding capacity, and the old data will not be replicated. This is due to the use of scalable segment centric distributed log storage system in Apache bookkeeper.

By utilizing distributed log storage, pulsar can maximize the segment placement options for high write and read availability. For example, with bookkeeper, the copy setting is equal to 2. As long as any two bookies are started, you can write to the theme partition. For read availability, as long as one replica set of the subject partition is active, users can read it without any inconsistency.


In conclusion, Apache pulsar, a unique segment centric publish / subscribe messaging system based on distributed log storage, can provide many advantages, such as reliable streaming system, including unlimited log storage, immediate expansion without partition rebalancing, rapid replication and repair, and high write and read availability options by maximizing data placement.

In this article, we give an overview of the architecture of Apache pulsar. Compared with other streaming messaging systems, the layered architecture and segmentation centered design in Apache pulsar are two unique design concepts. I hope readers can better understand Apache pulsar from the perspective of architecture and developers, and understand the differences between Apache pulsar and Apache Kafka.

If you are interested in pulsar, you can participate in the pulsar community in the following ways:

For general information about the Apache pulsar project, please visit the official website:http://pulsar.incubator.apach…You can also follow the twitter account @ Apache_ pulsar。

Recommended Today

Heavyweight Tencent cloud open source industry’s first etcd one-stop governance platform kstone

​ Kstone open source At the kubecon China Conference held by CNCF cloud native foundation on December 9, 2021,Tencent cloud container tke team released the open source project of kstone etcd governance platform. KstoneIt was initiated by the TKE team of Tencent cloud containerCloud native one-stop etcd governance project based on kubernetes。 The project originates […]