Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Time:2021-12-7

About Apache pulsar

Apache pulsar is a top-level project of the Apache Software Foundation. It is a native distributed message flow platform for the next generation cloud. It integrates message, storage and lightweight functional computing. It adopts a separate architecture design of computing and storage, supports multi tenant, persistent storage, multi machine room cross regional data replication, and has strong consistency, high throughput Stream data storage features such as low latency and high scalability. At present, many large Internet and traditional industry companies at home and abroad have adopted Apache pulsar. The cases are distributed in artificial intelligence, finance, telecom operators, live and short video, Internet of things, retail and e-commerce, online education and other industries, such as American cable TV network giants Comcast, Yahoo Tencent, China Telecom, China Mobile, bigo, vipkid, etc.

Confluent recently conducted a benchmark test to compare the throughput and latency differences of Kafka, pulsar and rabbit MQ. Confluent blog shows that kakfa can achieve “optimal throughput” with “low latency”, while rabbitmq can achieve “low latency” with “low throughput”. Overall, the benchmark results show that Kafka is undoubtedly better in “speed”.

Kafka technology is mature and perfect, but today many companies (from multinational companies to innovative start-ups) choose pulsar first. At the recent Splunk summit conf 20, sendur sellakumar, chief product officer of Splunk company, announced that they decided to replace Kafka with pulsar:

“… we have used Apache pulsar as the basic flow. We have put the future of the company on the long-term architecture of enterprise class multi tenant flow.”

–Sendur sellakumar, chief product officer, Splunk

Many companies are using pulsar, Splunk is just one example. The reason why these companies choose pulsar is that in the modern elastic cloud environment (such as kubernetes), pulsar can horizontally expand and process massive data in a cost-effective way without the problem of single point of failure. Meanwhile, pulsar has many built-in features, such as automatic data rebalancing, multi tenancy, cross regional replication and persistent tiered storage, which not only simplifies operation and maintenance, but also makes it easier for the team to focus on business objectives.

Developers finally chose pulsar because its unique functions and performance make pulsar the cornerstone of streaming data.

After understanding these situations, you also need to carefully study confluent’s benchmark settings and conclusions. We found two highly controversial issues. First, confluent has limited knowledge of pulsar, which is the biggest source of inaccurate conclusions. If you don’t know pulsar, you can’t test pulsar performance with the correct metrics.

Second, confluent’s performance test is based on a narrow set of test parameters. This limits the applicability of the results, and can not provide readers with accurate results matching different workloads and practical application scenarios.

In order to provide more accurate test results to the community, we decided to solve these problems and repeat the test. Important adjustments include:

  1. We adjusted the benchmark settings to include the persistence levels supported by pulsar and Kafka, and compared their throughput and latency at the same persistence level.
  2. We fixed the openmessaging Benchmarking (OMB) framework, eliminated variables due to the use of different instances, and corrected configuration errors in the OMB pulsar driver.
  3. Finally, we measured other performance factors and conditions, such as the different number of partitions and mixed workloads including write, tailing read, and catch up read, to gain a more comprehensive understanding of performance.

After completing these tasks, we repeated the test. The test results show that for scenarios closer to the real workload, the performance of pulsar is significantly better than Kafka, while for the basic scenarios used by confluent in the test, the performance of pulsar is equivalent to that of Kafka.

The following sections will highlight the most important conclusions of this test. In the streamnative benchmark results section, we describe the test setup and test report in detail.

Summary of streamnative benchmark results

1. Under the same persistence guarantee as Kafka, pulsar can achieve 605 MB / s release and end-to-end throughput (the same as Kafka) and 3.5 GB / s catch up read throughput (3.5 times higher than Kafka). The throughput of pulsar will not be affected by the increase of the number of partitions and the change of the persistence level, while the throughput of Kafka will be seriously affected by the change of the number of partitions or the persistence level.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

2. In different test cases (including different subscriptions, different topics and different persistence guarantees), the delay of pulsar is significantly lower than that of Kafka. Pulsarp99 has a delay between 5 and 15 milliseconds. Kafkap99 latency can be as long as several seconds and can be greatly affected by the number of topics, subscriptions and different persistence guarantees.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)
Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

3. Pulsar’s I / O isolation is significantly better than Kafka. When consumers catch up to read historical data, the release delay of pulsarp99 is still about 5ms. In contrast, Kafka’s delay will be seriously affected by catch up read. Kafkap99 release latency may increase from milliseconds to seconds.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

All our benchmarks are open source(  GitHub website), interested readers can generate their own results, or study the test results and the indicators provided in the warehouse more deeply.

Although our benchmark is more accurate and comprehensive than confluent’s benchmark, it does not cover all scenarios. In the final analysis, testing through your own hardware / actual workload cannot be replaced by any benchmark. We also encourage readers to evaluate other variables and scenarios and test them with their own settings and environment.

Delve deeper into confluent benchmarks

Confluent took the openmessaging benchmark (OMB) framework as the basis for its benchmark, and made some modifications. In this section, we will describe the problems found in the fluent benchmark and explain how these problems affect the accuracy of the fluent test results.

Setting problem of confluent

The conclusion of confluence benchmark test is incorrect because the pulsar parameter setting is unreasonable. We will explain these issues in detail in the streamnative benchmark section. In addition to pulsar tuning, confluent sets different persistence guarantees for pulsar and Kafka. The persistence level will affect the performance. If the persistence settings of the two systems are the same, the comparison is of reference value.

Confluent engineers adopt the default persistence guarantee for pulsar, which is higher than Kafka’s persistence level. Increasing the persistence level will seriously affect the delay and throughput, so the fluent test puts forward higher requirements for pulsar than Kafka. The pulsar version used by confluent does not support reducing the persistence to the same level as Kafka, but the upcoming version of pulsar supports this level, which is also used in this test. If the conflict engineer uses the same persistence settings on both systems, the test results should show an accurate comparison. Of course, we won’t blame confluent engineers for not using unpublished features. However, test records do not provide the necessary scenarios and are regarded as the result of equivalent persistence settings. This article will provide additional scenario descriptions.

OMB framework issues

The confluent benchmark follows the OMB framework guide, which recommends using the same instance type in multiple event flow systems. However, in the test, we found a large number of deviations in different instances of the same type, especially in the case of disk I / O failure. In order to minimize this difference, we use the same instances every time we run pulsar and Kafka. We find that these instances greatly improve the accuracy of the results. Small differences in disk I / O performance may cause great differences in the overall performance of the system. We propose to update the OMB framework guidelines and consider adopting this recommendation in the future.

Problems of confluent research method

The confluent benchmark tested only a limited number of scenarios. For example, the actual workload includes writing, tailing read, and catch up read. Tailing read occurs when a consumer is reading the latest message near the “tail” of the log. Confluent only tested this scenario. In contrast, catch up read occurs when consumers have a large number of historical messages and must be consumed to the “catch up” position to the tail of the log, which is a common key task in the actual system. If catch up read is not considered, the latency of writing and tailing read will be seriously affected. Because the confluent benchmark focuses only on throughput and end-to-end latency, it fails to provide comprehensive results on the expected behavior under various workloads. In order to further make the results closer to the actual application scenario, we think it is very important to benchmark different numbers of subscriptions and partitions. Few enterprises are only concerned with a small number of topics with several partitions and consumers. They need the ability to accommodate a large number of different consumers with different themes / partitions to map to business use cases.

In the following table, we summarize the specific problems of confluent research method.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Many problems with confluent benchmarking stem from limited understanding of pulsar. In order to help you avoid these problems during subsequent benchmarking, we share some pulsar technical insights with you.

In order to conduct accurate benchmarking, it is necessary to understand the durability guarantee of pulsar. We will discuss this issue as a starting point, first give an overall overview of the persistence of distributed systems, and then explain the differences between pulsar and Kafka in persistence assurance.

Overview of distributed system persistence

Persistence refers to the ability to maintain system consistency and availability in the face of external problems such as hardware or operating system failures. Single node storage systems such as RDBMS rely on fsync to write to disk to ensure maximum persistence. The operating system usually caches writes, which may be lost in the event of a failure, but fsync will ensure that these data are written to physical storage. In distributed systems, persistence usually comes from data replication, that is, multiple copies of data are distributed to different nodes that can fail independently. However, local persistence (fsync data) should not be confused with replication persistence for different purposes. Next, we will explain the importance of these features and the main differences.

Replication persistence and local persistence

Distributed systems usually have both replication persistence and local persistence. Various types of persistence are controlled by separate mechanisms. These mechanisms can be flexibly combined to set different persistence levels as needed.

Replication persistence is realized by creating multiple copies of data through one algorithm, so the same data can be stored in multiple locations, and the availability and accessibility can be improved. The number of replicas n determines the fault tolerance of the system. Many systems need “arbitration” or N / 2 + 1 nodes to confirm writing. Some systems can continue to serve existing data while any single copy is still available. This replication mechanism is very important to deal with the complete loss of instance data. New instances can re replicate data from existing replicas, which is also very important to availability and consensus (this issue is not discussed in this section).

In contrast, local persistence determines the different understanding of validation at each node level. Local persistence requires data fsync to persistent storage to ensure that no data will be lost even in case of power failure or hardware failure. Fsync of data ensures that the node has all the previously confirmed data after the machine recovers in a short time.

Persistence patterns: synchronous and asynchronous

Different types of systems provide different levels of persistence assurance. Generally, the overall persistence of the system is affected by the following factors:

  • Does the system fsync the data to the local disk
  • Does the system copy data to multiple locations
  • When does the system confirm replication to the peer system
  • When does the system confirm writing to the client

In different systems, these choices vary greatly, and not all systems support users to control these values. Systems that lack some of these mechanisms (such as replication in non distributed systems) have lower persistence.

We can define two persistence modes, both of which can control when the system confirms writing for internal replication and when it writes to the client, namely “synchronous” and “asynchronous”. These two modes operate as follows.

  • Synchronous persistence: the system returns a write response to the peer system / client only after the data is successfully fsync to the local disk (local persistence) or copied to multiple locations (replication persistence).
  • Asynchronous persistence: the system will return a write response to the peer system / client before the data is successfully fsync to the local disk (local persistence) or copied to multiple locations (replication persistence).

Persistence level: measure persistence assurance

Persistence guarantees exist in many forms, depending on the following variables:

  • Is the data stored locally, replicated in multiple locations, or both
  • When to acknowledge a write (synchronous / asynchronous)

Like the persistence pattern, we define four levels of persistence to distinguish different distributed systems. Table 6 lists the levels from highest persistence to lowest persistence.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Most distributed relational database management systems (such as newsql database) can guarantee the highest level of persistence, so they are classified as level 1.

Like databases, pulsar is a level 1 system that provides the highest level of persistence by default. In addition, pulsar can customize the required persistence level for each application. In contrast, most of Kafka’s production environment deployments are configured at level 2 or 4. It is reported that through setting  flush.messages=1  and  flush.ms=0, Kafka can also meet the level 1 standard. However, these two configurations will seriously affect throughput and latency, which will be discussed in detail in the benchmark.

Let’s start with pulsar and explore the persistence of each system in detail.

Pulsar persistence

Pulsar provides various levels of persistence assurance, which can copy data to multiple locations and fsync data to local disks. Pulsar has two persistence modes (synchronous and asynchronous as described above). Users can customize the first mock exam according to the usage scenario, use one mode alone or combine it.

Pulsar uses a raft equivalent, arbitration based replication protocol to control the persistence of replication. By adjusting  ack-quorum-size  and  write-quorum-size  Parameter to adjust the replication persistence mode. Table 7 lists the settings of these parameters, and table 8 lists the persistence levels supported by pulsar. (pulsar replication protocol and consensus algorithm are beyond the scope of this article. We will explore this field in a follow-up blog.)

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)
Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Pulsar controls local persistence by writing and / or fsync data to the log disk. Pulsar also provides the option to adjust the local persistence mode through the configuration parameters in Table 9.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Kafka persistence

Kafka provides three persistence levels: Level 1, level 2 (default) and level 4. Kafka provides replication persistence at level 2 and cannot provide persistence guarantee at level 4 because it does not have the ability to send data fysnc to disk before confirming writing. By setting  flush.messages=1  and  flush.ms=0, Kafka can reach the level 1 system level, but Kafka has rarely deployed this configuration in production environment.

Kafka’s ISR replication protocol controls replication persistence. By adjusting the associated with this Agreement  acks  and  min.insync.replicas  Parameter, you can adjust Kafka’s replication persistence mode. Table 10 lists the settings of these parameters. Table 11 lists the persistence levels supported by Kafka. (the detailed description of Kafka replication protocol is beyond the scope of this article. We will dig into the differences between Kafka protocol and pulsar protocol in subsequent blogs.)

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)
Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Unlike pulsar, Kafka does not write data to a separate log disk. Kafka will confirm the write operation first, and then fsync the data to the disk. This operation minimizes I / O contention between writes and reads and prevents performance degradation.

By setting for each message  flush.messages = 1  and  flush.ms = 0, Kafka can provide fsync function and greatly reduce the possibility of message loss, but this will seriously affect throughput and latency. Therefore, this setup has almost never been used for production environment deployment.

Kafka cannot transmit log data. In case of machine failure or power failure, there is a risk of data loss. This defect is obvious and has a great impact, which is also one of the main reasons why Tencent billing system selects pulsar.

Differences in persistence between pulsar and Kafka

Pulsar has flexible persistence settings. Users can optimize persistence settings according to specific needs to meet the requirements of various applications, application scenarios or hardware configurations.

Kafka has poor flexibility. According to scenario constraints, it cannot ensure that the persistence settings in the two systems are the same. This makes benchmarking more difficult. To solve this problem, the OMB framework recommends using the closest available settings.

With these backgrounds in mind, let’s take a look at the problems in the confluent benchmark. Confluent tries to simulate the fsync behavior of pulsar. In their benchmark, confluent sets the asynchronous persistence function for Kafka and the synchronous persistence function for pulsar. This inequality leads to incorrect test results and biased performance judgment. Our benchmark test shows that pulsar is comparable to or even better than Kafka, and pulsar also provides stronger persistence guarantee.

Streamnative benchmark

In order to understand pulsar performance more accurately, we need to use fluent benchmark to solve these problems. We focus on adjusting the configuration of pulsar to ensure that the persistence settings of the two systems are the same, and incorporate other performance factors and conditions, such as different partition numbers and mixed workloads, so as to measure the performance under different application scenarios. In the following sections, we will describe the configuration adjustment in our test in detail.

Streamnative test settings

Our benchmark settings include all the persistence levels supported by pulsar and Kafka. In this way, we can compare throughput and latency at the same persistence level. The persistence settings we use are as follows.

Replication persistence settings

Our replication persistence settings are the same as confluent without any changes. In order to maintain integrity, the settings listed in Table 12 are used here.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

New features of pulsar(  New features:)The option of skipping logging is provided for applications, which relaxes the local persistence guarantee, avoids write amplification and improves write throughput. (this feature is available in the next version of Apache bookkeeper.). We did not set it as the default feature, because it is still possible to lose messages, so it is not recommended to use it in most scenarios.

To ensure accurate performance comparison between the two systems, we used this feature in the benchmark. Logging bypassing pulsar provides the same local persistence guarantee as Kafka’s default fsync setting.

New features of pulsar include the local persistence mode (async – bypass Journal). We use this pattern to configure pulsar to match the default level of Kafka local persistence. Table 13 lists the specific settings of the benchmark.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Streamnative framework

We found some problems in the confluent OMB framework branch and fixed some configuration errors in the OMB pulsar driver. We have developed new benchmark code (including the following fixes) and put them in the open source repository.

Fix OMB framework issues

Confluent follows the OMB framework recommendations and uses two sets of examples – one for Kafka and one for pulsar. In our benchmark, we assigned a set of three instances to enhance the reliability of the test. In the first test, we ran three instances on pulsar. Then use the same set of examples to test Kafka.

We benchmark different systems on the same machine and clear the file system page cache before each run to ensure that the current test is not affected by previous tests.

Fix OMB pulsar driver configuration issues

We fixed many errors in confluent’s OMB pulsar driver configuration. The following sections describe the specific adjustments we have made to broker, bookie, producers, consumers and pulsar image.

Adjust broker configuration

Pulsar broker utilization  managedLedgerNewEntriesCheckDelayInMillis  Parameter to determine the length of time (in milliseconds) that catch up subscriptions must wait before distributing messages to their consumers. In the OMB framework, the value of this parameter is set to 10, which is the main reason for the inaccurate conclusion of the fluent benchmark. The conclusion of the fluent benchmark is that the pulsar delay is higher than Kafka. We change this value to 0 to simulate Kafka’s delay behavior on pulsar. After the change, the delay of pulsar is significantly lower than that of Kafka in all test scenarios.

To optimize performance, we will  bookkeeperNumberOfChannelsPerBookie  The parameter value is increased from 16 to 64 to prevent any single netty channel between broker and bookie from becoming a bottleneck. This bottleneck can lead to high latency when a large number of messages accumulate in the netty IO queue.

We will provide clearer guidelines in pulsar documentation to help users optimize end-to-end latency.

Adjust bookie configuration

We added a new configuration for bookie to test pulsar performance when bypassing logging. Pulsar and Kafka are evenly matched for durability assurance.

To test the performance of this feature, we built a custom image based on the official release of pulsar 2.6.1, covering this adjustment. (see pulsar image for details.)

We manually configured the following settings to bypass logging in pulsar.

journalWriteData = false
journalSyncData = false

In addition, we will  journalPageCacheFlushIntervalMSec  The value of the parameter is changed from 1 to 1000, and asynchronous local persistence is benchmarked in pulsar(journalSyncData = false)。 When this value is raised, pulsar can simulate Kafka’s swipe behavior as described below.

Kafka ensures local persistence by flushing the file system page cache to disk. The data is written by a set of background threads called pdflush. Pdflush can be set. The waiting time between two swipes is usually set to 5 seconds. Put pulsar’s  journalPageCacheFlushIntervalMSec  The parameter is set to 1000, which is equivalent to the 5-second pdflush interval on Kafka. After the change, we can benchmark asynchronous local persistence more accurately and compare pulsar and Kafka more accurately.

Adjust producer configuration

Our batch configuration is the same as confluent, with one exception: we have increased the switching interval to make it longer than the batch interval. Specifically, we will  batchingPartitionSwitchFrequencyByPublishDelay  Parameter value changed from 1 to 2. This change ensures that pulsar producers focus on only one partition during each batch.

Setting the switching interval and batch interval to the same value will cause pulsar to switch partitions frequently, resulting in too many small-scale batches, and may affect throughput. This risk can be minimized by setting the switching interval greater than the batch interval.

Adjust consumer configuration

When the application cannot process the incoming message quickly, the pulsar client uses the receiver queue to exert backpressure. The size of the consumer receiver queue affects the end-to-end delay. Larger queues can prefetch and cache more messages than smaller queues.

These two parameters determine the size of the receiver queue:receiverQueueSize  and  maxTotalReceiverQueueSizeAcrossPartitions。 Pulsar calculates the receiver’s queue size as follows:

Math.min(receiverQueueSize, maxTotalReceiverQueueSizeAcrossPartitions / number of partitions)

For example, if  maxTotalReceiverQueueSizeAcrossPartitions  Set to 50000. When there are 100 partitions, pulsar client will set the receiver queue size of consumers to 500 in each partition.

In our benchmark,maxTotalReceiverQueueSizeAcrossPartitions  From 50000 to 5000000. This tuning ensures that consumers do not exert back pressure.

Pulsar image

We built a custom version of pulsar (v.2.6.1-sn-16), including the pulsar and bookkeeper fixes described above. Version 2.6.1-sn-16 is based on the official version of pulsar 2.6.1 and can be downloaded from https://github.com/streamnative/pulsar/releases/download/v2.6.1-sn-16/apache-pulsar-2.6.1-sn-16-bin.tar.gz   Download.

Streamnative test method

We adjusted the testing method of confluent benchmark to fully understand the performance through the actual workload. The following adjustments have been made to the test:

  1. To evaluate the following, catch up read is added
  • The maximum throughput that each system can achieve when processing catch up read
  • How writes affect publishing and end-to-end latency
  1. Change the number of partitions to see how each change affects throughput and latency
  2. Change the number of subscriptions to see how each change affects throughput and latency

Our benchmark scenario tests the following types of workloads:

  • Maximum throughput: what is the maximum throughput of each system?
  • Delay between release and tailing read: the minimum release and end-to-end delay that each system can achieve under a given throughput?
  • Catch-up read: what is the maximum throughput that each system can achieve when reading messages from a large number of to-do items?
  • Mixed workload: what is the minimum release and end-to-end delay level that can be achieved by each system when the consumer performs a catch up operation? How does catch up read affect release latency and end-to-end latency?

Testbed

The OMB framework recommends that specific testbed definitions be used for instance types and JVM configurations; For the producer, consumer, and server side, use the workload driver configuration. Our benchmark uses the same testbed definition as confluent. For these testbed definitions, see the streamnative branch in the confluent OMB repository.

The following highlights the observed disk throughput and disk fsync latency. To interpret the benchmark results, these hardware metrics must be considered.

Disk throughput

Our benchmark uses the same instance type as confluent, specifically i3en.2xlarge (with 8 vcores, 64 GB ram, 2 x 2, 500 GB nvme SSD). We confirm that the i3en.2xlarge instance can support a write throughput of up to 655 MB / s between two disks. See DD results below.

Disk 1
dd if=/dev/zero of=/mnt/data-1/test bs=1M count=65536 oflag=direct
65536+0 records in
65536+0 records out
68719476736 bytes (69 GB) copied, 210.08 s, 327 MB/s
Disk 2
dd if=/dev/zero of=/mnt/data-2/test bs=1M count=65536 oflag=direct
65536+0 records in
65536+0 records out
68719476736 bytes (69 GB) copied, 209.635 s, 328 MB/s

Disk data synchronization delay

When performing delay related tests, it is important to capture fsync delays on nvme SSDs. We observed that the p99 fsync latency of these three instances is between 1 ms and 6 ms, as shown in the figure below. As mentioned earlier, disks vary greatly in different situations, which is mainly reflected in the delay. We found that the delay of a group of instances is consistent.

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

Streamnative benchmark results

Our benchmark results are summarized below. To view the complete benchmark report, you can download it on the streamnative official website or view it in the openmessaging benchmark warehouse.

Maximum throughput test

The purpose of the maximum throughput test is to determine the maximum throughput that each system can achieve when dealing with workloads with different persistence guarantees, including publishing and tailing read. We changed the number of topic partitions to see how each change affects maximum throughput.

We found that:

  1. When the persistence guarantee (synchronous replication persistence, synchronous local persistence) is configured as level 1, the maximum throughput of pulsar is about 300 MB / s, which is the physical limit of log disk bandwidth. With 100 partitions, Kafka can reach about 420 MB / s. It should be noted that when the persistence is level 1, pulsar is configured to use one disk as a log disk for writing and the other disk as a ledger disk for reading; Kafka uses two disks to read and write at the same time. Although the setting of pulsar can provide better I / O isolation, its throughput is also limited by the maximum bandwidth of a single disk (~ 300 MB / s). Configuring spare disks for pulsar enables more cost-effective operation. This topic will be discussed in the follow-up blog.
  2. When persistence (synchronous replication persistence and asynchronous local persistence) is configured as level 2, pulsar and Kafka can achieve a maximum throughput of about 600 MB / s. Both systems have reached the physical limit of disk bandwidth.
  3. The maximum throughput of Kafka on a partition is only half of the maximum throughput of pulsar.
  4. The throughput of pulsar will not be affected by changing the number of partitions, but the throughput of Kafka will be affected.
  • When the number of partitions increases from 100 to 2000, pulsar maintains the maximum throughput (about 300 MB / s under level 1 persistence guarantee and about 600 MB / s under level 2 persistence guarantee).
  • When the number of partitions increases from 100 to 2000, Kafka’s throughput decreases by half.

Release and end-to-end delay testing

Publishing and end-to-end delay testing aims to determine the minimum delay that each system can achieve when dealing with workloads with different persistence guarantees, including publishing and tailing read. We modified the number of subscriptions and partitions to see how each change affects publication and end-to-end latency.

We found that:

  1. In all test cases, pulsar’s publishing and end-to-end latency were significantly (hundreds of times) lower than Kafka, which evaluated various persistence guarantees and different numbers of partitions and subscriptions. Even if the number of partitions increases from 100 to 10000 or the number of subscriptions increases from 1 to 10, the pulsar p99 publication delay and end-to-end delay are within 10 milliseconds.
  2. Changes in the number of subscriptions and partitions have a significant impact on Kafka’s publishing and end-to-end latency.
  • When the number of subscriptions increases from 1 to 10, the publication and end-to-end latency increase from about 5 milliseconds to about 13 seconds.
  • When the number of topic partitions increases from 100 to 10000, the publishing and end-to-end latency increase from about 5 milliseconds to about 200 seconds.

Catch up read test

The catch up read test is designed to determine the maximum throughput that each system can achieve when dealing with a workload containing only catch up read. At the beginning of the test, the producer sends messages at a fixed rate of 200K per second. After the producer sends 512gb of data, the consumer starts reading the received message. The consumer processes the cumulative message, and when the producer continues to send new messages at the same speed, the consumer can keep pace with the producer.

When processing catch up read, the maximum throughput of pulsar is 3.5 times faster than Kafka. The maximum throughput of pulsar is 3.5 GB / S (3.5 million messages / s), while the throughput of Kafka is only 1 GB / S (1 million messages / s).

Mixed workload test

The purpose of the mixed workload test is to determine the impact of catch up read on publishing and tailing read in the mixed workload. At the beginning of the test, the producer sends messages at a fixed rate of 200K per second, while the consumer consumes messages in tailing mode. After the producer generates 512gb messages, it will start a new set of catch up consumers to read all messages from scratch. At the same time, producers and existing tailing read consumers continue to publish and use messages at the same speed.

We tested Kafka and pulsar with different persistence settings and found that catch up read will seriously affect the release delay of Kafka, but has little impact on pulsar. The Kafka p99 release delay increased from 5 milliseconds to 1-3 seconds, while the pulsar p99 release delay remained between a few milliseconds and tens of milliseconds.

conclusion

Benchmarking usually only presents a narrow combination of business logic and configuration options, which may or may not reflect actual application scenarios or best practices, which is the tricky part of benchmarking. Benchmarking may be biased due to its framework, setting and research methods. We found these problems in confluent’s recent benchmark.

At the request of the community, the streamnative team began the benchmark to provide insights and views on the real performance of pulsar. In order to make the benchmark more accurate, we fixed the problems existing in the confluent benchmark and added new test parameters to help us deeply explore the comparison results of various technologies in real use cases.

According to our benchmark test results, under the same persistence guarantee, pulsar outperforms Kafka in workloads similar to real application scenarios; In the same limited test case of confluent application, pulsar can achieve the same end-to-end throughput as Kafka. In addition, in each different test instance (including different number of subscriptions, number of topics and persistence guarantee), pulsar has better latency than Kafka, and I / O isolation is better than Kafka.

As mentioned earlier, no benchmark can replace the tests on their respective hardware according to the real workload. Readers are encouraged to test pulsar and Kafka with their own settings and workloads to understand the performance of each system in a specific production environment. If you have any questions about pulsar best practices, please contact us directly or join pulsar slack at any time.

In the coming months, we will publish a series of blogs to help the community better understand and make use of them   Pulsar   Meet their business needs. We will introduce   Pulsar   Performance in different workloads and settings, describes how to select and resize hardware in different cloud providers and local environments, and how to leverage   Pulsar   Build the most cost-effective streaming data platform.

About the author:

Guo Sijie, co-founder and CEO of streamnative. Guo Sijie has been engaged in the message and stream data industry for more than ten years. He is a senior expert in message and stream data. Before founding streamnative, he co founded streamlio, focusing on real-time solutions. He was the technical director of the twitter messaging infrastructure team and co founded distributedlog and twitter eventbus. During his tenure at Yahoo, he led the team to develop bookkeeper and pulsar. He is vice president of Apache bookkeeper and a member of Apache pulsar PMC.

Peng Hui Li, streamnative software engineer, member of Apache pulsar Committee / PMC. Li Penghui once worked in Zhilian recruitment. During this period, as the main promoter, he landed Apache pulsar in Zhilian recruitment. His work experience has always focused on messaging systems and microservices. At present, he has fully invested in the world of pulsar.

Reference link:

Related reading

Pulsar and Kafka benchmark: accurate analysis of pulsar performance (full version)

clicklink, download the English version