New features of Kop 2.8.0 (video attached)


Introduction: at tgip-cn live broadcast on April 11, we invited streamnative engineer Xu Yunze, who shared the prospect of new features of Kop 2.8.0. The following is the concise text sorting version of Xu Yunze’s sharing video for your reference.

In the live broadcast of tgip-cn on April 11, Xu Yunze, a software engineer from streamnative, brought you the preview of Kop 2.8.0 features. The following is a concise text sorting version of its sharing video. Please refer to it.

New features of Kop 2.8.0 (video attached)New features of Kop 2.8.0 (video attached)

The content shared today is “prospect of new features of Kop (Kafka on pulsar) 2.8.0”. First of all, I would like to introduce myself briefly: I work for streamnational, a contributor of Apache pulsar and a major maintainer of Kop.

About Kop version number specification

First, let’s talk about the version number of Kop.

Apache pulsar has major release. Because the version management of Kop is chaotic in the early stage, the version number of Kop is basically the same as pulsar from The Kop master branch will update the dependent pulsar version from time to time. In this way, if you want to have some new functions, you can mention a PR in pulsar, and then Kop can rely on this method. Kop 2.8.0 is a version that can be used for production.

Today, I will launch this live broadcast for you from the following four main points:

  • Why Kop
  • Basic implementation of Kop
  • Recent progress of Kop 2.8.0-snapshot version
  • Recent plans and future prospects

Kafka Vs Pulsar

First of all, let’s talk about my views on Kafka and pulsar systems. Aside from some miscellaneous features, the two systems are still very similar, but the biggest difference is their storage model.

Kafka’s broker is both computing and storage. The so-called calculation refers to abstracting the data sent by the client into different topics and partitions. It may also do some processing such as schema. After processing, the message will be written to the storage. After processing, Kafka’s broker will directly write to the file system on this machine. Pulsar is different. It writes to a bookie cluster, and each bookie node is peer-to-peer.

Tiered storage brings many benefits. For example, if you want to increase throughput, you can add a broker; If you want to increase the disk capacity, you can add bookie. Because each node of bookie is peer-to-peer, there is no need to rebalance, and there is no difference between Kafka’s leader and its follower. Of course, this is not the focus of this talk. What I want to express is that the biggest difference in their architecture is that Kafka writes local files and pulsar writes bookie.

On the other hand, although I think there is no absolute advantage or disadvantage between the two, everyone has the freedom to choose. I believe there are many scenarios where pulsar can be used instead of Kafka.

Migrating from Kafka to pulsar

If I see some advantages of pulsar and want to migrate from Kafka to pulsar, what problems will I encounter?

  1. Drive business to replace clients?
  • The business said it was too troublesome to change.
  • Pulsar adaptors? (pulsar has launched an adaptor. Kafka’s code does not need to be changed, but Maven’s dependency can be changed.)
  • It looks good, but I’m not using the Java client.
  • I don’t mind the trouble, but I only know PHP.
  1. What if users directly use Kafka connectors (nearly 100 kinds) to connect to external systems?
  2. What if the user uses the connector of the external system to connect to Kafka?

KoP(Kafka on Pulsar)

Facing the above problems of migrating from Kafka to pulsar, Kop (Kafka on pulsar) project came into being. Kop introduces Kafka protocol processing plug-in into pulsar broker to realize Apache pulsar’s support for native Apache Kafka protocol. With Kop, users can migrate existing Kafka applications and services to pulsar without modifying the code, so as to use the powerful functions of pulsar. For the background of Kop project, you can learnKop related information, I won’t repeat it here.

New features of Kop 2.8.0 (video attached)

As shown in the figure above, protocol handler was introduced from pulsar 2.5.0, which runs on the broker service. The default is that the pulsar protocol handler is actually just a concept. It communicates with the pulsar client. The Kafka protocol handler is dynamically loaded. If it is configured, it is equivalent to loading a layer of plug-ins. It communicates with the Kafka client through this plug-in.

The use of Kop is very simple. You only need to put the NAR package of protocol handler into the protocols subdirectory under the pulsar directory to broker Conf or standalone Add the corresponding configuration to conf, and the service on port 9092 will be started by default, similar to Kafka.

Currently, Kop supports clients:

  • Java >= 1.0
  • C/C++: librdkafka
  • Golang: sarama
  • NodeJS:
  • Other rdkafka based clients

Protocol Handler

Protocol handler is actually an interface. We can implement our own protocol handler. Broker startup process:

Load the protocol handler from the directory, then load the class, and use theacceptMethods andprotocolNameMethod, and then there are three step-by-step steps:

  • initialize()
  • start()
  • newChannelInitializer()

New features of Kop 2.8.0 (video attached)

Step 1: load the configuration of protocol handler. Protocol handler and broker share the same configuration, so service configuration is also used here. Start is the most important step because it passes in the parameter brokerservice.

Brokerservice controls all resources of each broker:

  • Connected producers, subscriptions
  • Topics held and their corresponding managed ledgers
  • Built in admin and client

Implementation of Kop

Topic & Partition

Kafka and pulsar are similar in many places. In Kafka, topicpartition is a string and int; Pulsar is a little more complicated, which is divided into the following parts:

  • Persistent
  • tenant
  • Namespace
  • theme
  • Zone number

There are three configurations in Kop:

  • Default tenant: kafkatenant = public
  • Default namespace: kafkanamespace = Default
  • Prohibit automatic creation of non partitioned topic: allowuutotopiccreationtype = partitioned

Why configure a configuration that prohibits the automatic creation of non partitioned topics? Because Kafka has only the concept of partitioned topic, but not the concept of non partitioned topic. If the pulsar client is used to automatically create a topic, Kafka’s client may not be able to access the topic. Do some simple processing in Kop, and map the default tenant to the namespace independently.

Produce & fetch request

Product request:

  • Find the persistenttopic object (containing managedledger) by topic name.
  • Convert the message format.
  • Write messages to bookie asynchronously.

Fetch request:

  • Find the persistenttopic object by topic name.
  • Find the corresponding managedcursor through offset.
  • Read the entry from the corresponding position of managedcursor.
  • After converting the entry format, the message is returned to the client.

Group Coordinator

New features of Kop 2.8.0 (video attached)

Group coordinator is used to rebalance and determine the mapping relationship between partition and group. Because the group has multiple consumers, the group coordinator decides which partitions consumers will access.

When a consumer joins (subscribes to) a group:

  • A joingroup request will be sent to notify the broker that a new consumer has joined.
  • A syncgroup request is sent for the allocation of the partition.

It will also send the information to the client, and the consumer will send a new request to get some allocated information from the broker. The group Coordinator will write the group related information into a special topic.

New features of Kop 2.8.0 (video attached)

Kop has also made some configurations here. This special topic will exist in a default namespace, and its number of partitions is 8 by default. Kafka group is basically equivalent to pulsar failover subscription. If you want Kafka’s offset to be recognized by the pulsar client, you need to ack the messageid corresponding to the offset. Therefore, there is a component in Kop called offsetacker, which maintains a group of consumers. Each time the group coordinator wants to make an ACK, it will create a consumer corresponding to the partition to make the group ack.

The concept of “namespace bundle” will be mentioned here. The group coordinator determines the mapping relationship between the consumer and the partition.

New features of Kop 2.8.0 (video attached)

In Apache pulsar, each broker owns some bundle ranges (as shown in the example above); Topic will be hashed to one of the bundle ranges by name. The owner broker of this range is the owner broker of topic, and the topic you subscribe to will be connected to the broker. Here, you should pay attention to two problems: one is that the bundle may split (you can also configure it to prohibit splitting), and the other is that the broker may hang up, which may change the mapping relationship between the bundle and the broker. Therefore, in order to prevent the occurrence of these two problems, Kop registers a listener to sense the change of bundle ownership. Once the bundle ownership changes, it notifies the group coordinator to call the processing function for processing.

Kafka Offset

First, introduce the concepts of Kafka offset and pulsar messageid. Kafka offset is a 64 bit integer used to identify the location of message storage. Kafka messages are stored locally, so you can use an integer to represent the sequence number of messages. Pulsar stores messages on bookie, which may be distributed on multiple machines. Therefore, bookie uses ledger ID and entry ID to represent the location of messages. Ledger ID can be understood to correspond to segment in Kafka, and entry ID is approximately equivalent to Kafka offset. The entry in pulsar does not correspond to a single message, but a packaged message. Therefore, batch index is generated. Therefore, the three fields ledger ID, entry ID and batch index are required to mark a pulsar message together.

New features of Kop 2.8.0 (video attached)

Then, Kafka offset cannot be simply mapped to the messageid of pulsar. Such simple processing may cause the loss of pulsar messages. Before Kop 2.8.0, 20 bits, 32 bits and 12 bits were allocated to pulsar ledger ID, entry ID and batch index respectively to form a Kafka offset (as shown in the figure above). This allocation strategy is feasible in most cases and can ensure the order of Kafka offset. However, it is still difficult to put forward an “appropriate” allocation scheme in the face of message ID splitting, There are several problems:

  • For example, if 20 bytes are allocated to the ledger ID, the ledger ID will be exhausted at 2 ^ 20, which is easy to cause the batch index bytes to run out;
  • Only one entry can be read from the cursor, otherwise it may causeMaximum offset delta exceededProblems;
  • Some third-party components (such as spark) rely on the function of continuous offset

In view of the above problems about Kafka offset, streamnative and Tencent engineers jointly put forward an optimization scheme based on broker entry metadataPIP 70: Introduce lightweight broker entry metadata, the new scheme can refer to the diagram on the right of the figure below.

New features of Kop 2.8.0 (video attached)

The left side of the figure above: at present, pulsar message consists of metadata and payload. Payload refers to the specific written data, and metadata refers to metadata, such as publishing timestamp. Pulsar broker will write the message to the client and save the message to bookie.

The right side of the figure above shows the improvement scheme proposed by PIP 70. In the new scheme, the broker will still write messages to the client, but raw message is written to bookie — what is raw message? Brokerentrymetadata is added to the original message. As can be seen from the above figure, the client cannot obtain raw messages, and only the broker can obtain raw messages. As mentioned earlier, the protocol handler can obtain all permissions of the broker, so the protocol handler also obtains raw message. If offset is set in pulsar, Kop can obtain offset.

We have implemented this: there are two fields in the protocol buffer file, mainly the second field. The index corresponds to the offset of Kafka, which is equivalent to implementing Kafka in pulsar. There are two interceptors, managedledgerinterceptor

   private boolean beforeAddEntry(OpAddEntry addOperation) {
        // if no interceptor, just return true to make sure addOperation will be 
        if (managedLedgerInterceptor == null) {
            return true;
        try {
            return true;

And brokerentrymetadatainterceptor.

    public OpAddEntry beforeAddEntry(OpAddEntry op, int numberOfMessages) {
        if (op == null || numberOfMessages <= 0) {
            return op;
brokerEntryMetadataInterceptors, numberOfMessages));
        return op;

Addoperation contains the bytes and number of messages sent from the producer, so that the interceptor can intercept all production messages. andCommands.addBrokerEntryMetadataThe function of is to add a brokerentrymetadata before opaddentry data. The reason for this is that for ease of parsing, you can first read whether the preceding field is magic number. If yes, you can then read brokerentrymetadata. If not, you can parse ordinary metadata according to the normal protocol. The brokerentrymetadatainterceptor is equivalent to an interceptor on the broker side.

Therefore, it is easy to implement continuous offset based on brokerentrymetadata in Kop:

  • Fetch request: directly read bookie and parse brokerentrymetadata;
  • Product request: pass the managedledger into the context of asynchronous bookie writing, and get the offset from the interceptor of the managedledger
  • COMMIT_ Offset request: for__ consumer_ Offsets, write the topic intact. For the cumulative acknowledgement of pulsar, perform a binary search on the managedledger.

In view of the above changes, the following configuration must be made in Kop 2.8.0 to ensure the normal use of offset operation:


Message encoding and decoding

This is also an important part of the improvement of Kop 2.8.0.

Before Kop 2.8.0, the production and consumption of messages by Kop required operations such as message decompression and debatch, with serious operation delay. We also raised a question: why should Kop be compatible with pulsar clients? If you migrate from Kafka to pulsar, there may only be Kafka client in most cases, and there is unlikely to be interaction between Kafka client and pulsar client. Therefore, it is unnecessary to encode and decode messages. When producing messages, write the ByteBuffer inside memoryrecords directly to bookie.

The message consumption is relatively different. We use managedcursor to read, and we also need to convert several entries into a bytebuf. However, in the actual application scenario, it is found that this method is still expensive. After further investigation, it is found thatappendWithOffsetIf the number of batches is large, the number of calculations is too many, resulting in unnecessary overhead. To solve this problem, bigo team members gave relevant PR schemes and submitted a simplified version of appendwithoffset (as shown in the figure below), eliminating unnecessary actions. Of course, the proposal is also based on the continuous offset improvement submitted earlier.

New features of Kop 2.8.0 (video attached)

Performance test (WIP)

The performance test is still in the WIP (work in progress) stage, and some problems have been found. First, in the figure below, the end-to-end delay is 6ms to 4ms, which is within the acceptable range. However, in the follow-up investigation, I found that the full GC is often up to 600 ms, or even the delay is higher. We are investigating this problem.

New features of Kop 2.8.0 (video attached)

The following figures show the monitoring of handleproducerequest, produceencode, messagequeuedlatency and messagepublish. From the monitoring point of view, the delay of handleproducerequest (from the beginning of processing the product request to the successful writing of all messages in the request to bookie) is about 4 ms, which is similar to that of pulsar client, but there is less round-trip to the network.

New features of Kop 2.8.0 (video attached)

We mainly look at the encoding time of produceencode (encoding time of Kafka message). My test uses Kafka’s entryformat, and we can see that it takes less than 0.1 ms; If the entryformat of pulsar is used, the monitoring result is between a few milliseconds and a few milliseconds.

New features of Kop 2.8.0 (video attached)

In fact, there are still some problems in the implementation here, because a queue is still used, so there will be the indicator messagequeuedlatency in the figure below. Messagequeuedlatency is the time from the message queue of each partition to the preparation for asynchronous transmission. We doubt whether the queue leads to poor performance, but from the monitoring point of view, the delay of 0.1 ms has little effect.

New features of Kop 2.8.0 (video attached)

Finally, messagepublish is the delay of bookie, that is, the time from asynchronous sending of messages from a single partition to successful writing to bookie. The monitoring results are ideal, so we will study the source of GC problems in the near future.

New features of Kop 2.8.0 (video attached)

KoP Authentication

Before version 2.8.0

In the actual production environment, if you need to deploy to the cloud, you need to support authentication. Before 2.8.0, Kop’s support for authentication was relatively simple, and its support was limited to SASL / plan mechanism. It is based on pulsar’s JSON web token authentication. In addition to the basic configuration of the broker, it only needs additional configurationsaslAllowedMechanism=Plain。 The client needs to enter namespace and token as the user name and password of JAAS configuration.

security.protocol=SASL_PLAINTEXT    # or security.protocol=SASL_SSL if SSL connection is used
sasl.mechanism=PLAIN \
Required username=''public/default'' password=''token:xxx'';

OAuth 2.0 support

Recently, Kop 2.8.0 supports OAuth 2.0 authentication, that is, SASL / oauthreader mechanism. In brief, it uses a simple third-party service. First, Client obtains authorization from Resource Owner, Resource Owner can be similar to the WeChat official account’s authorization code, or it can be Grant that the real person gave in advance. Then send grant to the authorization server, that is, the server of OAuth 2. You can obtain the access token through the authorization code to access the resource server, that is, pulsar’s broker, which will verify the token. The token is obtained through third-party authentication, which is relatively safe.

The default handler of Kop is the same as Kafka. Similar to Kafka, Kop also needs to configure the server callback handler on the broker side for token verification:

  • Kopoauth2authenticatecallbackhandler: handler class
  • Kopoauth2configfile: configuration file path

The JAAS method is used and a separate configuration file is used. Kop provides an implementation class, which is verified based on the autnationprovider configured by pulsar broker. Because Kop has a broker service, that is, it has all the permissions of the broker and can call the provider authentication method configured by the broker for authentication. Therefore, only auth needs to be configured in the configuration file validate. Method =, which corresponds to the of the providergetAuthNa meMethod returns a value. If JWT authentication is used, this method is token; If OAuth 2 authentication is used, this method may be different.


For Kafka client, Kop provides a login callback handler implementation. Kafka java client OAuth 2.0 authentication:
security.protocol=SASL_PLAINTEXT # or security.protocol=SASL_SSL if SSL connection is used sasl.mechanism=OAUTHBEARER \
    required oauth.issuer.url=""\

The server callback is used to verify the client token, and the login callback handler is used to obtain the third-party token from the third-party OAuth 2 service. My implementation refers to the implementation of pulsar, and the configuration is configured according to Kafka’s JAAS. There are three main configurations: issueurl, credentialsurl, and audience. Its meaning is the same as pulsar’s Java client authentication, so you can refer to pulsar’s documentation. Pulsar java client OAuth 2.0 authentication:

String issuerUrl = ";
String credentialsUrl = "file:///path/to/KeyFile.json";
String audience = "";
PulsarClient client = PulsarClient.builder() 
        AuthenticationFactoryOAuth2.clientcredentials(issuerUrl, credentialsUrl, audience)) .build();

Therefore, Kop supports OAuth 2 by providing callback handlers on the client side and the default server side. When Kafka uses OAuth 2 authentication, it needs to write its own handler; However, the implementation of Kop and pulsar is similar. You do not need to write your own handler, which can be used out of the box.

Kop 2.8.0 other developments

  • The transaction coordinator of Kafka was transplanted. To enable transaction, you need to add the following configuration:
  • Based on prometheusrawmetricsprovider, Kop custom metrics is added. This feature is added by Chen Hang of bigo, that is, the monitoring just shown.
  • Expose advertised listeners to support envoy Kafka filter for proxy. In the previous Kop, the unfriendly point is that the configured listener must be the same as the broker’s advertised listener. In the new version, we separate the listener from the advertised listener, which can support agents. For example, when deployed on the cloud, we can use the invoke agent.
  • Improve support for Kafka adminclient. This is a point that was ignored before. We think that pulsar’s admin is enough. In fact, on the one hand, users are used to using Kafka adminclient. On the other hand, some user configured components have built-in adminclient. If this protocol is not supported, it will affect the use.

Recent plan

Pulsar 2.8.0 is expected to be released at the end of April. Before the official release, you need to troubleshoot some performance test problems:

  1. Add more detailed metrics.
  2. Check the problems of continuous memory growth and full GC during pressure measurement.
  3. Conduct more systematic performance tests.
  4. Deal with recent feedback from the community.

Related reading

New features of Kop 2.8.0 (video attached)

clicklink, get Apache pulsar hard core dry goods information!