Apache pulsar ecological project Kop new maintainer: Wu Zhanpeng

Time:2021-9-16

About Apache pulsar

Apache pulsar is a top-level project of the Apache Software Foundation. It is a native distributed message flow platform for the next generation cloud. It integrates message, storage and lightweight functional computing. It adopts a separate architecture design of computing and storage, and supports multi tenant, persistent storage and multi machine room cross regional data replication with strong consistency Highly scalable streaming data storage with high throughput and low latency.
After Tencent maintainer joined, Kop (Kafka on pulsar), the top project of Apache Software Foundation and the distributed message flow platform Apache pulsar ecological project, added a community maintainer, who is Wu Zhanpeng, a senior engineer from bigo. Bigo will migrate the message system from Kafka to pulsar in 2020. It has rich experience in the application of Kop and has made many contributions to the community in the direction of pulsar Flink. The maintainer from bigo joins the Kop project on behalf of bigo’s recognition and further investment in the pulsar project.

What is Kop

“Kop” (Kafka on pulsar) is jointly open source by streamnational and ovhcloud, which mainly meets the strong needs of users who want to switch from Kafka application to pulsar.

Kop introduces Kafka protocol processing plug-in into pulsar broker to realize Apache pulsar’s support for native Apache Kafka protocol. After adding the Kop protocol processing plug-in to the existing pulsar cluster, users can migrate existing Kafka applications and services to pulsar without modifying the code, so as to use the powerful functions of pulsar, such as:

• simplify operations with enterprise class multi tenant features;
• avoid data relocation and simplify operation;
• persistent retention of event streams using Apache bookkeeper and tiered storage;
• use pulsar functions for serverless event processing.
Addition of new maintainer

So far, Wu Zhanpeng has contributed 6 commit and 7 PR to the Kop project; Contributed 5 commit and 6 pr to pulsar project. His contribution optimizes the performance of the Kop project:

• optimize the performance of Kop consumer side, increase the consumption throughput by 3 times, and reduce the decoding time to 1 / 5 of the original;
• optimize the memory on the consumption side of Kop to prevent entry data from entering the broker heap memory and reduce the impact of GC;
Contributor message

I’m glad to be Kafka on pulse maintainer! As a user and contributor of Kop, becoming a maintainer represents the community’s recognition of me. In the process of using and contributing, I met many partners in pulsar community. I look forward to more and more people joining the Kop project, using and contributing, and meeting like-minded engineers like me. In the future, I will devote more energy to the Kop project, contribute code and ideas, and lead more people to get to know and join Kop!

As the maintainer of Kop project, Wu Zhanpeng is expected to bring new strength to the Kop maintainer team, and we are expected to continue to promote the improvement of the functions of Kop project and better serve the community.

At present, Apache pulsar community has a large number of technical requirements for Kop project. Streamnative will regularly release Kop version updates together with community contributors to continuously meet the needs of the community. Warmly welcome the community to participate in the project, communicate and grow together. It is hoped that more and more senior contributor “players” will unlock the maintainer role in the future.

Try Kop

• Kop uses Apache license V2, and the project address is:https://github.com/streamnati…
• if the pulsar cluster is already running and you want it to support the Kafka protocol, you can install the Kop protocol processing plug-in into the existing pulsar cluster. Please refer to the description [1] for details.
Welcome friends who want to switch from other Kafka applications to pulsar to use Kop. The community has also opened AOP (AMQP on pulsar) [2], mop (mqtt on pulsar) [3], ROP (rocket MQ on pulsar) [4] to help users move rabbitmq / mqtt / rocketmq applications and services to pulsar.

We look forward to your issue and PR. If you encounter problems during use, you can add BOT friends, note “join the group” and enter the pulsar communication group to discuss with us.

As a global open source project, as of August 4, 2021, Apache pulsar has 430 contributors, 9.4k + star and 2.3k + fork. It is the contribution of community developers distributed all over the world that makes the Apache pulsar project develop and move forward.

If you also want to contribute to the Apache pulsar project and march into maintainer, committer and even PMC members, please refer to the following link to join us:

Apache pulsar official contribution guide [5]

Related reading

• Apache pulsar ecological project rocketmq on pulsar adds 3 Tencent maintainers
• Apache pulsar ecological project AOP adds two China Mobile maintainers!
• Tencent joining: Kafka on pulsar project welcomes two Tencent maintainers!
Reference link

[1] Description:https://github.com/streamnati…
[2] AoP(AMQP-on-Pulsar): https://github.com/streamnati…
[3] MoP(MQTT-on-Pulsar): https://github.com/streamnati…
[4] RoP(Rocket-MQ-on-Pulsar): https://github.com/streamnati…
[5] Apache pulsar official contribution Guide:http://pulsar.apache.org/en/c…

ApachePulsar

The top-level project of Apache Software Foundation, the next generation cloud native distributed message flow platform, integrates message, storage and lightweight functional computing, adopts the design of separation of computing and storage architecture, supports multi tenant, persistent storage and multi machine room cross regional data replication, and has the characteristics of stream data storage such as strong consistency, high throughput, low delay and high scalability.