China Mobile Zhang Hao: design and application of AMQP on pulsar

Time:2020-9-26

In this issue of tgip-cn live broadcast, we invited Zhang Hao from China Mobile cloud capability center. He participated a lot in the AMQP on pulsar project and helped promote the development and update of the project. The following content is mainly based on the introduction and details of AOP in this live broadcast.

Relevant background

First of all, a very important question is why do we have to make our own AMQP message queue in Mobile? The answer is obvious.

First, in order to meet the needs of internal components, China Mobile is a heavy user of openstack, and in openstack, rabbitmq is used as RPC communication component by default. There are many problems in the online environment deployment and operation and maintenance using rabbitmq. As our middleware team mainly adopts Java architecture, we can only do some peripheral transformation around rabbitmq.

At the same time, there are many customers in China Mobile’s public cloud that need to use AMQP message queue, but the existing rabbitmq does not meet the cloud access conditions. Therefore, the middleware team of China Mobile began to study AMQP message queue. After comparing qpid, rocketmq and pulsar, we find that pulsar’s computing separated storage architecture is very suitable for the current needs.

At the same time, after investigating pulsar, it is found that streamnnative has open source Kop, which confirms the feasibility of developing AMQP based on pulsar. At the same time, pulsar community has a high degree of activity and community support for the project is very high.

Therefore, we started the journey of developing AOP protocol processing plug-ins with streamnnative, and worked together to transform AOP from 0 to 1.

AMQP 0.9.1

AMQP 0.9.1 (Advanced message queuing protocol) is a messaging protocol that allows compliant client applications to communicate with standard compliant messaging middleware agents.

In the initial version of AOP, version 0.9.1 was first implemented. Here are the following concepts to be understood:

  • Virtualhost: logical grouping and isolation of resources
  • Exchange: message routing
  • Queue: message store
  • Binding: routing rules

China Mobile Zhang Hao: design and application of AMQP on pulsar

The general operation process is shown in the figure above: first, messages will be sent to exchange, and then they will be routed to different queues according to different settings / types. When consumers consume information, they are directly from the queue.

Protocol Handler

After understanding the model of AMQP 0.9.1, let’s take a look at the component on which AOP implementation depends – protocol handler.

In pulsar architecture, a set of distributed process encapsulation based on managed Leger has been implemented, including how to store messages and prevent message loss. Broker itself also implements some features, such as load balancer, geo replicator, etc.

China Mobile Zhang Hao: design and application of AMQP on pulsar

The protocol handler in the upper layer is a lightweight tool, which mainly processes TCP requests sent by producers and consumers in pulsar, and converts them into readable operations.

After pulsar version 2.5, the protocol handler interface is separated from each other. Using this framework, you can realize the conversion of custom protocols, such as Kafka, AMQP, mqtt, etc. With the implementation of protocol handler of different protocols, pulsar broker has the ability to read / write / parse other protocols. The following figure shows the architecture model adopted by AMQP on pulsar.

China Mobile Zhang Hao: design and application of AMQP on pulsar

How to implement AOP

There are four parts in the implementation: model transformation (connecting AMQP model into pulsar), message sending, message consumption and proxy.

Model transformation

AMQP 0.9.1 introduces some basic concepts, such as exchagne, queue, etc. These are quite different from pulsar’s model. So we need to find a way to support the use of some of pulsar’s existing features and link them together. The following figure shows the flow of messages in AOP, and discusses the details of message persistence, message routing, and message delivery.

China Mobile Zhang Hao: design and application of AMQP on pulsar

🧰 AmqpExchange

Amqpexchange contains a raw message topic to hold messages sent by AMQP producer. The replicator of amqpexchange processes the message into the AMQP queue. Replicator is a pulsar based persistent cursor that ensures that messages are successfully sent to the queue without losing them.

China Mobile Zhang Hao: design and application of AMQP on pulsar

🧰 AmqpMessageRouter

Amqpmessage router is used to maintain message routing types and routing rules for routing messages from amqpexchange to amqpqueue. The original data of route type and routing rules are persisted in pulsar storage. So even if the broker is restarted, we can restore the amqpmessage router.

🧰 AmqpQueue

Amqpqueue provides an index message topic to store the indexmessage that is routed to this queue. Indexmessage consists of the ID of the original message and the name of the exchange where the message is stored. When an amqpqueue sends a message to a consumer, the amqpqueue reads the original message data based on the indexmessage and sends it to the consumer.

China Mobile Zhang Hao: design and application of AMQP on pulsar

Vhost allocation

In AOP, an AMQP Vhost can only be served by one pulsar broker, while a pulsar broker can serve multiple vhosts. Therefore, the number of vbrokers can be increased. By using more vhosts, users can build larger AOP clusters.
China Mobile Zhang Hao: design and application of AMQP on pulsar

In AOP, a Vhost is based on a pulsar namespace, and this namespace can only have one bundle. If a broker crashes, other brokers can take over the Vhost maintained by the broken broker. Vhost can also take advantage of broker’s load balancing mechanism. Broker can transfer Vhost from a high load machine to an idle machine. The following figure shows the allocation of Vhost on the broker.

China Mobile Zhang Hao: design and application of AMQP on pulsar

message sending

At the message sending level, for example, AMQP producer can do basic publish messages and send them to the broker. When the broker receives the message, it will convert the AMQP message body into the pulsar message body, find the topic corresponding to the current exchange, and write the message. According to different routing relationships and parameter configurations, messages in the topic will be distributed to different queues.

China Mobile Zhang Hao: design and application of AMQP on pulsar

News consumption

In the consumer side, we mainly use the exclusive (exclusive subscription) and shared (shared subscription) types in the pulsar subscription model.

Shared is the ability to attach the required number of consumers to the same subscription. The message is delivered in the form of a circular attempt distribution of multiple consumers, and any given message is delivered to only one consumer. When a consumer disconnects, all messages delivered to it that are not acknowledged are rescheduled for delivery to the remaining consumers at the subscription.
China Mobile Zhang Hao: design and application of AMQP on pulsar

According to the definition of AMQP protocol, messages received by the queue are distributed to all consumers in a round robin manner, which is consistent with the pulsar shared subscription mode. Therefore, the shared subscription mode is defined as the default subscription mode in the implementation of AOP.

The above is applicable to the case that the queue is non exclusive. If the queue is exclusive, the exclusive subscription model is adopted.

No matter how many consumers exist at the same time, only one consumer will be active, that is, only one consumer can receive all messages of this topic. This mode is exclusive subscription in pulsar subscription mode, which is consistent with the exclusive queue in AMQP.

China Mobile Zhang Hao: design and application of AMQP on pulsar

Pulsar’s consumption model is opposite to rocketmq model, one is similar to “push” model and the other is similar to “pull” model. In pulsar, the consumer sends a flow request to the broker, and informs the broker that the message can be pushed to the consumer. At this time, the broker will push the message in the broker to the consumer after checking the permission.

There are two consumption models in AMQP, one is consumption and the other is get. Consumption corresponds to the push model, and get corresponds to the pull model.

China Mobile Zhang Hao: design and application of AMQP on pulsar

In the push model, we define a consumer named “amqppushconsumer”, which inherits the consumer in pulsar broker. The biggest difference is that it is modifiedsendMessage()method. After the pulsar message is converted into an AMQP message, it is directly pushed to the AMQP consumer.

At the same time, add amqppushconsumer to the subscription mode, and callconsumer.handleFlowCommand.

China Mobile Zhang Hao: design and application of AMQP on pulsar

In the pull model, in order to ensure the consistency of consumption sites and avoid message loss, an “amqppullconsumer” is added. It also inherits pulsar broker consumer, but modifies the following two methods:

  • getAvailablePermits()-> 0
  • isBlocked()-> true

This model reads messages from the cursor and shares the read position with the consumer in the push model. The message signing method is also consistent with the push model.

Proxy

AOP proxy is used to help find the owner broker responsible for processing Vhost data when the client connects with AMQP service, and transfer data between the client and the owner broker of Vhost. The following figure illustrates the service flow of AOP proxy.

China Mobile Zhang Hao: design and application of AMQP on pulsar

  1. The AMQP client establishes a connection with AOP proxy.
  2. AOP proxy sends a lookup request to the pulsar cluster to determine the URL address of the owner broker of Vhost.
  3. The pulsar cluster returns the URL address of the owner broker to AOP proxy.
  4. AOP proxy establishes a connection with the owner broker of vhose and starts to transfer data between AMQP client and the owner broker of Vhost.

Currently, AOP proxy works with pulsar broker. Users can choose whether to open AOP proxy service by configuring amqpproxy enable.

How to use AOP

At present, AOP project is an open source project in streamnnative project library, using Apache license V2 license. The project address is as follows: https://github.com/streamnati… 。

Welcome to participate in the contribution and use, the specific steps can refer to the text introduction of readme.

Follow up plan

  • At present, the protocol support of AMQP version 0.9.1 has been basically implemented. In the future, AMQP 1.0 protocol will be supported
  • AMQP metadata is stored in an integrated way to facilitate centralized management of information
  • Enrich AMQP related monitoring items
  • In the future, we will support the multi bundle mechanism of the namespace, which is still a single bundle mechanism

summary

This sharing mainly focuses on the production process and implementation details of AMQP on pulsar project. We hope that through this sharing, we can have a clearer understanding of AOP project, and can also participate in the project contribution if you are interested!

Recommended Today

Comparison and analysis of Py = > redis and python operation redis syntax

preface R: For redis cli P: Redis for Python get ready pip install redis pool = redis.ConnectionPool(host=’39.107.86.223′, port=6379, db=1) redis = redis.Redis(connection_pool=pool) Redis. All commands I have omitted all the following commands. If there are conflicts with Python built-in functions, I will add redis Global command Dbsize (number of returned keys) R: dbsize P: print(redis.dbsize()) […]