Pulsar: is the next generation messaging engine really so powerful?

Time:2021-5-13

Pulsar: is the next generation messaging engine really so powerful?

background

Recently, we are doing the technology selection of new business, which involves the selection of message middleware; Combined with our actual situation, we hope it can meet the following requirements:

  • Friendly cloud native support: because now the main language isGoAt the same time, the operation and maintenance can be simple enough.
  • Official support for multilingualSDKThere are some morePythonJavaThe relevant code needs to be maintained.
  • It’s better to have some convenient and easy-to-use features, such as delay message, dead letter queue, multi tenant, etc.

Of course, there are some features such as horizontal expansion, throughput and low latency. Almost all mature message middleware can meet these requirements.

Based on the above screening conditions,PulsarInto our vision.

AsApacheUnder the top projects, the above features can be well supported.

Now let’s see what’s extraordinary about it.

framework

Pulsar: is the next generation messaging engine really so powerful?

It can be seen from the official architecture diagramPulsarIt mainly consists of the following components:

  1. BrokerStateless component, which can be extended horizontally, is mainly used to connect producers and consumers; Similar to Kafka’s broker, but without data storage function, it is easier to expand.
  2. BookKeeperCluster: mainly used for persistent storage of data.
  3. ZookeeperFor storagebrokerAndBookKeeperMetadata for.

On the whole, it seems that Kafka relies on more components than Kafka, which really provides the complexity of the system; But the same benefits are obvious.

PulsarThe storage of is separated from the calculation. When it needs to be expanded, it will be very simple to add directlybrokerNo other mental burden.

When storage becomes a bottleneck, it only needs to be expandedBookKeeperThere’s no need for artificial balancing,BookKeeperIt will load automatically.

The same operation,KafkaIt’s much more complicated.

characteristic

multi-tenancy

Multi tenant is also a just need function, which can isolate the data of different businesses and teams in the same cluster.

persistent://core/order/create-order

Take the topic name as an examplecoreThere is one under this tenantorderOfnamespaceIn the endcreate-orderOftopicname.

In practice, tenants are generally divided according to the business team,namespaceDifferent businesses under the current team; In this way, you can manage topics clearly.

Usually, only when there is contrast can there be harm. How to deal with this kind of problem in message middleware without multi tenancy

  1. Simply do not divide into such details, mix all business lines, when the team is small, it may not be a big problem; Once the business increases, the management will be very troublesome.
  2. I do a layer of abstraction before topic, but in fact, I am also implementing multi tenant.
  3. Each business team maintains its own cluster, which of course can solve the problem, but the complexity of operation and maintenance is naturally increased.

The above is very intuitive to see the importance of multi tenant.

Function calculation

PulsarIt also supports lightweight function calculation. For example, some messages need to be cleaned, transformed, and then published to another topic.

You can write a simple function,PulsarProvidedSDKCan be convenient for data processing, and finally use the official tool to publish tobrokerIn the middle.

Before that, such simple requirements may also need to deal with the flow processing engine itself.

application

In addition, the upper level applications, such as producers and consumers, are similar to the users.

such asPulsarFour consumption patterns are supported

  • Exclusive: exclusive mode, only one consumer can start and consume data at the same time; adoptSubscriptionNameThe scope of application is small.
  • FailoverFail over mode: on the basis of exclusive mode, multiple users can be started at the same timeconsumerOnce aconsumerAfter hanging up, the rest can be quickly topped, but there is only oneconsumerIt can be consumed; Some scenarios are available.
  • SharedSharing mode: there can be n consumers running at the same time, and the message will be sent according to theround-robinPolling delivery to eachconsumerMedium; When someoneconsumerNo downtimeackThe message will be delivered to other consumers. This kind of consumption pattern can improve the consumption ability, but the information can not be orderly.
  • KeySharedSharing mode: Based on sharing mode; Equivalent to the sametopicThe messages in the same group can only be consumed orderly by the same consumer.

The third shared consumption mode should be the most widely used one, which can be used when there are sequence requirements for messagesKeySharedpattern.

SDK

Pulsar: is the next generation messaging engine really so powerful?

Officially supportedSDKVery rich; I’m also in the officialSDKOn the basis of the encapsulation of an internal useSDK

Because we usedigThis kind of lightweight depends on the note storage, so it’s like this when it’s used

    SetUpPulsar(lookupURL)
    container := dig.New()
    container.Provide(func() ConsumerConfigInstance {
        return NewConsumer(&pulsar.ConsumerOptions{
            Topic:            "persistent://core/order/create-order",
            SubscriptionName: "order-sub",
            Type:             pulsar.Shared,
            Name:             "consumer01",
        }, ConsumerOrder)

    })

    container.Provide(func() ConsumerConfigInstance {
        return NewConsumer(&pulsar.ConsumerOptions{
            Topic:            "persistent://core/order/update-order",
            SubscriptionName: "order-sub",
            Type:             pulsar.Shared,
            Name:             "consumer02",
        }, ConsumerInvoice)

    })

    container.Invoke(StartConsumer)

Two of themcontainer.Provide()Function for injectionconsumerObject.

container.Invoke(StartConsumer)It will take all of them out of the containerconsumerAt the same time, they start to consume.

At this time, with my limitedGoDevelopment experience is also thinking about a problem in theGoDo you need dependency injection in?

Let’s look at how to use it firstDigThe benefits of such libraries are as follows:

  • The object is managed by the container, which is convenient to implement singleton.
  • When the dependency relationship of each object is complex, it can reduce a lot of code to create and obtain objects, and the dependency relationship is clearer.

The same disadvantages are:

  • It’s not so intuitive to follow the code and see at a glance how a dependent object is created.
  • It’s not consistent with the simplicity advocated by go.

For usedSpringOfJavaFor developers, it must be really fragrant. After all, it’s still familiar; But for those who have not been exposed to similar needs at allGopherIt seems that it is not just needed.

At present, all kinds of go dependency injection libraries are emerging one after another in the market, and there are also many products produced by large manufacturers. It can be seen that there is still a great market.

I believe there are manyGopherI’m very disgusted with itJavaSome complex concepts in are introduced intoGoBut I think that dependency injection itself is not limited by languages, and various languages have their own implementations. Spring in Java is not only a dependency injection framework, but also has many complex functions, which makes many developers afraid.

If we only rely on the injection of this subdivision requirement, the implementation is not complicated and will not bring too much complexity. If you take the time to look at the source code, you can quickly grasp it on the basis of understanding the concept.

go back toSDKIn itself,GoOfSDKAt this stage, it’s better than thatJavaThe version has few features (to be exact, onlyJavaVersion of the most abundant features), but the core have, does not affect the daily use.

summary

This paper introducesPulsarSome basic concepts and advantages ofGoSecond, the dependency injection is implemented; If you are doing technology selection like us, you might as well consider itPulsar

We will continue to share pulsar related content in the future, and friends with relevant experience can also leave their own opinions in the comments area.

Pulsar: is the next generation messaging engine really so powerful?