Known as a new generation of message oriented middleware! See how awesome it is

Time:2021-9-23

Recently, the Apache pulsar message middleware is very popular, known as the next generation message middleware. Today, let’s see how awesome it is?

summary

Apache pulsar is a pub / sub messaging platform that uses Apache bookkeeper to provide persistence. It is a server-to-server messaging middleware. It was originally developed by Yahoo and opened source in 2016. It is currently incubated under the Apache foundation. It can provide the following features:

  • Cross region replication
  • multi-tenancy
  • Zero data loss
  • Zero rebalancing time
  • Unified queue and flow model
  • high scalability
  • High throughput
  • Pulsar Proxy
  • function

framework

Known as a new generation of message oriented middleware! See how awesome it is

Pulsar uses a hierarchical structure to isolate the storage mechanism from the broker. This architecture provides pulsar with the following benefits:

  • Independent extension broker
  • Independent extended storage (bookies)
  • Easier to container zookeeper, broker and bookies
  • Zookeeper provides cluster configuration and status storage

In the pulsar cluster, one or more agents process and load balance incoming messages from producers, dispatch messages to consumers, communicate with pulsar configuration storage to handle various coordination tasks, store messages in bookkeeper instances (also known as books), rely on cluster specific zookeeper cluster tasks, and so on.

  • A bookkeeper cluster consisting of one or more bookies handles the persistent storage of messages.
  • The zookeeper cluster specific to this cluster handles coordination tasks between pulsar clusters.

Known as a new generation of message oriented middleware! See how awesome it is

For more information about pulsar architecture, see:https://pulsar.apache.org/doc…

Four subscription modes

There are four subscription modes in pulsar: exclusive, shared, failover and key\_ shared。 These modes are shown in the figure below.

Known as a new generation of message oriented middleware! See how awesome it is

Known as a new generation of message oriented middleware! See how awesome it is

Known as a new generation of message oriented middleware! See how awesome it is

Known as a new generation of message oriented middleware! See how awesome it is

For details, see:https://pulsar.apache.org/doc…

Better performance than Kafka

The best performance of pulsar is its performance. The speed of pulsar is much faster than Kafka. Compared with Kafka, the speed of pulsar is increased by 2.5 times and the delay is reduced by 40%.

Data source:https://streaml.io/pdf/Gigaom…

Known as a new generation of message oriented middleware! See how awesome it is

Known as a new generation of message oriented middleware! See how awesome it is

Note: the comparison is for one subject in one partition, which contains 100 byte messages. Pulsar can send 220000 + messages per second.

install

Binary version installation pulsar
#Download official binary package
[[email protected] ~]# wget https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz
#Decompress
[[email protected] ~]# tar zxf apache-pulsar-2.8.0-bin.tar.gz
[[email protected] ~]# cd apache-pulsar-2.8.0
[[email protected] apache-pulsar-2.8.0]# ll
total 72
drwxr-xr-x 3 root root   225 Jan 22  2020 bin
drwxr-xr-x 5 root root  4096 Jan 22  2020 conf
drwxr-xr-x 3 root root   132 Jul  6 11:47 examples
drwxr-xr-x 4 root root    66 Jul  6 11:47 instances
drwxr-xr-x 3 root root 16384 Jul  6 11:47 lib
-rw-r--r-- 1 root root 31639 Jan 22  2020 LICENSE
drwxr-xr-x 2 root root  4096 Jan 22  2020 licenses
-rw-r--r-- 1 root root  6612 Jan 22  2020 NOTICE
-rw-r--r-- 1 root root  1269 Jan 22  2020 README
#There is a direct start command in the bin directory
Docker installation (emphasis)
[[email protected] ~]# docker run -it \
 -p 6650:6650 \
 -p 8080:8080 \
 --mount source=pulsardata,target=/pulsar/data \
 --mount source=pulsarconf,target=/pulsar/conf \
 apachepulsar/pulsar:2.8.0 \
 bin/pulsar standalone

Port 8080 is used for HTTP protocol access, and port 6650 is used for pulsar protocol (Java, Python and other clients).

Pulsar manager, an official visualization tool, can visually manage multiple pulsars.https://pulsar.apache.org/doc…

[[email protected] ~]# docker pull apachepulsar/pulsar-manager:v0.2.0

[[email protected] ~]# docker run -it \
   -p 9527:9527 -p 7750:7750 \
   -e SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties \
   apachepulsar/pulsar-manager:v0.2.0

Set administrator user and password

[[email protected] ~]# CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
  -H 'X-XSRF-TOKEN: $CSRF_TOKEN' \
  -H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \
  -H "Content-Type: application/json" \
  -X PUT http://localhost:7750/pulsar-manager/users/superuser \
  -d '{"name": "admin", "password": "admin123", "description": "test", "email": "[email protected]"}'

{"message":"Add super user success, please login"}

Browser direct input http://server_ IP: 9527 login as follows

Known as a new generation of message oriented middleware! See how awesome it is

Enter the user and password just created and configure the server of management

Known as a new generation of message oriented middleware! See how awesome it is

list

Known as a new generation of message oriented middleware! See how awesome it is

Topic list

Known as a new generation of message oriented middleware! See how awesome it is

Topic details

Known as a new generation of message oriented middleware! See how awesome it is

Client configuration

Java client

The following is an example of a Java consumer configuration using shared Subscriptions:

import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.SubscriptionType;

String SERVICE_URL = "pulsar://localhost:6650";
String TOPIC = "persistent://public/default/mq-topic-1";
String subscription = "sub-1";

PulsarClient client = PulsarClient.builder()
        .serviceUrl(SERVICE_URL)
        .build();

Consumer consumer = client.newConsumer()
        .topic(TOPIC)
        .subscriptionName(subscription)
        .subscriptionType(SubscriptionType.Shared)
        // If you'd like to restrict the receiver queue size
        .receiverQueueSize(10)
        .subscribe();
Python client

The following is an example of a python consumer configuration using shared Subscriptions:

from pulsar import Client, ConsumerType

SERVICE_URL = "pulsar://localhost:6650"
TOPIC = "persistent://public/default/mq-topic-1"
SUBSCRIPTION = "sub-1"

client = Client(SERVICE_URL)
consumer = client.subscribe(
    TOPIC,
    SUBSCRIPTION,
    # If you'd like to restrict the receiver queue size
    receiver_queue_size=10,
    consumer_type=ConsumerType.Shared)
C + + client

The following is an example of a C + + consumer configuration using shared Subscriptions:

#include <pulsar/Client.h>

std::string serviceUrl = "pulsar://localhost:6650";
std::string topic = "persistent://public/defaultmq-topic-1";
std::string subscription = "sub-1";

Client client(serviceUrl);

ConsumerConfiguration consumerConfig;
consumerConfig.setConsumerType(ConsumerType.ConsumerShared);
// If you'd like to restrict the receiver queue size
consumerConfig.setReceiverQueueSize(10);

Consumer consumer;

Result result = client.subscribe(topic, subscription, consumerConfig, consumer);

More configuration and operation instructions are clearly written in the official documents. The official documents:https://pulsar.apache.org/docs/

summary

As the next generation distributed message queue, plusar has many attractive features and makes up for the shortcomings of some other competitors, such as regional replication, multi tenancy, scalability, read-write isolation and so on.

Known as a new generation of message oriented middleware! See how awesome it is