Blog recommended | how to realize isolation in pulsar through in-depth analysis

Time:2021-11-29

The authors of this paper are Li Penghui and Liu Yu, engineers of streamnative.

About Apache pulsar
Apache pulsar is a top-level project of the Apache Software Foundation. It is a native distributed message flow platform for the next generation cloud. It integrates message, storage and lightweight functional computing. It adopts a separate architecture design of computing and storage, supports multi tenant, persistent storage, multi machine room cross regional data replication, and has strong consistency, high throughput Stream data storage features such as low latency and high scalability.
GitHub address:http://github.com/apache/pulsar/

One of the great advantages of Apache pulsar is that pulsar’s multi-layer partition architecture and hierarchical resource management provide a solid foundation for isolation. Users can isolate resources in the required way, avoid resource competition and ensure system stability.

This blog series will discuss in depthPulsar isolation。 This is the first article in this series. It mainly introduces how to realize isolation in pulsar in the following ways:

  • Individual pulsar clusters
  • Shared bookkeeper cluster
  • Single pulsar cluster

Individual pulsar clusters

In this environment, you need to create their own pulsar clusters for the isolation units.

working principle

Figure 1 shows how isolation can be achieved by deploying a separate pulsar cluster.

Blog recommended | how to realize isolation in pulsar through in-depth analysis

Key points of working principle:

Migrate namespace

To migrate namespaces between different clusters, you need to enable the cross region replication function, and turn off the cross region replication function after all the data is replicated to the target cluster. For information on how to set up cross regional replication for namespaces, seeSet up cross region replication for namespace

Expansion and contraction node

The operations of capacity expansion broker or bookie need to be carried out in the corresponding cluster.

Shared bookkeeper cluster

In this way, you need to deploy a bookkeeper cluster shared by multiple broker clusters.

working principle

Figure 2 shows how isolation can be achieved by deploying a shared bookkeeper cluster.

Blog recommended | how to realize isolation in pulsar through in-depth analysis

Key points of working principle:

  • eachPulsar clusterAccess the service through DNS entry point and ensure that clients can access the cluster through DNS entry point. Clients can use the connection addresses of one or more pulsar clusters.
  • Each pulsar cluster has one or morebroker
  • Each pulsar cluster has a metadata store.
  • Multiple pulsar clusters share a bookkeeper cluster.
  • Pulsar’s layered resource management provides a solid foundation for isolation. To isolate namespaces, you need toSpecify a cluster for the namespace。 Cluster must belong toList of clusters allowed by tenants。 Topics belonging to this namespace will also be assigned to this cluster. For information on how to set up clusters for namespaces, seeSet cluster for namespace。 For information on how to manage pulsar clusters, seeManaging pulsar clusters
  • Storage isolation is realized through different bookie affinity groups, as shown in Figure 3.

    • All bookie isolation groups share bookkeeper clusters and metadata stores.
    • Each bookie isolation group has one or more bookies.
    • Users can specify one or more namespacesPrimary group or secondary group。 First create the topic in the namespace on the bookie of the primary group, and then create the topic in the namespace on the bookie of the secondary group. For information on how to set up a bookie affinity group, seeSet bookie affinity group

Blog recommended | how to realize isolation in pulsar through in-depth analysis

Migrate namespace

To migrate the message service on the namespace to another broker cluster, you need toChange cluster for namespace。 To migrate the namespace to another bookie affinity group, you need to change the bookie affinity group. For information on how to set up a bookie affinity group, seeSet bookie affinity group。 Since all broker clusters share the same bookkeeper cluster, there is no need to copy data to the new bookkeeper cluster.

Expansion and contraction node

Broker

When expanding or shrinking a broker, you should pay attention to the following points:

  • When expanding a broker, you need to usePrimary group or secondary groupSpecify the broker isolation group for the new broker.
  • When shrinking brokers, ensure that there are enough brokers in the broker isolation group.

Bookie

The following points should be paid attention to when expanding and shrinking the capacity:

  • When expanding a bookie, you need to specify a bookie affinity group for the new bookie.
  • When reducing the size of bookie, ensure that the bookie affinity group has a sufficient number of bookies. For information on how to set up a bookie affinity group, seeSet bookie affinity group

Single pulsar cluster

To achieve isolation in this way, you only need to manage a single pulsar cluster without deploying multiple broker clusters and multiple bookie clusters.

working principle

Figure 4 shows how isolation can be achieved by deploying a single pulsar cluster.

Blog recommended | how to realize isolation in pulsar through in-depth analysis

Key points of working principle:

  • eachPulsar clusterAccess the service through DNS entry point and ensure that clients can access the cluster through DNS entry point. The client can use the connection address of the pulsar cluster.
  • Implement broker isolation through different broker isolation groups (pulsar assigns topic to the broker in a specific broker isolation). For how to set a broker isolation group, seeSet broker isolation group
  • Storage isolation is realized through different bookie affinity groups. For information on how to set up a bookie affinity group, seeSet bookie affinity group

Migrate namespace

To migrate a namespace to another broker isolation group, you need to change the namespace isolation policy. For how to set the namespace isolation policy, seeSet namespace isolation policy

To migrate the namespace to another bookie affinity group (without moving the original data to the target bookie affinity group), you need to change the bookie affinity group. For information on how to set up a bookie affinity group, seeSet bookie affinity group

Expansion and contraction mode

Broker

When expanding or shrinking a broker, you should pay attention to the following points:

  • When expanding a broker, you need to usePrimary group or secondary groupSpecify the broker isolation group for the new broker.
  • When shrinking brokers, ensure that there are enough brokers in the broker isolation group.

Bookie

The following points should be paid attention to when expanding and shrinking the capacity:

  • When expanding a bookie, you need to specify a bookie affinity group for the new bookie.
  • When reducing the size of bookie, ensure that the bookie affinity group has a sufficient number of bookies. For information on how to set up a bookie affinity group, seeSet bookie affinity group

Reference

In the production environment, users can use the three pulsar isolation methods mentioned in this article at the same time according to their needs, or choose other methods. Generally speaking, the following precautions can be referred to when selecting the isolation method: some key services (such as billing, advertising, etc.) use multiple small pulsar clusters and do not share storage or local zookeeper with other clusters. Using multiple small pulsar clusters provides the highest level of isolation for critical workloads. Large enterprises (with multiple teams) can deploy a large pulsar cluster, use different namespaces for different isolation groups, and determine the isolation group according to capacity or workload. For example, a scenario with a large number of fan outs does not use the same hardware as the scenario with the lowest end-to-end delay.

Related reading

Blog recommended | how to realize isolation in pulsar through in-depth analysis

clicklink, get Apache pulsar hard core dry goods information!