Community knowledge base FAQ collection No. 4: Message retention and delay, broker, pulsar permissions and other related issues

Time:2022-3-14

At ordinary times, in the pulsar communication group, we find that we will encounter similar problems repeatedly in the process of contacting and using pulsar. In order to solve these “high-frequency questions” more efficiently and thank the friends who raised high-quality questions, we have specially established a FAQ knowledge base to collect and answer your questions.

We will regularly collect and screen the high-frequency questions raised by the community, and select the high-quality questions to answer by the community experts. After integration and optimization, we will share them with the small partners in the community as a priority reference when encountering problems, hoping to help you solve the problems in the process of using pulsar.

Let’s take a look at the questions collected this time:

Keep message

Question 1:In the message retention mechanism of pulsar, the message of pulsar will enter the retention policy after being acked by a consumer. If multiple consumers subscribe to a topic and some consumer acks of a message, will the message enter the retention policy?

Answer: only when messages are acked in all subscriptions will they be further processed by the retention policy. For the same subscription, it doesn’t matter which consumer makes the ACK. As long as it is ack once, the message in this subscription is the state that it has been ack. For details, please refer to:https://pulsar.apache.org/doc…

retry count

Question 2:Does pulsar support parameter configuration retry times?

Answer: not supported. Pulsar implements the backoff retry strategy according to the time, which is similar to the retry number mechanism. The internal retry strategy of the client uses the backoff mechanism, which can be configured for the client builderbackoffParameters to control.

 /**
     * Set the duration of time for a backoff interval.
     *
     * @param duration the duration of the interval
     * @param unit the time unit in which the duration is defined
     * @return the client builder instance
     */
    ClientBuilder startingBackoffInterval(long duration, TimeUnit unit);

    /**
     * Set the maximum duration of time for a backoff interval.
     *
     * @param duration the duration of the interval
     * @param unit the time unit in which the duration is defined
     * @return the client builder instance
     */

From the producer’s point of view, the total send timeout can be configured in producer builder.

    /**
     * Set the send timeout <i>(default: 30 seconds)</i>.
     *
     * <p>If a message is not acknowledged by the server before the sendTimeout expires, an error will be reported.
     *
     * <p>Setting the timeout to zero, for example {@code setTimeout(0, TimeUnit.SECONDS)} will set the timeout
     * to infinity, which can be useful when using Pulsar's message deduplication feature, since the client
     * library will retry forever to publish a message. No errors will be propagated back to the application.
     *
     * @param sendTimeout
     *            the send timeout
     * @param unit
     *            the time unit of the {@code sendTimeout}
     * @return the producer builder instance
     */
    ProducerBuilder<T> sendTimeout(int sendTimeout, TimeUnit unit)

Pulsar Perf

Question 3:Pulsar perf is used to produce data, but there is no backlog.

Answer: create a subscription before pulsar perf production data.

Pressure measuring tool

Question 4:How to measure the pulsar pressure? What tools are used?

Answer: pulsar pressure measurement tools currently include pulsar perf and openmessaging benchmark. For details, please refer to the following links:

Message delay

Question 5:Does pulsar server support delayed delivery by default?

Answer: delay message only works in shared subscription mode. If it does not take effect, first check whether the subscription mode is shared. Pulsar itself defaults to exclusive subscription mode; At the same time, it should be avoided that delay message takes effect in keyshared mode, and delay message and key_ Shared semantics are violated.

Broker downtime

Question 6:If the broker goes down, how will the message be notified to the next broker?

Answer: the broker itself is stateless.

  • When a broker goes down, it will not actively notify other brokers. All bundles distributed on this broker (bundles are the topic set and the smallest unit of pulsar topic for load balancing) will perform unload operation. The process of unload operation is as follows:
  1. Close the producers, consumers and replicators of all topics in unload;
  2. Close the managedledger. When all bundles are unloaded, the broker can exit normally.
  • If the broker exits abnormally, the corresponding ownership of all bundles on the broker will also be forcibly removed, that is, the bundle does not belong to the broker.

When the producer or consumer client needs to continue sending / receiving messages to a topic, it will first execute the lookup request, find the target broker node (currently the node with the lowest load) according to the loadbalance policy, and load the corresponding bundle onload to the target broker. After onload is completed, the broker can continue to provide read-write services for the topic.

In addition, some temporary node information of broker on zookeeper will be deleted actively; Or disconnect due to timeout. After other brokers listen, corresponding operations will be carried out.

Current limiting

Question 7:The disk occupation is very serious, and the speed of disk file cleaning is far from keeping up with the speed of writing. The bookkeeper parameter is adjustedgcWaitTime、majorCompactionInterval、minorCompactionIntervalThe effect is not obvious. Are there other solutions?

Answer: the comparison has a current limiting strategy. From your description, it should be that the comparison is slow and can be adjustedisThrottleByBytes=true, and increasecompactionRateByBytesCurrent limiting threshold of.

# Throttle compaction by bytes or by entries.
isThrottleByBytes=false

# Set the rate at which compaction will readd entries. The unit is adds per second.
compactionRateByEntries=1000

# Set the rate at which compaction will readd entries. The unit is bytes added per second.
compactionRateByBytes=1000000

Pulsar permissions

Question 8:Pulsar authority related information.

answer:

The above is the summary of the 4th Community FAQ. I would like to thank the partners who participate in the daily questions and answers of the community. Let’s look forward to the next FAQ!

Relevant recommendations