Play Kafka in one minute

Time:2021-1-17

Speaking of Kafka, many users love and hate it. Kafka is a distributed, publish / Subscribe Based messaging system, and its extreme experience is irresistible. However, users still need to pay a lot of back work to worry about the operation and maintenance, complex security policies, lack of reliability and ease of use, not the ultimate performance, and not rich messaging services. Even if you are a veteran of Kafka, you will inevitably have the same troubles mentioned above.

At present, the mainstream cloud service providers in China provide Kafka services on the cloud to provide asynchronous message queuing services for application systems. Through the highly available message buffer queue, application decoupling, burst traffic processing and interworking and integration with the third party are realized. It has the characteristics of large-scale, high reliability, high concurrent access, scalability and full hosting. For example, distributed message queue DMS helps to decouple application components in the cloud, which is cost-effective. Distributed message queuing (DMS) embraces open source and provides compatible open source Kafka interface. Users can migrate seamlessly and use it on demand.

Advantages of distributed message queue DMS

Fully compatible with Kafka to improve efficiency

Compatible with open source
The business system is developed based on open source Kafka. With only a small amount of authentication security configuration, the Kafka queue of DMS can be used to achieve seamless migration.

Message exchange
The distributed message queue DMS interface can send and receive messages to Kafka queue, or open source Kafka client.

Safety assurance
The unique security reinforcement system provides effective security measures such as business operation traceability, message storage encryption and effective isolation between tenants.

High reliability and worry free operation and maintenance

High reliability of data
Message persistence, multi copy storage.

Service highly available
Background multi cluster deployment, support automatic fault migration and fault tolerance, ensure the reliable operation of users’ key business.

Worry free operation and maintenance
Provide a complete set of monitoring alarm and other operation and maintenance services, automatic fault detection and alarm, avoid 7 * 24 hours of manual duty.

Core features

Fine grained flexible control
Distributed message queuing DMS supports RBAC (role-based access control) and PBAC (policy based access control), which provides more secure and flexible access policies. It can realize the security policy access control of message queue granularity and API call action. Combined with the Iam service of Huawei cloud, it can meet almost all the security requirements of users for the use of message service.

Multi protocol access
Distributed message queuing (DMS) provides a variety of protocol access methods, including 1) http rest compliant access, multi language access, in cloud and out of cloud access. 2) The simple SDK mode based on TCP provides a more high-performance access interface; in addition, it also provides an open interface compatible with open source Kafka, which can better help users quickly put Kafka applications into the cloud.

Distributed reliable cluster and massive queue capability
The built-in distributed cluster technology makes the service highly scalable; the unlimited number of queues and scalable high-performance mechanism guarantee the access ability in high concurrency, high performance and large-scale scenarios, and easily realize the accumulation and access ability of 10 billion level messages. The built-in redundant message storage ensures the reliability of message storage and effectively avoids service node failure.

Dead letter management
Dead letter is an optional scheme to provide users with a message that can not be processed normally in the consumption process, so as to prevent the subsequent messages from being consumed due to the abnormality of individual messages, resulting in business blocking. DMS provides users with visual dead letter switch, parameter configuration and API interface. When the dead letter queue function is not turned on, the system will roll back and re post the messages that fail to confirm or time out. No matter whether the dead letter function is enabled or not, the distributed message queuing DMS Service will ensure that messages are not lost.

Custom message function
Distributed message queuing (DMS) not only provides the core function of message queuing, but also provides additional enhanced capabilities to provide users with high-yield, low-cost and cost-effective message services. Message consumption reset allows users to set any consumable time point to control the forward or backward consumption progress and improve the flexibility of consumption; message filtering based on message tag can selectively consume messages with specified tags; message attributes are supported, and different attributes can be set on each message.

Worry free operation and maintenance
Message service is a cloud native service, and the operation and maintenance of the whole message service is transparent to users. Users don’t need to care about the background operation and maintenance, they only need to pay attention to their own message queue indicators, support the monitoring of the number of queued messages, requests, consumed messages, message accumulation, etc., and support the configuration of alarm rules. Users can obtain the operation and load status of the service message queue through SMS, e-mail, etc. for the first time.

Applicable scenarios

Business decoupling
The message service is used for the non core or unimportant parts of the business that depend on other systems at the same time, and there is no need to wait for the processing results of other systems synchronously. For example, after the e-commerce website obtains the user’s order, the information is put into the message queue, and the information of delivery and delivery tasks is read from the queue and then executed.

Final consistency
The state used for both systems eventually remains the same, either both succeed or both fail. For example, it is used for highly reliable data transmission of transaction system to realize the final consistency of cross system transactions and reduce the difficulty and cost of implementation, such as booking tickets and software taxi.

Peak shifting flow control
When the processing capacity of the upstream and downstream systems is different, the message service can be used to dump the communication data between the systems, provide the message stacking buffer capacity, and process the messages when the downstream system has the ability to process the messages, so as to reduce congestion, system crash and other problems, improve the availability of the system and reduce the complexity. Such as registration, rush purchase and appointment during peak hours.

Log synchronization
The application synchronizes the log messages to the message service in a reliable asynchronous way, and then analyzes the log in real time or offline through other components. It can also be used for Key log information collection and application monitoring. Such as the user’s personal information when registering.

One minute Kafka, that’s it~