Background of Message Oriented Middleware
Problems that message middleware can solve
In many business scenarios, we need to turn synchronous tasks into asynchronous tasks.
Take the registration function of an e-commerce platform for a simple analysis. Users’ registration of this service is not only completed by inserting a piece of data into the database, but also need to send a series of operations, such as sending activation email, sending new red packets or points, and sending marketing SMS. If every operation in this process needs 1s, then the whole registration process will take 4S to respond to the user.
Then we need to take these operations apart and optimize them into asynchronous processing logic.
We can use blocking queue + thread pool to implement producer consumer mode.
- However, this method is only applicable to a single machine. Once the machine goes down, the data stored in the blocking queue will be lost.
- Using message middleware to process
Write the request to the queue first. Message queue has length. If the length of message queue exceeds the specified length, it will be discarded directly. This cuts off the peak flow.
The specific core processing business of seckill is to receive messages in the message queue for processing. The message processing capacity here depends on the throughput of the consumer
Communication between different programming languages can be achieved through message queuing.
The ability to consume messages without worrying about the application hanging up
Of course, there are more application scenarios of message middleware. For example, in the weak consistency transaction model, distributed message queue can be used to realize maximum capacity notification to realize the final consistency of data
Thinking about the design of message middleware
We can start with basic needs
The most basic support message sending and receiving
- Network communication will consider NiO
Storage of messages
- Persistent, non persistent
- Serialization, deserialization of messages
- Cross language
Message acknowledgement mechanism
- How to avoid message retransmission
- Ordering of messages
- Do you support event messages
- Performance of message sending and receiving, support for high concurrency big data
- Is cluster supported
- Reliable transmission of messages
- Is multi protocol supported
The development of Message Oriented Middleware
In fact, the development of message oriented middleware is also very interesting. We know that the emergence of any technology is to solve practical problems, which is through a common software
BusThat is, a communication system to solve the problem between applications
Heavy information communicationWork.
The earliest mouse was the field of financial trading, because at that time, traders needed to complete transactions through different terminals, and each terminal displayed different information.
If the message bus is connected, the trader only needs to operate on one terminal and then subscribe to the messages of interest to other terminals. Therefore, PubSub was born, and the information bus, the first modern message queuing software in the world, was born. TiB allows developers to establish a series of rules to describe the content of messages. As long as the messages are published according to these rules, any consumer application can subscribe to the messages of interest.
With the benefits of TiB being widely used in various fields, IBM also began to research and develop its own message oriented middleware. Three years later, IBM’s message queuing IBM MQ product series was released. After a period of time, MQ Series evolved into the business message queuing platform market dominated by WebSphere MQ.
Microsoft also developed its own message queuing (MSMQ) in the later period
Major manufacturers have studied their own MQ, but they operate their own MQ software in a commercial mode. What commercial MQ wants to solve is the problem of application interoperability, rather than creating standard interfaces to allow different MQ products to interoperate.
Switching between different message oriented middleware
Therefore, some large financial companies may use MQ products from multiple suppliers to serve different applications within the enterprise. Then the problem comes. If the application has already subscribed to TiB MQ messages and then suddenly needs to consume IBM MQ messages, the whole implementation process will be very troublesome.
In order to solve this problem, Java Message Service (JMS) was born in 2001. By providing a common Java API, JMS hides the implementation interface of individual MQ product suppliers, thus crossing different MQ consumption and solving interoperability problems. From the technical level, Java applications only need to program for the JMS API and select the appropriate MQ driver. JMS handles other parts. However, there are a lot of problems with the interface integration.
How can two programs using two different programming languages communicate with each other through their asynchronous messaging mechanism.
The emergence of AMQP
At this time, it is necessary to define a general standard for asynchronous messaging, so AMQP (Advanced message queuing protocol) advanced message queuing protocol came into being. It uses a set of standard underlying protocols and adds many other features to support interoperability. It enriches the requirements of message passing for modern applications, aiming at the task of standard coding Anyone can interact with an MQ server provided by any AMQP vendor.
In addition to the JMS and AMQP specifications, there is also an mqtt (message queuing telemetry transport), which is specially designed for small devices. Because devices with low computational performance cannot adapt to the complex operations on AMQP, they need a simple and interoperable way to communicate.
This is the basic requirement of mqtt, and today, mqtt is one of the main components of the Internet of things (IOT) ecosystem
Kafka, which does not follow the protocol specifications mentioned above, focuses on throughput, similar to UDP and TCP
Introduction to Kafka
What is Kafka
Kafka is a distributed message publishing and subscribing system, which is characterized by high performance and high throughput.
It was originally designed as a conduit for LinkedIn’s activity flow and operational data. These data are mainly used for user profile analysis and server performance data monitoring
Therefore, Kafka was designed as a distributed, high-throughput message system at the beginning, so it is suitable for big data transmission scenarios.
Application scenarios of Kafka
Because Kafka has the advantages of better throughput, built-in partition, redundancy and fault tolerance (Kafka can process hundreds of thousands of messages per second), Kafka has become a good solution for large-scale message processing applications.
Therefore, in the enterprise application, it will be mainly applied in the following aspects
Kafka can be used to track user browsing, searching and other behaviors. Through publish subscribe mode, it is recorded in the corresponding topic in real time. Through the back-end big data platform access processing and analysis, further real-time processing and monitoring are carried out
In terms of log collection, there are many excellent products, such as Apache flume. Many companies use Kafka agent log aggregation.
Log aggregation means that log files are collected from the server and then put into a centralized platform (file server) for processing. In the actual application development, the log of our application program will be output to the local disk. If the application forms a load balancing cluster and there are more than dozens of machines in the cluster, it will be very troublesome to quickly locate the problem through the log. Therefore, a unified log collection platform is generally used to manage log logs, which are used to quickly query important application problems. Therefore, many companies focus their application logs on Kafka and then import them to es and HDFS respectively for real-time retrieval and analysis and offline statistical data backup. On the other hand, Kafka itself provides a good API to integrate logs and do log collection
Kafka’s own architecture
A typical Kafka cluster contains several producers
- The message that can be generated by the application node
- It can also be events generated by collecting logs through flume
- Several brokers (Kafka supports horizontal expansion)
- Several consumer groups
- A zookeeper cluster.
Kafka manages cluster configuration and service collaboration through zookeeper.
The producer publishes the message to the broker using the push mode, and the consumer subscribes and consumes the message from the broker by listening.
Multiple brokers work together, producer and consumer are deployed in each business logic.
The request and forwarding are managed by zookeeper. In this way, a high-performance distributed message publishing and subscribing system is formed.
There is a detail in the figure that is different from other MQ middleware. The process of producer sending message to broker is push, while the process of consumer consuming message from broker is pull, which actively pulls data. Instead of the broker actively sending data to the consumer
Explanation of terms
The Kafka cluster contains one or more servers, which are called brokers.
The broker side does not maintain the consumption state of data, which improves the performance.
Direct use of disk for storage, linear read-write, fast speed:
It avoids the replication of data between the JVM memory and the system memory, and reduces the performance consumption of object creation and garbage collection.
Responsible for publishing information to Kafka broker
The message consumer is the client who reads the message to Kafka broker, and the consumer pulls and processes the data from the broker.
Each message published to the Kafka cluster has a category called topic.
- Messages from different topics are stored separately
- Although the message of a topic is stored in one or more brokers, the user only needs to specify the topic of the message to produce or consume data, and does not care where the data is stored
Partition is a physical concept. Each topic contains one or more partitions
Each consumer belongs to a specific consumer group (the group name can be specified for each consumer, otherwise it belongs to the default group)
Topic & Partition
Topic can be regarded as a queue logically. Each consumption must specify its topic, which can be simply understood as having to specify
Put this message in which queue.
In order to increase the throughput of Kafka linearly, the topic is physically divided into one or more partitions. Each partition is physically corresponding to a folder under which all messages and index files of the partition are stored. If two topics, topic1 and Topic2, are created with 13 and 19 partitions respectively, a total of 32 folders will be generated on the whole cluster (the cluster used in this paper has 8 nodes, where both topic1 and Topic2 replication factor are 1).
install and configure
tar -zxvf kafka_2.11-2.0.0.tgz -C /usr/local
Since Kafka relies on zookeeper for master election and other data maintenance, it is necessary to start the zookeeper node first
Kafka has built-in zookeeper services, so these scripts are provided in the bin directory
In the con ﬁ g directory, there are some configuration files
So we can start ZK service through the following script. Of course, we can also build ZK cluster by ourselves
sudo sh zookeeper-server-start.sh -daemon ../config/zookeeper.properties
Start and stop Kafka
We can use our own zookeeper
Modification server.properties , add the configuration of zookeeper
sudo bin/kafka-server-start.sh -daemon config/server.properties
sh bin/kafka-server-stop.sh -daemon config/server.properties
Basic operation of Kafka
sh bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 -partitions 1 --topic test
- This indicates that several copies of the topic need to be saved in different brokers. Setting it to 1 here means that two copies are saved in two brokers
- Number of partitions
sh bin/kafka-topics.sh --list --zookeeper localhost:2181
[[email protected] kafka_2.11-2.0.0]$ sh bin/kafka-topics.sh --list --zookeeper localhost:2181 __consumer_offsets first_topic test test_partition
sh bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
sh bin/kafka-console-producer.sh --broker-list localhost:9092 --topic first_topic
Cluster environment installation
- Prepare three virtual machines
- Deploy Kafka’s installation package on three machines
The following configuration changes are server.properties
- Modify the server.properties Configuration, the ID of each machine in the same cluster must be unique
broker.id=0 broker.id=1 broker.id=2
Modify the connection configuration of zookeeper (specify the machine that is currently starting zookeeper)
Modify listeners configuration
- If listeners are configured, message producers and consumers will use the configuration of listeners to send and receive messages. Otherwise, they will use localhost
Plaintext indicates the protocol, which is plaintext by default. Other encryption protocols can be selected
- Start three servers respectively