Transformation of message oriented middleware under distributed services

Time:2021-10-22

1、 Background introduction

In the early stage of system development, it is easy for developers on different business lines to give priority to familiar ones in model selection due to the influence of technology stack and version time, such as Kafka, rocket, rabbit, etc. commonly used in MQ middleware, so it is easy to ignore the component differences between various projects;

In the middle and late stage of system development, after the business is relatively stable, the modules with high resource occupation will be gradually reconstructed and the public services will be integrated and managed, so as to make the system more integrated. In this process, solving the middleware differences of different projects is usually the first to bear the brunt, such as common cache center, MQ message management, etc;

Generally speaking, this situation is difficult to avoid. At the initial stage of the system, many pits are buried in order to quickly support the business. Once the business can develop stably and the sustainability is verified, appropriate consideration will be given to gradually carrying out module reconfiguration to reduce the cost.

2、 Reconstruction ideas

2.1 initial problems

In the early stage of research and development of a startup company, there were five projects developed in parallel on the business line. At that time, the use of MQ was as follows:

  • Rocket: there are three core business projects with different versions;
  • Kafka: the data weight is too high, which is adopted for one project;
  • Redis: Based on Python connection, queue message mode;

At the beginning, it was not used much and the whole was still under control. With the continuous iteration of the business, there was a need for communication between projects, which began to be chaotic and difficult to maintain. Then, it was forced to start reconstruction and unified messaging components.

2.2 secondary selection

Based on the comprehensive consideration of business, MQ is redesigned for several existing projects. The overall architecture idea is as follows:

  • MQ component selection: rocketmq;
  • Replace the queue mode of redis component;
  • Change the Python based system to Java language;
  • Provide two services: message production and consumption;
  • The functions of MQ are uniformly maintained by the above services;

There is no change in component selection on the core business line here. One reason for replacing Kafka is that it involves a large number of settlement businesses, the redis queue mode is abandoned, and the Python based management system has few functions. It is just replaced here to unify the programming language of the business line. After this design, it will be much more reasonable from the overall idea.

3、 Transformation process

3.1 overall idea

Description of core roles, from left to right:

  • Production client: the node that needs to request the server to communicate can call the message sending interface encapsulated by the production server;
  • Production server: encapsulate message sending API, maintain routing management, permission identification, message landing storage, etc;
  • Message storage layer: it is mainly stored based on message middleware, and the database layer is used to handle secondary scheduling under specific circumstances;
  • Consumer server: encapsulate the message receiving API, and request the specified consumer interface according to the routing ID to complete the communication;
  • Consumer client: respond to the request of the consumer server and encapsulate the specific business logic during consumption;

There is not much difference in the overall technical difficulty, but the introduction of two services [production and consumption] is used to manage MQ communication processes, adapt specific business logic, and introduce the database for one-time floor storage. It is more flexible in handling abnormal processes, so that the whole message module has strong scalability.

3.2 detailed description

  • Component selection

There are many choices of message oriented middleware, but in view of the familiarity of business line developers and referring to the test comparison reports provided by many parties, rocketmq components are finally selected. At the same time, rocketmq related characteristics: high performance, high reliability and support for distributed transactions are also the core considerations.

  • Microservice architecture

Based on the current microservice architecture mode, the MQ function itself is integrated into two core services for unified management and iteration, as well as component version control. For all produced messages, the global routing control is carried out, and in specific cases, the message delay consumption and the secondary scheduling execution of failure messages are realized through the application service layer function design, And manual triggering of some single messages.

  • data storage

The secondary storage of message entities is mainly to adapt to some specific function points. Some messages can be processed with delay. For example, when the MQ queue accumulates or reaches the monitored early warning line, you can intervene through configuration means to store and store some messages only, do not push MQ, and wait for the service to be sent when it is relatively idle.

As a stable support for decoupling between systems, message oriented middleware needs to have a clear design route and the monitoring and recording of key process nodes to ensure the stability and fault tolerance of the whole link.

Same seriesDistributed Concept | Distributed transaction | Kafka cluster | Rocketmq component | Redis cluster

4、 Source code address

Gitee · address
https://gitee.com/cicadasmile
Wiki · address
https://gitee.com/cicadasmile/butte-java-note/wikis

Reading labels

Java Foundation】【Design pattern】【Structure and algorithm】【Linux system】【database

Distributed architecture】【Microservices】【Big data component】【Springboot advanced】【Spring & Boot Foundation

Data analysis】【Technical map】【 Workplace