Interviewer: if you were asked to design a highly concurrent message middleware, what would you do?

Time:2021-7-21

Write on the front

Many small partners go to a large factory for an interview and almost always encounter some open-ended questions. These open-ended questions have no fixed answers, but they can truly reflect the interviewers’ real system design ability and technical skills. If you answer perfectly, you can stand out from many interviewers through this kind of open-ended questions. Today, let’s talk about a common open-ended question when we went to a large factory for an interview: if you were asked to design a highly concurrent message middleware, what would you do?

Knowledge points of message middleware

In order to design a message middleware with high concurrency, it is necessary to understand the specific knowledge points involved in message middleware. Generally, the knowledge points needed to design a good message middleware are as follows:

  • Producer consumer model.
  • Support distributed architecture.
  • High availability of data.
  • Message data is not lost.

Next, we will talk about these technical points for message middleware.

Producer consumer model

I believe many small partners have a better understanding of the producer and consumer models. In a word, message middleware enables other applications to produce messages and consume corresponding messages.

For the producer and consumer model, we need to consider more problems. Next, I will guide you to think step by step.

First of all, let’s think about this questionIf the producer produces the message, how should the message middleware store the corresponding data? Stored in memory? Storage on disk? Or is it stored in both memory and disk?

If the message data is stored in memory and disk at the same time, how do we deal with the data? Does the producer write the data to disk immediately after delivering the message to the message middleware? Or does it mean that the data first resides in the memory, and then brushes to the disk every other period of time? If it’s on the disk every other period of time, we have to consider the problem of disk file segmentation, that is, how many disk files do we need to be divided into( You can’t put all the data in one disk file. If it is necessary to split into multiple disk files, what are the rules?

The above problems are the problems we need to consider when designing a message middleware. However, this is only a small part of the problem. If you want to stand out in the interview, you need to continue to look down, there are some important issues to pay attention to.

If the file is divided into multiple disk files according to certain rules, do you need some metadata management to identify the specific messages of the data (for example, the metadata information of the datanode is stored in the namenode node in Hadoop, and the namenode node can better manage the datanode through the metadata information)? These metadata can include: the offset of the message data, or the unique ID of the message data.

After considering the problem of data storage, we also need to consider: how does the message middleware deliver the data to the corresponding consumers?

When designing producers and consumers, we need to consider another very important question: what is the consumption mode we use when designing message middleware? Will data be evenly distributed to consumers? Or will the data be delivered to consumers through some other rules?

Support distributed architecture

If we design the message middleware, it will carry high concurrency and high throughput write operations of TB level data every day. Here, we need to consider the message middleware design component layout architecture.

In the design of distributed architecture, we also need to consider the storage of relatively large data, made into a piece of storage, data fragmentation and other operations.

In addition to these, we also need to consider another core issue: for message oriented middleware, we need to support automatic capacity expansion.

In addition, it discusses whether data fragmentation is supported, how to realize the expansion of data fragmentation and automatic data load balancing migration.

High availability of data

Generally, the high availability of Internet applications is realized through local heap memory and a copy of data on different servers. At this time, any storage node downtime will not affect the overall high availability.

Message data is not lost

At this point, we need to provide a manual ack mechanism, that is, when the consumer has finished consuming the message, the message middleware will return the “processing completed” identification, and the message middleware will delete the corresponding processed message.

However, to refine, we need two sets of ACK mechanisms:

  • An ACK corresponds to the production end. If the ACK message has not been received, the producer needs to resend a message to ensure the success of the production message.
  • Another kind of ACK corresponds to the consumer side. Once a message is processed successfully, an ACK must be returned to the message middleware, and then the message middleware can delete the message. Otherwise, once the consumer is down, it is necessary to resend this message to other consumer instances to ensure that the message will be processed successfully.

Today, we don’t talk about specific business points, but consider from the whole: if we implement a message middleware, we need to pay attention to various knowledge points and professional skills! Well, that’s all for today. See you next time!

Heavyweight benefits

WeChat search official account official, WeChat’s official account, pay attention to this deep programmer, read hard core technology dry cargo every day, reply to PDF, and I have prepared a big factory interview information and my original PDF technology document, and I have prepared many resume templates for you (constantly updated). I hope you can find your favorite job. Learning is a way of sometimes depressed and sometimes laughing. Come on. If you succeed in entering the company you like, don’t slack off. Career growth is the same as new technology learning. If you are lucky, I’ll see you in the world!

In addition, I open source PDF, I will continue to update and maintain, thank you for your long-term support to Glacier!!

Write at the end

If you think glacier‘s writing is good, please search on wechat and follow “Glacier TechnologyThe official account of WeChat is learning high concurrent, distributed, micro services, big data, Internet and cloud native technology with glaciers.Glacier TechnologyThe official account of WeChat has updated a lot of technical topics, and every technical article is full of dry cargo. Many readers have read theGlacier TechnologyWeChat’s official account of the public, who has been interviewing the official, has successfully switched to a big factory. There are also many readers to achieve a technological leap, become the company’s technical backbone! If you want to improve your ability, realize the leap of technical ability, enter a large factory, get a promotion and raise salary, then pay attention toGlacier TechnologyWeChat official account, updating the hard core technology dry cargo every day, so that you can’t lose your knowledge of how to improve the technical capability.
Interviewer: if you were asked to design a highly concurrent message middleware, what would you do?