Some experience of building high concurrency system


Some experience summary on how to build high concurrency system is for reference only. Welcome to exchange.

  • preface
  • infrastructure
  • database
  • framework
  • application
  • standard
  • summary


It has been a while since I left hungry. During the period of hungry, I began to take over the development and maintenance of Waybill system in 2017, from the earliest daily average of one million orders to the daily average of ten million orders when I left. With the rapid development of business and the characteristics of takeout business, the business volume is concentrated in the two peak periods of noon peak and evening peak, so the amount of concurrent requests in the peak period is also rising, and I have to face the challenge of high concurrency every day. Take the waybill system as an example. The QPS of the daily afternoon peak core query service is more than 200000, the QPS of redis cluster is more than one million, the database QPS is also more than 100000, and the TPS is more than 20000.

Under such a large flow, the main work is also focused on how to build the stability and improve the capacity of the system. The following mainly talks about how to build a highly concurrent system from the aspects of infrastructure, database, architecture, application and specification. The following is a summary of my personal experience in recent years. There is no silver bullet in the architecture, so it is not called a best practice,For reference only


In a layered architecture, the lowest level is infrastructure. Generally speaking, basic settings include physical server, IDC, deployment mode, etc. Like a pyramid, infrastructure is the base of the pyramid. Only when the base is stable can the upper layer be stable.
Some experience of building high concurrency system

Live more in different places

Hungry? In 2017, we did multi activity in different places. The whole scheme is led by the company’s infrastructure department. The business side needs to cooperate with the multi activity scheme to make corresponding transformation. Through this project, we also learned some knowledge about multi activity architecture.

Multiple activities can be divided into multiple activities in the same city, multiple activities in different places, etc. there are also a variety of implementation methods, such as the modular scheme used by Ali and the multi center scheme used by hungry. You can refer to the implementation of hungry multiple activities…。 At that time, the main starting point of doing more work was to ensure the high availability of the system and avoid the single point of failure of a single IDC. At the same time, because the flow of each computer room became 1 / N of the total flow, it also increased the system capacity in a disguised way, which can resist more traffic in the scenario of high concurrency. The following figure shows the overall architecture of hungry moduohuo, which comes from the sharing article on the implementation of hungry moduohuo above.

Some experience of building high concurrency system


Database is one of the most important components of the whole system. In the scenario of high concurrency, a large part of the work is carried out around the database. The main problem to be solved is how to improve the database capacity.

Read write separation

The separation of reading and writing capacity of most internet systems can reduce the effective load of reading and writing. The core idea is that the master database bears the write traffic and the slave database bears the read traffic. In the read-write separation architecture, it is generally configured with one master and multiple slaves, and multiple slave libraries share the high concurrent query traffic. For example, there are 10000 QPS and 1K TPS. Suppose that under the configuration of one master and five slaves, the master database only undertakes 1K TPS and each slave database undertakes 2K QPS. This order of magnitude is completely acceptable for DB. Compared with before the read-write separation transformation, the pressure of DB is significantly lower.

Some experience of building high concurrency system

The advantage of this mode is that it is simple. There is almost no code transformation cost or only a small amount of code transformation cost. You only need to configure the master-slave database. The disadvantages are equally obvious:

Master slave delay

MySQL’s default master-slave replication is asynchronous. If you query the slave database immediately after inserting data into the master database, you may not find it. Under normal circumstances, there will be millisecond delay in master-slave replication. In the case of high DB load, there may be second delay or even longer, but even millisecond delay can not be ignored for businesses with high real-time requirements. Therefore, in some key query scenarios, we will bind the query request to the master database to avoid the problem of master-slave delay. There are also many articles on the optimization of master-slave delay, which will not be repeated here.

The number of slave libraries is limited

The number of slave libraries that a master library can mount is very limited, and there is no way to achieve unlimited horizontal expansion. The more slave libraries there are, the higher the QPS they can bear in theory, but too many slave libraries will lead to greater IO pressure of master-slave replication of the master library, resulting in higher delay, which will affect the business. Therefore, generally speaking, only a limited number of slave libraries will be mounted after the master library.

The problem of high TPS cannot be solved

Although the slave library can solve the problem of high QPS, it cannot solve the problem of high TPS. Only the master library can handle all write requests. Once the TPS is too high, the DB still has the risk of downtime.

Sub database and sub table

When the separation of reading and writing cannot meet the business needs, we need to consider the use of sub database and sub table mode. When we decide to optimize the database, we should give priority to the use of the read-write separation mode. Only when the read-write separation mode can no longer bear the traffic of the business, can we consider the database and table separation mode. The final effect of sub database and sub table mode is to change single database and single table into multi database and multi table, as shown in the figure below.

Some experience of building high concurrency system

First, let’s talk about the following sub table, which can be divided into vertical split and horizontal split. Vertical splitting is to split by business dimension. Assuming that there are 100 fields in the original order table, it can be divided into multiple tables according to different business dimensions, such as one table of user information, one table of payment information, etc., so that there will not be too many fields in each table.

Some experience of building high concurrency system

Horizontal splitting is to split a table into n tables, such as one order table into 512 order sub tables.

Some experience of building high concurrency system

In practice, you can only do horizontal or vertical splitting, or you can do horizontal and vertical splitting at the same time.

Some experience of building high concurrency system

Having finished the sub table, what is the sub library? Database splitting is to split the tables originally in one DB instance into n DB instances according to certain rules. Each DB instance will have a master, which is equivalent to the architecture of multiple matter. At the same time, in order to ensure high availability, each master must have at least one slave to ensure that the slave can be on top in time when the Master goes down and that the data is not lost. After splitting, there will only be some tables in each DB instance.

Due to the multi master architecture, in addition to all the advantages of the read-write separation mode, the sub database and sub table can also solve the problem of too high TPS that cannot be solved in the read-write separation architecture. At the same time, the sub database and sub table can be expanded infinitely horizontally in theory, and also solve the problem of limited number of slave libraries under the read-write separation architecture. Of course, in actual engineering practice, it is generally necessary to estimate the capacity in advance, because the database is stateful. If it is found that the capacity is insufficient, it is very troublesome to expand it, which should be avoided as far as possible.

In the mode of sub database and sub table, the problem of master-slave delay can be avoided by not enabling the query of slave database, that is, the reading and writing are in the master database, because after the sub database, the traffic on each master only accounts for 1 / N of the total traffic. In most cases, it can carry the business traffic. The slave database is only used as a backup of the master. When the master database is down, the master-slave switch is performed to replace the master to provide services.
After finishing the benefits, let’s talk about the problems caused by sub database and sub table, mainly including the following points:

High transformation cost

Sub database and sub table generally need the support of middleware. There are two common modes: client mode and agent mode. The client mode implements the logic of database and table division by referencing the client package on the service. The most representative is the open source sharding JDBC.
Agent mode refers to the way that all services are not directly connected to MySQL, but through connecting agents, and then connecting agents to MySQL. Agents need to implement MySQL related protocols.

Some experience of building high concurrency system

The two modes have their own advantages and disadvantages. The agent mode will be relatively more complex, but because there is one more layer of agent, more things can be done in the agent layer, which is also more convenient for upgrading. Moreover, connecting to the database through the agent can also ensure the stability of the number of connections to the database. The advantage of using the client mode is that the implementation is relatively simple, there is no intermediate agent, and the performance will be better in theory. However, when upgrading, the business party needs to modify the code, so it will be more difficult to upgrade than the agent mode. The agent mode is used in hungry. Hungry has a unified database Access Middleware – Dal, which is responsible for acting on all databases, realizing the logic of database and table division, and maintaining transparency to the business.

Transaction issues

In business, we will use transactions to handle multiple database operations, and ensure the correctness of business processes through the four characteristics of transactions – consistency, atomicity, persistence and isolation. After splitting databases and tables, a table will be split into n sub tables, which may be in different DB instances. Therefore, although it looks like a table logically, it is no longer in a DB instance, which leads to the problem that transactions cannot be used.

The most common is that in batch operation, we can put the operation of multiple orders into one transaction at the same time before splitting the database and table, but we can’t do so after splitting the database and table, because different orders may belong to different users. Assuming that we divide the database and table according to users, the order tables of different users are located in different DB instances, and multiple DB instances obviously can’t use one transaction, This requires some other means to solve this problem. After dividing databases and tables, we should try to avoid this kind of cross DB instance operation. If it must be used in this way, we should give priority to using compensation and other methods to ensure the final consistency of data. If it must be strong consistency, the common scheme is through distributed transactions.

Cannot support multi-dimensional queries

The sub database and sub table can only be divided according to 1-2 dimensions, which is the so-called dimensionsharding key。 Common dimensions include user, merchant and other dimensions. If tables are divided according to the user’s dimension, the simplest way to implement it is to take the module according to the user ID to locate which sub database and sub table, which means that all subsequent read-write requests must carry the user ID. however, in actual business, it is inevitable that there will be multiple dimension queries, and not all queries will have user IDs, which requires us to transform the system.

Some experience of building high concurrency system

In order to support multi-dimensional query after database and table division, there are two common solutions. The first is to introduce an index table. Whether this index table has no database and table division, or take the database and table division by user ID as an example. The mapping relationship between various dimensions and user ID is recorded in the index table. The request needs to query the index table through other dimensions to obtain user ID, and then query the table after database and table division through user ID. In this way, one more IO is required, and the index table is easy to become a system bottleneck because there is no sub database and sub table.

The second scheme is through the introduction ofNoSQLThe more common combination isES+MySQL, orHBase+MySQLIn essence, this scheme still acts as the index table in the first scheme through NoSQL, but compared with using the index table directly,NoSQLWith better horizontal scalability and scalability, it is generally not easy to become the bottleneck of the system as long as it is properly designed.

data migration

Data migration is generally required for sub database and sub table. The original single table data is migrated to the database table after sub database and sub table through data migration. There are two common data migration schemes. The first is downtime migration. As the name suggests, this method is simple and rough. The advantages are that it can be achieved in one step, the migration cycle is short, and the data consistency can be guaranteed. The disadvantage is that it is detrimental to the business. Some key businesses may not accept the business loss caused by downtime migration for a few minutes or more.

Another scheme is double write, which is mainly aimed at the new incremental data. The stock data can be synchronized directly. There have been many shares on how to carry out double write migration on the Internet, which will not be repeated here. The core idea is to write the old database and the new database at the same time. The advantage of double write is that it has little impact on the business, but it is also more complex, the migration cycle is longer, and it is prone to data inconsistency. It needs to be supported by a complete data consistency assurance scheme.


The read-write separation mode and the sub database and sub table mode are recommended to give priority to the use of the read-write separation mode. The sub database and sub table mode can be considered only when the business needs are not satisfied. The reason is that although the database and table splitting mode can significantly improve the capacity of the database, it will increase the complexity of the system. In a sense, it is also a limitation to the business system because it can only support a few dimensions of reading and writing. Therefore, when designing the database and table splitting scheme, it needs to be considered more comprehensively in combination with specific business scenarios.


Architecture is also very important in the construction of high concurrency systems. Here we share some experience in caching, message queuing, resource isolation and so on.


In the highly concurrent system architecture, caching is the most effective weapon, which can be said to be none of them. The biggest function of cache is to improve the system performance, protect the back-end storage from being destroyed by large traffic, and increase the scalability of the system. The concept of cache originated from the CPU. In order to improve the processing speed of the CPU, L1, L2 and L3 caches are introduced to speed up access. Now the cache used in the system also draws lessons from the practice of cache in the CPU.

Caching is a very big topic. It’s no exaggeration to write a book alone. Here I summarize some problems and solutions I personally encountered in the design and implementation of Waybill system caching. Cache is mainly divided into local cache and distributed cache, such asGuava CacheEHCacheSuch as distributed cacheRedisMemcachedDistributed cache is mainly used in the waybill system.

How to ensure the data consistency between cache and database

The first is how to ensure the data consistency between the cache and the database. This problem will be encountered when using the cache. At the same time, it is also a high-frequency interview question. The problem of using cache in the waybill system I am responsible for is even more prominent. Firstly, the waybill will be updated frequently, and the waybill system has very high requirements for data consistency. It is basically not able to accept data inconsistency, so we can’t simply set an expiration time to invalidate the cache.

About the cache read-write mode, it is recommended to read uncle mouse’s article:…, which summarizes several commonly used read-write cache routines. I also refer to the cache read-write mode in the waybill systemWrite throughThe pattern, through pseudo code, is roughly as follows:

Lock (waybill ID){
    //Delete cache
    //Update DB
    //Rebuild cache

Since it isWrite throughMode, the update of the cache is carried out in the write request. First of all, in order to prevent concurrency problems, write requests need to add distributed locks. The granularity of locks is based on the waybill ID as the key. After executing the business logic, delete the cache first, then update the DB, and finally rebuild the cache. These operations are carried out synchronously. In the read request, query the cache first, and return directly if the cache hits. If the cache does not hit, query the DB, and then return directly, that is, the cache will not be operated in the read request, This method converges the cache operation to the write request, and the write request is locked, which effectively prevents the problem of writing dirty cache data caused by read-write concurrency.

Design of cache data structure

The cache should avoid the problems of large keys and hot keys. For example, if you useredisMediumhashData structure, it is easier to have large key and hot key problems than ordinary string type keys, so if it is not necessary to use ithashSome specific operations can be consideredhashBreak it up into a single key / value pair and use the ordinary key / value pairstringType of key storage, which can preventhashLarge key problems caused by too many elements can also be avoidedhash keyThe problem of overheating.

Read write performance

There are two main points to consider about read-write performance. The first is write performance. The main factor affecting write performance is the data size of key / value. Relatively simple scenarios can be usedJSONBut using JSON in high concurrency scenarios can not meet the performance requirements, and it also takes up more storage space. The more common alternatives areprotobufthriftAnd so on. There are also some performance comparisons on these serialization / deserialization schemes online. Please refer to…

Some experience of building high concurrency system

The main influencing factor of read performance is the size of data packets read each time. Recommended in practiceredis pipeline+Batch operation, for example, if it is a string type key, it ispipeline+mgetAssume oncemget10 keys, 100 keysmgetFor a batch of pipelines, the network IO can query 1000 cache keys at that time. Of course, the specific number of batches here depends on the packet size of cache keys. There is no unified value.

Appropriate redundancy

Appropriate redundancy means that we can make some redundancy when designing external business query interfaces. This experience comes from the fact that when we were designing the external query interface of the waybill system, in order to pursue universality, we designed the return value of the interface into a large object and directly exposed all the fields on the waybill in this large object. The advantage is that there is no need to develop different interfaces for different query parties. Anyway, the fields are in the interface and you can get what you want.

It’s OK to do this at first, but when we need to add cache to the query interface, we find that since all business parties query waybill data through this interface, we can’t know their business scenarios and their requirements for interface data consistency, such as whether they can accept short-term data consistency, and we don’t know which fields in the interface they use, Some fields in the interface will not change, and some fields will change frequently. In fact, different cache design schemes can be adopted for different update frequencies, but it is a pity that we design the interface too much in pursuit of generality, so it is very troublesome to do cache optimization. We can only design the scheme according to the worst case, that is, all business parties have high requirements for data consistency, As a result, the final scheme spent a lot of energy on data consistency.

If we can make some appropriate redundancy when designing the external query interface at the beginning and distinguish different business scenarios, although this will inevitably lead to the similar functions of some interfaces, we can have a clear aim when adding cache and design different schemes for different business scenarios. For example, key processes should pay attention to the guarantee of data, Non critical scenarios allow temporary data inconsistencies to reduce the cost of cache implementation. At the same time, it is better to distinguish the updated fields from the fields that will not be updated in the interface. In this way, when designing the cache scheme, you can set a longer expiration time for the fields that will not be updated, while for the fields that will be updated, you can only set a shorter expiration time, and you need to design the cache update scheme to ensure data consistency.

Message queue

In the architecture of high concurrency system, message queue (MQ) is essential. When large traffic comes, we can increase the scalability of the system through the asynchronous processing of message queue and the characteristics of peak shaving and valley filling to prevent large traffic from breaking the system. In addition, the use of message queue can also achieve the purpose of full decoupling between systems.

Some experience of building high concurrency system

The core model of message queue is composed of producer, consumer and message broker. At present, the open source solutions commonly used in the industry areActiveMQRabbitMQKafkaRocketMQCompared with those in recent yearsPulsar, for comparison of various message oriented middleware, please refer to the following article:…

After using message queue, you can change the requests processed synchronously to consume MQ messages asynchronously, which can reduce the pressure of system processing and increase the system throughput. There are many articles on how to use message queue. My experience here is that when considering using message queue, you should decide whether to introduce message queue in combination with specific business scenarios, Because the use of message queue actually increases the complexity of the system. For things that can be solved through a synchronous request, additional dependencies need to be introduced, and the consumption of messages is asynchronous. Asynchronous is inherently more complex than synchronization. It also needs to consider the problems of message disorder, delay, loss and so on. How to solve these problems is a big topic. There is no free lunch in the world, Any architecture design is a trade-off process, which requires careful consideration of gains and losses before making a decision.

Governance Services

Service governance is a big topic, which can be mentioned separately. Here, I also put it into the architecture. Service governance is defined as

It generally refers to providing some system guarantee measures for reliable operation of the system independent of business logic.

Common safeguard measures include service registration and discovery, observability (monitoring), current limiting, timeout, fusing, etc. in the microservice architecture, service governance is generally completed through the service governance framework. Open source solutions includeSpring CloudDubboWait.

In highly concurrent systems, service governance is a very important piece of content. Compared with the large pieces of content such as cache and database, service governance is more about details, such as whether the timeout setting of the interface is 1 second or 3 seconds, how to monitor, etc. there is a saying that details determine success or failure. Sometimes, I have seen large-scale failures caused by unreasonable timeout setting of an interface, Especially in highly concurrent systems, we must pay attention to these details.

Some experience of building high concurrency system


The principle of timeout is:Everything has a timeout。 No matter RPC call, redis operation, consumption message / send message, DB operation, etc., there must be timeout. When you were hungry, you had to rely on external components, but you didn’t set a reasonable timeout. When the external dependency failed, all threads of the service were blocked, resulting in resource depletion and failure to respond to external requests. These are “blood” training.

In addition to setting the timeout, it is also important to set a reasonable timeout. As mentioned above, even if the timeout is set, the service will still collapse due to external dependency failure if the timeout is too long.
How to set a reasonable timeout is very particular, which can be considered from the aspects of whether it is a key business scenario and whether it is strongly dependent. There are no general rules, which need to be combined with specific business scenarios. For example, in some C-side display interfaces, setting the timeout of 1 second seems to be no problem, but in some performance sensitive scenarios, 1 second may be too long. In short, it needs to be set in combination with specific business scenarios, but anyway, the principle is the same: everything has timeout.


Monitoring is the eye of the system. A system without monitoring is like a black box. If we don’t know the operation inside from the outside, we can’t manage and operate the system. Therefore, the monitoring system is very important.
The observability of the system mainly includes three parts——loggingtracingmetrics。 The self-developed monitoring system was mainly used in hungry before. I have to say that it is really very easy to use. For specific introduction, please refer to:…
When building a high concurrency system, we must have a perfect monitoring system, including system level monitoring (CPU, memory, network, etc.), application level monitoring (JVM, performance, etc.), business level monitoring (various business curves, etc.), in addition to monitoring, there must be a perfect alarm, because it is impossible for someone to watch the monitoring for 24 hours. Once there is any risk, we must alarm and intervene in time to prevent the risk.


In the microservice framework, the feature of fusing is generally built in. The purpose of fusing is to protect its own services in case of downstream service failure. The realization of fusing usually has a circuit breaker(Crit Breaker), the circuit breaker will judge whether to trigger the fusing according to the rules such as interface success rate / times, and the circuit breaker will control the fusing state to flow in the process of closing, opening and semi opening. The restoration of fusing will go through the half open state through the mechanism of time window. If the success rate reaches the threshold, the fusing state will be closed.

Some experience of building high concurrency system

If there are no special requirements, there is generally no need to do anything for fusing in the business system, and the framework will automatically turn on and off the fusing switch. One thing that may need attention is to avoidInvalid fuse, what is an invalid fuse? In the past, there was a fault that the service provider threw unreasonable exceptions (such as system exceptions) in some normal business verification, resulting in interface fusing and affecting normal business. Therefore, when throwing an exception or returning an exception code in the interface, we must distinguish between business and system exceptions. Generally speaking, business exceptions do not need to be fused. If a business exception throws a system exception, it will lead to fusing and the normal business process will be affected.


Downgrade is not a specific technology, but more like a methodology of architecture design. It is a strategy of losing soldiers and protecting the commander. The core idea is to limit some of their abilities in abnormal circumstances to ensure the availability of core functions. There are many ways to realize degradation, such as configuration, switching, current limiting and so on. Degradation is divided into active degradation and passive degradation.

Some experience of building high concurrency system

When the e-commerce system is greatly promoted, we will temporarily close some non core functions to ensure the stability of core functions, or degrade the downstream services in order to ensure the stability of their own services when the downstream services fail and cannot be recovered in a short time. These are active degradation.

Passive degradation refers to, for example, calling a downstream interface, but the interface times out. At this time, in order to continue the execution of the business process, we usually choose to use it in the codecatchException, print an error log, and then continue to execute business logic. This degradation is passive.

It is very important to downgrade in highly concurrent systems. For example, when the number of requests is large, it is inevitable that there will be timeouts. If the business process is interrupted every time, it will greatly affect the normal business. A reasonable approach is that we should carefully distinguish between strong and weak dependencies. For weak dependencies, we should adopt the passive degradation method, while for strong dependencies, we cannot downgrade. Similar to circuit breaker, degradation is also the protection of our own services to avoid the collapse of our own services in case of external dependence failure. Therefore, we should make sufficient degradation plans.

Current limiting

There are also many articles and introductions about current limiting, and the specific technical implementation can refer to the articles on the Internet. My personal experience about current limiting is that before setting current limiting, we must fully estimate the system capacity through pressure measurement and other methods. Don’t beat your head. Generally speaking, current limiting is detrimental to the user experience and should be used as a means to reveal the bottom rather than a conventional means.

Resource isolation

There are various types of resource isolation, including server resources and middleware resources at the physical level, thread pools and connection pools at the code level. The resource isolation introduced here is mainly at the application deployment level, such asSetwait. The above-mentioned multi activity in different places is also a kind of set.

When I was in charge of the waybill system, I also did some similar optimization on resource isolation. The background is that there was an online failure at that time. The reason is that the servers deployed by a service are all in a cluster, and they are not divided into separate clusters according to traffic, resulting in the failure caused by the interaction of key business and non key business traffic. Therefore, after this failure, I also decided to deploy the server in isolation by cluster. The isolation dimension is mainly divided into three categories: key cluster, sub key cluster and non key cluster, which can avoid the interaction between key and non key businesses.

Some experience of building high concurrency system


In terms of architecture, I am not a professional architect, and I have been learning relevant technologies and methodologies. Many of the technologies and architecture design patterns introduced above are learned and practiced at work. If we have to sum up some experience, I think we should pay attention to details. Personally, I think that the architecture is not only the methodology of height, but also the technical details. It is the so-called details that determine success or failure. Sometimes forgetting to set a small timeout may lead to the collapse of the whole system.


In highly concurrent systems, there are many optimizations that can be done at the application level. This part mainly shares the optimizations on compensation, idempotent, asynchronization, preheating and so on.


Under the microservice architecture, different services will be split according to each business field. The services interact with each other through RPC requests or MQ messages. In the distributed environment, there is bound to be call failure. Especially in the highly concurrent system, due to the higher server load, the probability of failure will be greater, so compensation is more important. There are two common compensation modes:Timed task modeperhapsMessage queuing mode

Timed task mode

The mode of timing task compensation generally needs to cooperate with the database. A timing task will be started during compensation. When the timing task is executed, it will scan whether there is data to be compensated in the database. If there is, it will execute the compensation logic. The advantage of this scheme is that the data is kept in the database for a long time, which is relatively stable and not easy to cause problems. The disadvantage is that it depends on the database, When the amount of data is large, it will cause a certain pressure on the database, and the timing task is executed periodically, so there will be a certain delay in general compensation.

Message queuing mode

The mode of message queue compensation generally uses the characteristics of delayed messages in message queue. If the processing fails, send a delay message and try again after delaying n minutes / second / hour. The advantage of this scheme is that it is relatively lightweight. There is no external dependency except MQ, the implementation is relatively simple, and it is relatively more real-time. The disadvantage is that it is not persistent in the database, there is a risk of data loss, and it is not stable enough.
Therefore, my personal experience is that the mode of timed task is used in the compensation of critical links, and the mode of message queue can be used in the compensation of non critical links. In addition, there is a particularly important point in compensationIdempotencyDesign.


Idempotent operations are characterized byThe impact of any multiple execution is the same as that of one execution, reflected in the business, is that the results of one request or multiple requests initiated by the user for the same operation are consistent, and there will be no side effects due to multiple requests.
In distributed systems, system errors are inevitable. When errors occur, Retry, compensation and other means will be used to improve fault tolerance. In highly concurrent systems, the probability of system errors is higher. Therefore, at this time, interface idempotence is very important to prevent the side effects caused by multiple requests.

Idempotent needs to be implemented through a unique business ID or token. The general process is to first query whether the unique business ID or token exists in the dB or cache and whether the status is processed. If so, it means that it is a repeated request. Then we need idempotent processing, that is, we can return directly without any operation.

Some experience of building high concurrency system

When designing idempotency, it should be noted that not all scenarios need to be idempotent, such as users’ repeated transfer, withdrawal, etc., because idempotency will make the external system perceive that the call is successful and does not block the subsequent process, but in fact, there is no operation in our system. Similar to the above-mentioned scenarios, users will mistakenly think that the operation has been successful. Therefore, we should carefully distinguish between business scenarios that require idempotence and those that cannot. For business scenarios that cannot be idempotent, we still need to throw business exceptions or return specific exception codes to block subsequent processes and prevent business problems.


The message queue mentioned above is also a kind of asynchronization. In addition to relying on external middleware, we can also do asynchronization in the application through thread pool and collaborative process.

Some experience of building high concurrency system

As for the implementation principle of thread pool, take the thread pool model in Java as an example. The core is realized by matching task queue and reusing threads. There are many articles about these sharing on the Internet. When using thread pool or co process and other similar technologies, my personal experience is that the following two points need special attention:

Key business scenarios need to be compensated

As we all know, both the thread pool and the collaboration process are based on memory. If the server goes down or restarts unexpectedly, the data in memory will be lost, and the thread pool will reject the task when the resources are insufficient. Therefore, if thread pool and other similar technologies are used in some key business situations, they need to be used together with compensation to avoid the business impact caused by the loss of data in memory. In the waybill system I maintain, a key business scenario is receipt. In short, it is to receive upstream requests and generate waybills in the system. This is the entrance of the whole logistics performance flow. It is a particularly key business scenario.

Because the whole process of generating waybills is relatively long and depends on more than 10 external interfaces, in order to pursue high performance and throughput, the asynchronous mode was designed, that is, processing in the online process pool. At the same time, in order to prevent data loss, perfect compensation measures were taken. There were basically no problems in the entry of waybills in the past few years. Due to the asynchronous design, the performance was very good, What exactly do we do.

Some experience of building high concurrency system

The first step of the whole process is the failure of all requests. If all requests fail, the first step of the whole process will fail. After successfully dropping the database, encapsulate a task and submit it to the thread pool, and then directly return success to the upstream. All subsequent processing is carried out in the process pool. In addition, there is a scheduled task that will compensate regularly. The data source of compensation is the data dropped in the first step. Each record dropped in the database will have a flag field to indicate the processing status. If it is found that it is not processed or the processing fails, the compensation logic will be triggered again through the scheduled task. After the compensation is successful, the flag field will be updated to successful processing.

Do a good job of monitoring

In microservices, such as RPC interface calls and MQ message consumption, including middleware and infrastructure monitoring, these will basically be targeted and well monitored. However, similar to thread pool, there is generally no ready-made monitoring, which needs to be reported and monitored by the user, which is easy to be omitted. We know that the implementation of thread pool will have memory queue, and we generally set a maximum value for memory queue. If the maximum value is exceeded, tasks may be discarded. At this time, similar problems cannot be found without monitoring. Therefore, we must monitor the use of thread pool.
What are the indicators of thread pool that can be monitored? In my experience, thread pool is generally reportedNumber of active threadsas well asNumber of work queue tasks, I think these two indicators are the most important. Other indicators have different opinions and can be selectively reported in combination with specific business scenarios.


Warm Up。 When the system is at low water level for a long time and the flow increases suddenly, directly raising the system to high water level may crush the system instantly. Through “cold start”, let the flow through increase slowly and gradually increase to the upper limit of the threshold within a certain time, so as to give the cold system a warm-up time to avoid the cold system being crushed.

Referring to the online definition, to put it bluntly, if the service has been at a low water level, a sudden wave of high concurrent traffic may break the system all at once. The preheating of the system generally includes JVM preheating, cache preheating, DB preheating, etc. by preheating, the system can be “hot” first to prepare for the arrival of high concurrent traffic.
There are many scenarios for preheating practical applications. For example, before the promotion of e-commerce, we can load some hot commodities into the cache in advance to prevent large traffic from impacting the DB. Another example is java service. Due to the dynamic class loading mechanism of JVM, we can do a wave of pressure test on the service after startup and load the classes into memory in advance. At the same time, we can trigger JIT compilation and code cache in advance.

Another way to warm up is to make use of the characteristics of the businessPreloadFor example, when maintaining the waybill system, we made such an optimization. In a normal takeout business process, after the user places an order, the user generates an order through the user trading system, and then goes through the process of payment – > merchant receiving an order – > requesting distribution. Therefore, there is a time difference from seconds to minutes from the user placing an order to requesting distribution. We can perceive the user’s action of placing an order, Use this time difference to load some data in advance.

Some experience of building high concurrency system

In this way, when the actual request comes, we only need to get it from the cache, which is very significant for some time-consuming operations. Previously, we used this method to improve the interface performance by more than 50%. Of course, one thing to note is that if some data may be changed, it may not be suitable for preheating, because after preheating, the data is stored in the cache, and the interface will not be requested later, which will lead to data inconsistency, which needs special attention.


When designing highly concurrent systems, we always pay special attention to architecture, infrastructure and so on. These are indeed very important, but in fact, there are many optimizations that can be done at the application level, and the cost will be much lower than that of architecture and infrastructure. In many cases, the optimization at the application level needs to be combined with specific business scenarios and use specific business scenarios to make reasonable design, such as caching and asynchronization. We need to think about which business scenarios can be cached and asynchronized, and which need synchronization or query dB. We must combine business to make better design and optimization.


This is the last part of the experience sharing on building high concurrency systems, but I think the importance of specifications is not lower than infrastructure, architecture, database and application, and may be more important than these. According to the 28 law, in the whole life cycle of software, we spend 20% of the time creating the system, but we spend 80% of the time maintaining the system, which reminds me that in a word, some people say that the code is mainly for people to read and the machine to run, which actually reflects the importance of maintainability.

After we have used the tall architecture and made various optimizations, the system does have a better design, but the problem is how to prevent architecture corruption in the subsequent maintenance process. At this time, it needs to be standardized.

Specifications include code specifications, change specifications, design specifications, etc. of course, I won’t introduce how to design these specifications here. What I want to say is that we must pay attention to specifications. Only with specifications can the maintainability of the system be guaranteed. According to the broken window theory, through various specifications, we try not to let the system have the first broken window.


Having said so much about design and optimization methods, I would like to share two more points.

The first point is that there is a famous saying -“Premature optimization is the root of all evil”Personally, I agree that all these designs and optimizations I do are only done when the system encounters actual problems or bottlenecks. Do not optimize too early without departing from the actual scene, otherwise it is likely to do useless work or even outweigh the gains and losses.

The second point is to follow when designingkiss principle , that is, keep it simple, stupid. Simplicity means higher maintainability and less prone to problems. Perhaps this is the so-called simplicity of the road.

Some experience of building high concurrency system

These are all my experiences in maintaining high concurrency systems during my work. In view of the length and personal technical level, some parts may not be introduced in particular in detail and in depth, which can be regarded as throwing a brick to attract jade. If there is anything wrong, you are also welcome to point it out, and you are also welcome to exchange and discuss it.

Recommended Today

JVM + GC parsing (premise knowledge concatenation)

Premise preparation JVM GC garbage collection JVM virtual machine monitoring, tuning and troubleshooting Tomcat and microservice optimization 1. Premise review 1.1. JVM memory structure 1.1.1、 JVM Architecture Overview The gray part in the figure isThread private, there is almost no garbage collectionOrange partThread sharing, the main place where garbage recycling occurs What is the class […]