Welcome to “Keegan small steel” public address for more articles.
Matchmaking engine development: the beginning
Match engine development: MVP version
Match engine development: data structure design
Match engine development: docking black box
The previous several articles have talked about the design of the black box, including the core software structure, data structure, directory structure, etc. From the beginning of this section, we will go deeper to decrypt more design and implementation details inside the black box.
The first step of decrypting the black box is to know how to process the data in it. When we want to design a new system, it is the same. The first step is to sort out the business process and data flow. For matchmaking engine, it is to understand:What are the processes from input to output。
As mentioned in the previous article, this matchmaking engine defines three types of input:Open match, process order, close match。 Let’s take a look at the process behind these three types of input.
To start matching is to start the matching engine of a certain transaction target (transaction pair). The transaction target that has not been opened cannot process the order, and the transaction target that has been opened matching cannot be opened again. Otherwise, there will be two engines processing the same transaction target order at the same time, which is unreasonable. The order of the same transaction target can only be serialized by one engine Handle.
Why can’t we do it in parallel? If the orders of the same transaction target can be processed by multiple engines in parallel, at least several problems will arise:
- What is the transaction price?In theory, there can only be one transaction price at a time. After parallel, there will be multiple transaction prices, which will be difficult to determine.
- How to maintain a unified entrusted ledger?In theory, each transaction object has a trust ledger that holds all the trust orders. After parallel, how to maintain this unified ledger among multiple engines? If the database is maintained uniformly, it will undoubtedly reduce the matching performance; if it is divided into multiple sub ledgers, it is difficult to guarantee the principle of price priority and time priority.
The above two problems are not easy to solve, so all orders can only be sequenced first, and then dropped into the engine for serial processing.
When it comes to sequencing, a sequencing queue is naturally needed. Therefore, when enabling matching, the order sequencing queue corresponding to the transaction target needs to be initialized. After initializing the sequencing queue, you can actually start the corresponding transaction target engine. In the Go program, the engine of each transaction target runs as an independent goroutine, while in other languages, such as Java, it runs as an independent thread.
After the engine is started, the transaction entrustment ledger needs to be initialized to save the entrustment document. After that, we wait for the orders in the sequencing queue to be taken out one by one for processing.
In addition, consider another scenario, what happens when the matchmaker restarts? Is it necessary to restore the transaction targets that have been matched after restart? If necessary, how to recover? The simplest solution is to use caching. Redis is used to cache the matched transaction targets. When restarting, you can load and reopen the transaction targets from redis.
Therefore, there are two scenarios to trigger the start of matchmaking: one is triggered by the active call of the interface, and the other is automatically loaded and started from the Redis cache after the program is restarted.
Finally, the result of enabling the match is returned synchronously, so it has no asynchronous output.
In summary, the internal process of opening matchmaking is roughly as follows:
After enabling the match, you can receive the input for processing the order. When the matcher receives the request to process the order, the first step needs to do some checks, including whether each parameter is valid, whether the order is repeated or exists, whether the engine of the corresponding transaction target has been opened, etc. After passing the check, you can cache the whole order to redis, and then add it to the sequencing queue of the corresponding transaction subject, waiting for the engine of the corresponding transaction subject to consume it for matching processing. The flow chart is as follows:
When the order is successfully added to the sequencing queue, the interface can synchronously return the successful response results. Subsequent processing results are output through asynchronous MQ. After the transaction target engine receives the order, it will produce different output results according to different situations.
We know that there are two ways to process ordersaction：Place an orderandCancel the order。 The business logic of cancellation is very simple. It is to query whether the order exists in the transaction entrustment account book. If it exists, delete the order from the entrustment account book, and then output the cancellation result of successful cancellation; if not, output the cancellation result of failed cancellation. The business logic of placing an order is more complex, and it needs to be handled differently according to different order types. There are six different versions of matchmaking program when writing this articletype, including two typesFixed priceTypes and fourmarket priceType. Next, we will explain the results of different order types under different conditions.
- limit: regular price limit. When there is a delegation order matching the order in the delegation account book, one or more transaction records may be generated, and each transaction record will generate asynchronous output; when there is no matching delegation order in the delegation account book, the order (total quantity or remaining quantity) will be added to the delegation account book, and no output will be generated at this time.
- limit-ioc: IOC limit – immediate closing and cancellation of balance. When there is a delegation order matching the order in the delegation account book, one or more transaction records may be generated, and each transaction record will generate asynchronous output; when there is no matching delegation order in the delegation account book, the order (all or remaining quantity) will be cancelled, and a successful output of cancellation will be generated.
- market: default market price – immediate closing residual cancellation. Like IOC price limit, when there is a commission order in the order queue (also known as the counterparty) in the opposite direction of the order in the entrustment account book, one or more transaction records may be generated, and each transaction record will generate asynchronous output; when the counterparty in the entrustment account book does not have a commission order, the order (total or remaining quantity) will be cancelled, and a cancellation will be generated Successful output. The difference between IOC price limit order and IOC price limit order is that the user specifies the commission price, while the market price does not need to specify the commission price, and it will directly deal with the head order of the counterparty until the order has been completed or the counterparty no longer has a commission order.
- market-top5: market price – the best five real-time transaction residual cancellation. Market can deal with orders of all price ranges of the counterparty, but market-top5 can only deal with orders within five price ranges of the counterparty at most, and orders beyond five ranges will not. All the remaining unsettled orders will be cancelled and a successful cancellation output will be generated.
- market-top10: market price – the best ten real-time transaction residual cancellation. At most, only orders within 10 price ranges of the counterparty will be closed.
- market-opponent: market price – the best price of the counterparty. If the counterparty does not have an order, the order will be cancelled directly and a successful output will be generated; if the counterparty has an order, only one transaction will be completed at most; if there is still an outstanding volume, the price of the counterparty will be converted into a price limit order and added to the entrusted ledger, and no output will be generated at this time.
It can be expressed as follows:
In addition, every request to process an order, whether an order is placed or cancelled, will also be cached in redis, and the cache will be updated when changes are made. In this way, the order can be resumed after the program is restarted.
The engine needs to be shut down when the subject matter of a transaction is ready to be removed from the market, or when the transaction is cancelled, or when the transaction is suspended. Before shutting down the engine, it’s better for the upstream service to stop calling the interface to process the order, or some unexpected errors may occur, although the program has done fault tolerance processing.
When shutting down the engine, there are also some simple judgments, such as judging whether the target engine of the transaction has been turned on, and the engine that has not been turned on cannot be turned off naturally.
When shutting down the engine, if there are still unprocessed orders in the sequencing queue, you should wait for these orders to finish processing before shutting down the engine.
Finally, you also need to clear the cache to clear all orders of the transaction target from the cache.
The result of shutting down the engine is also returned synchronously, and there is no asynchronous output.
The flow chart is also relatively simple:
This section explains the core business process of matching black box, including the internal logic of opening matching, processing order and closing matching. After understanding these processes, we will start to talk about code implementation in the next chapter.
The Convention leaves a few questions to ponder: if there are concurrent requests for orders while closing matching, is it easy to generate problems? If so, where? What’s the problem? How can it be solved?
Author’s personal blog
Scan the following two-dimensional code to pay attention to the public number (public name: Keegan steel).