Welcome to “Keegan small steel” public address for more articles.
Matchmaking engine development: the beginning
Match engine development: MVP version
Match engine development: data structure design
Match engine development: docking black box
Match engine development: decryption black box process
Match engine development: code implementation of process
After the opening article, my matchmaking engine was positioned as a toy by a super Tycoon (once the chief architect of Shanghai Stock Exchange), directly comparing my matchmaking engine with the national matchmaking engine. If my matchmaking engine reaches the level of Shanghai Stock Exchange, it’s worth more than 50000 yuan. It’s estimated to be worth at least 5 million yuan. However, with the continuous upgrading and iteration of my matchmaking engine, it may reach the national level in the future. In order to avoid this embarrassment again, I would like to clarify the positioning of this matchmaking engine.
MVP version requirements
Lean entrepreneurship has a core concept calledMVP(minimum visible product), i.eMinimum feasible product。 The first version of my matchmaking engine is also an MVP, which only realizes the most basic functions. The so-called most basic function, that is to say, if the function is removed, the whole system cannot complete the work. Of course, we also need to add application scenarios, which are applied to a small start-up trading platform and a huge trading platform such as Huo coin, Qian’an and even Shenzhen Stock Exchange and Shanghai Stock Exchange. The definition range of basic functions is very different. The MVP version I want to make, as long as it is suitable for small trading platform.
Here I’d like to talk about product design. Many teams, especially start-up teams, always think this function is very important and that function is very important when they make the first version of the product. They all add it to the first version of the product. In fact, when we do the first edition, we should do subtraction more than addition. A lot of functions that seem to be very important, most of them belong to the kind with it better, but without it, the whole product will not run.
Let’s discuss the specific functions of the matching engine of the MVP version.
We know that there arecall auctionandContinuous auctionTwo ways, but for our MVP version, is it necessary to support both ways? In fact, in the currency circle, whether it is a small exchange or a large exchange, only continuous bidding is basically used. The precious metal trading platform I used to work on also didn’t call for bids. This also shows that call auction is not necessary for an exchange. In this case, the first version of the matchmaking engine can actually cut off the call auction function first.
It is necessary to support order placement and cancellation. This is the most basic function of an exchange. Without these two functions, an exchange is meaningless. There are generally several types of delegation orders, includingFixed price、market price、Stop loss and stop lossWait, the simplest is price limit, which is also the transaction type that all exchanges must support. Start-up exchanges generally only support price limit transactions first, so our MVP version only supports price limit transactions first.
The results of order placement and cancellation also need to be sent by means of events. Other services will listen to these events and do corresponding subsequent processing.
MaintainOrder bookMatching is also necessary. Matching is to match and close the order in the orderbook. Those not closed temporarily will be saved in the orderbook.
In addition, we also need to adoptMemory matchTechnology, so orderbook is actually stored directly in the program’s memory. Then, if the program exits abnormally, the saved data will be cleared. Therefore, we also need to introduce cache to back up data. When the program is restarted, the data can be reloaded from the cache.
MVP version also supports the matching of multiple transaction targets, because our MVP version matching engine is only a stand-alone version of the program, it can not only support one transaction target.
It also supports the function of opening and closing the matching of specified transaction targets. When opening the matching, some initialization operations need to be done, including initializing the opening price. After closing the matching, data will be deleted and resources will be released.
In summary, our MVP version should realize the following functions:
- Support matching method of continuous bidding;
- Support limited price transaction and cancellation;
- Support the distribution of order and cancellation results;
- Adopt memory matching technology to maintain the transaction entrustment account book in memory;
- Need to cache data, when the program is restarted, data can be recovered;
- Support the matching of multiple transaction targets;
- Support to turn on and off the matching function of specified transaction target.
The requirements are determined, and then the technical scheme is to be determined. Let’s talk about some technical selection first.
The first is to develop the language. My choice isGolang, the reason is very simple. Golang is close to the execution performance of C / C + +, but it has higher development efficiency than C / C + +. It can not only meet the performance requirements of matchmaking engine, but also meet the requirements of our rapid product realization. Of course, it can be realized in other languages. After all, the design idea is universal.
Placing and cancelling orders, opening and closing matchmaking, and the distribution of results all involve communication with other services. There are two main options for communication between services: synchronous callingRPCAnd asynchronously calledMQ。 Synchronous call can make the request get immediate response, the communication is relatively efficient and reliable, but it is only suitable for one-to-one communication, and if the concurrent request is overloaded, it may cause a lot of request timeout or even service downtime. MQ supports one to many communication, and because of the cache queue, it can avoid the situation that the service is unavailable when the concurrent request reaches the peak, but also because of the multiple message middleware, the transmission is delayed, and the request cannot get the immediate response, and there is the possibility of losing the message, so the reliability is less than the synchronous RPC mode.
For our application scenario, MQ is the only way to distribute results, because we don’t know how many downstream services will consume our result messages, nor can we require downstream services to provide a unified RPC interface for us to call. For order and cancellation requests, it is better to use RPC synchronous method. First, it can respond to some invalid requests immediately. Second, it can reduce MQ transmission delay. Third, it can ensure reliability. For the problem of concurrent request overload, we should do a good job of load balancing in the higher level gateway service, instead of leaving it to the matching engine.
However, RPC and MQ also have a variety of specific implementation schemes. RPC schemes haveREST、gRPC、Thrift、rpcxWait, MQ scheme hasKafka、RocketMQ、RabbitMQ、RedisAnd so on. I will not expand the differences between these different specific programs. Interested readers can go to Baidu or Google by themselves. We can choose the simplest rest for RPC scheme, which is convenient for development, docking and testing. In MQ scheme, redis is selected because redis has been introduced since version 5.0StreamThe data structure provides the message queuing function similar to Kafka, but because redis data is stored in memory, its processing speed is much faster than other MQ. In addition, we also need to use redis for caching, which is more convenient to use the same middleware.
The above figure is the software structure design diagram of our MVP matching service. It’s very simple. In fact, it is just layered according to the business process. Layering is the simplest way of architecture. In fact, any huge and complex system can be layered according to business process in the first step of disassembly.
HandlerTo receive HTTP requests from upstream services, we only need to define three interfaces:
- OpenMatching: to enable matching, you only need to receive two parameters: the target transaction (transaction pair) and the opening price.
- CloseMatching: to close the match, only one parameter is required: the target of the transaction (transaction pair).
- HandleOrder: there will be a parameter when receiving a delegationActionIndicates whether the action is to place an order or cancel an order. Other parameters are the data of the order, including order ID, transaction pair, trading direction, delegated quantity, delegated price, etc.
After the handler performs some routine verification on the request, it will be forwarded to the corresponding process for processing. We also define three correspondingProcess：
- NewEngine: create a new orchestration / thread as the matchmaking engine for the specified transaction pair.
- CloseEngine: turn off the matching engine of the specified transaction pair.
- Dispatch: distribute the orders of different transaction pairs to the corresponding matching engine.
EngineIt is the matching engine process or thread of each transaction pair. Since the matching engine of each transaction pair must process the order in a serial way, it is necessary to first throw the order into the orderly queue of different transaction pairs when dispatching, and then the engine will take the order out of their respective queues for matching.
RedisIt is used for both data caching and message queuing. The cached data mainly refers to the transaction pairs that have been matched and the transaction entrustment ledger in the matching engine. In order to ensure the consistency of data, every change of each order in the ledger needs to be updated to redis. When the matching engine is restarted, it is necessary to read the cached order from redis and reinitialize the transaction order book. In this way, the program can be restarted after exiting and can be restored to the state before exiting. Message queuing can be implemented with redis’s new data structure stream to send transaction records and cancellation results.
In the first version, we first make an MVP, a single version of matching service, which supports continuous bidding, price limit Commission, cancellation, opening and closing of matching, multiple transaction peer-to-peer functions, and adopts memory matching technology. The software structure is mainly divided into three levels: handler, process and engine. Redis is used as data cache and message queue in the bottom layer. Next let’s design the data structure.
Leave a question to think about: how can an ordered queue be implemented when dispatching a delegation to the engine?
Author’s personal blog
Scan the following two-dimensional code to pay attention to the public number (public name: Keegan steel).
Author’s personal blog