Business scenario actual combat (I) evolution of meituan home commodity inventory

Time:2022-5-16

catalogue

  • General catalogue of series
  • Architecture evolution
    • Anti oversold
    • Unitization
    • Platformization
  • Anti oversold
    • MySQL synchronization to prevent oversold
    • Redis synchronization to prevent oversold
  • uniformity
    • MySQL synchronization to achieve consistency against oversold
    • Redis synchronization realizes the consistency of anti oversold
  • Reference articles

General catalogue of series


Architecture evolution

Business scenario actual combat (I) evolution of meituan home commodity inventory

Architecture evolution diagram png

Anti oversold

  • At first, the inventory of goods only needs to support lbs e-commerce, such as external and flash purchase. Because this kind of e-commerce is a location-based service, the category is limited and the inventory is limited. To prevent oversold, you can use Mysql to synchronously deduct the inventory. This business scenario is suitable for rapid iteration in the early stage of business development.

Unitization

  • With the development of business, it is necessary to support higher QPS, better expansibility and stronger disaster recovery capability. To resist the wandering flood peak, massive inventory deduction, 10000 TPS. Native MySQL cannot be supported. Here, the scheme of redis synchronous deduction and monitoring AOF synchronous MySQL is selected. For a long time, mtsql (meituan self-developed SQL + corresponding degradation solution) is adopted for self-development. The overall architecture is as follows. Cell is kV storage. In case of disaster recovery, it is multi machine room construction and multi IDC deployment.

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    Unitization png

Platformization

  • It takes half a month for the application platform to respond quickly, and it takes only half a month for the application platform to change its application side. Small businesses reuse large business processes, and large businesses are compatible with small business processes to avoid various if else.
  • Platformization should ensure that only its own line of business is returned, and the code is concise without redundancy. This is the real platformization.
  • Based on the overall platform architecture, we can see that each business side has its own redis MySQL elasticsearch. In the long run, it should be a micro kernel pluggable service similar to skywalking. The platform provides SPI, which is implemented on the business side. Each business side is independent, so there will be no need for simple field modification on the business side. The platform only responds in half a month.
  • Referring to the article [vivo comments on the traffic and data isolation practice of the middle station], it is pointed out that the general es redis MySQL can be set to adapt to the business with relatively small traffic, and isolate the ES redis Mysql to the business with relatively large traffic or special needs. Make a compromise.

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    Platform png

Anti oversold

MySQL synchronization to prevent oversold

  • This scheme supports limited TPS and limited performance, which is suitable for the architecture when the initial QPS is not high.

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    MySQL synchronization to prevent oversold png
  1. The inventory transaction service uses Mysql to synchronously deduct inventory. DTS (open source can use canal) monitors incremental binlog. After listening to binlog, the inventory asynchronous service records inventory flow.
  2. The inventory asynchronous service will verify the flow. If the verification fails, it will compensate. Of course, if the inventory transaction service is abnormal, it will also send a compensation message, and the inventory asynchronous service will compensate.
  3. For MySQL solutions, there will be one case of pipeline verification failure: the inconsistency caused by the timeout of the upstream socket calling the inventory service, and the distributed things are useless.

Redis synchronization to prevent oversold

  • For the city wide delivery, the products with rich categories, rich inventory and high QPS can be deducted synchronously by redis.
  • The architecture is as follows:

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    Redis synchronizes to prevent oversold png
  1. The inventory transaction service synchronizes the Lua script to redis, and then the deduction is successfully sent to the flow MQ. If an exception is submitted, the compensation MQ is sent.
  2. The inventory flow service monitors the AOF inventory flow of redis, and then aggregates it to MySQL in batches.

uniformity

MySQL synchronization to achieve consistency against oversold

  • framework

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    MySQL synchronization realizes the consistency of anti oversold png
  1. If the binlog backlog or DTS failure causes redis, the inventory asynchronous service data is old, which can be solved by delaying processing. DTS can be guaranteed through two channels (as shown in the figure, one main channel of DTS and the standby channel of meituan BCP).
  2. If the data of the inventory asynchronous service is lost in redis, it will be reloaded. The inventory asynchronous service has a verification thread.

Redis synchronization realizes the consistency of anti oversold

  • framework

    Business scenario actual combat (I) evolution of meituan home commodity inventory

    Redis synchronizes to realize the consistency of anti oversold png
  1. The master and slave of redis cluster may be inconsistent. For example, after the master is updated, the AOF of the master has not yet been synchronized to the slave, the Master goes down, and the slave is upgraded to master. It can be processed through dual channel flow verification and inventory correction. Of course, in the long run, the best way is to reconstruct the redis raft protocol, but the cost is relatively large.
  2. Redis an IDC cluster goes down completely. At this time, the trigger inventory does not exist, dual channel verification, and reload correction does not exist.

Reference articles