The practice of tidb in linkdoc big data medical system

Time:2020-5-23

Company introduction

As the world’s leading AI and medical big data platform, zero krypton technology has the largest medical big data resource base and the most advantageous technical support service system in China. Over the years, zero krypton technology, relying on its core technology advantages in the integration, processing and analysis of medical big data and advanced artificial intelligence technology, has been committed to providing high-quality overall solutions for medical big data for society, industry, government departments, medical institutions at all levels, domestic and foreign medical device manufacturers, pharmaceutical enterprises, etc., as well as artificial intelligence assisted decision-making system (assisted management decision-making system , help clinical scientific research, AI intelligent diagnosis and treatment), patient whole process management, hospital public opinion monitoring and brand building, research and development of medical equipment, insurance control fee and other integrated services.

Main application scenarios of linkdoc

Linkdoc applies the real case data and algorithm model of patients to cancer treatment, builds accurate diagnosis and treatment model and provides data support, so as to assist hospital management decision-making, scientific research and clinical diagnosis and treatment. At present, Hubble system “risk prediction of leaping lymph node metastasis of lung cancer” module can avoid the recurrence of lung cancer patients 8-10 months in advance due to misjudgment, and can prolong the life of nearly 20000 patients by 8-10 months every year. The “Ai – pulmonary nodule intelligent diagnosis” module of Hubble system can automatically recognize all the nodules in CT images, and the recognition rate is 91.5%. Linkdoc hopes to improve the global medical standard of Chinese doctors by virtue of the core technical advantages in the integration, processing and analysis of medical big data and the innovative research and development of Internet artificial intelligence, and to enable every patient to enjoy inclusive and accurate medical services by supporting the development of drug research and development and medical insurance industry.

The underlying database platform supporting linkdoc business is also facing new technologies in the medical industry& Business challenges, such as the rapid growth of data volume (billion level), data erasure of cleaning logic under large data volume, and the reading pressure of analytical things on database require us to re explore the database platform and choose a database solution suitable for medical big data business.

Select tidb

  1. Business pain points
  • The data volume is large, and the expansion operation of single instance MySQL is complex;
  • The traditional MySQL master-slave architecture can not meet the needs of the project because of the high write volume and high master-slave delay. The main database becomes a performance bottleneck when a large amount of data is written;
  • With the increasing amount of data, the speed of some statistical queries is slow;
  • The development and maintenance cost of sub database and sub table business is high.
  1. demand
  • High reliability & stability;
  • Scalability, which can rapidly improve the storage request processing capacity with the growth of data request quantity;
  • Lower latency.
  1. Scheme Research

Before tidb was chosen, we investigated the middleware solutions such as MYCAT, Cobar and atlas. These middleware as a whole make users feel “screwed up”. They are not satisfactory in terms of community support, MySQL function compatibility and system stability. They need a lot of business transformation, and the switching cost is too high for fast-growing companies.

We investigated tidb with the recommendation of Wang Xiaozhe, chief architect of linkdoc. The following features of tidb brought us a bright future:

  • It is compatible with most SQL functions (which means that the business can be smoothly migrated to tidb after simple transformation);
  • Horizontal expansion capability;
  • Distributed transaction;
  • Fast fault recovery capability;
  • Monitoring index coverage.

Online tidb

  1. Compatibility test

After the compatibility test, we have made the following simple transformation to the business:

  • Blob type data is migrated to HBase for key value storage;
  • Batch delete is changed to small batch and multiple operations, and 1000 items are deleted in one batch.
  1. Gray line

Due to the high requirement of master-slave synchronization delay, we switched our first application with the scheme of service double write. In the first phase of grayscale business, MySQL and tidb are written at the same time, MySQL is read, and data consistency is verified. After two weeks of verification, we have the second phase of grayscale business. In the second phase of grayscale, the service writes tidb and MYSQL, and the read service uses tidb. After a month of business validation, we dropped MySQL completely.

  1. system architecture

During the online process, we also encountered a small pit. The previous Alibaba cloud ordinary instance + SSD cloud disk ran tidb, which often encountered performance jitter problems under this configuration. At the suggestion of pingcap, we changed the Alibaba cloud local SSD model, and the system is running well at present.

The system configuration & Architecture is as follows:

Production cluster deployment (the machine is based on Alibaba cloud):

Current situation and next planning

At present, tidb has carried two services with the largest data volume in linkdoc. At ordinary times, QPS is 6K and the peak value is 12K.

In the future, tidb will be used to host more large data volume business libraries and research tispark. Through tidb, a unified database HTAP platform compatible with analysis and transaction is constructed.

To pingcap

Thank you very much for pingcap’s strong support. From hardware selection, business optimization, system training to online support, pingcap has demonstrated a warm service attitude and professional technical ability to help linkdoc successfully launch tidb, solve system problems and support rapid business development. I believe that with the efforts of such a group of small partners, tidb will become more and more mature, carry more business scenarios, and use technology to create miracles.

The author introduces: Yang Hao is now the operation and maintenance & safety director of zero krypton technology, and once worked in Alibaba technology security department CDN. Focus on CDN, security, automatic operation and maintenance, big data and other fields.