360 x tidb improves performance by 10 times, how can 360 easily resist double 11 traffic

Time:2021-6-29

“We’ve used it.”, is our favorite words, a few words behind represents the heavy trust and trust. Starting today, we will pass the“Believe in the power of openness”A series of in-depth case sharing, from the perspective of business, to see the business value of a database for users in various industries.This article will introduce the application and practice of tidb in 360 net shield business, smart business, advertising material data business and other core scenarios.

360 x tidb improves performance by 10 times, how can 360 easily resist double 11 traffic

Driven by science and technology, make the world safer and better

Founded in 2005, 360 company is a leading Internet and security service provider in China. It has launched 360 security guard, 360 mobile phone guard, 360 security browser and other security products.

With the deepening of the digitalization of the whole society and the whole industry, the era of “big security” is accelerating. With the mission of “making the world safer and better”, 360 is committed to realizing the vision of “continuously creating black technology and being an all-round Guardian”. The following business development is more comprehensive, including: government enterprise service, financial technology, live broadcast, personal service Smart wear and other services.

Business challenges

Netshield service

360 net shield is a free online protection software, which can intercept Trojans, fraudulent websites and so on, and protect consumers from the fraud of viruses and false websites. Its operation mechanism is to analyze each URL, and after the identification of the URL detection system, it can accurately identify which category the URL belongs to. When the URL with problems is published to the cloud, other security service providers can subscribe to the cloud service to detect whether the relevant URL is secure, so as to provide their own users with more secure web content services.

At present, the core scenario of network shield service can be divided into the following four parts:

  • Website threat monitoring: 100 million data and 800 million resource link data are stored every day;
  • Association analysis scenario: conduct association analysis of large-scale malicious websites, pornography, gambling, drugs and other websites;
  • High speed return: 89.7 billion data tables, 100 + queries for each scene need to be returned in 5 seconds;
  • Manual operation analysis: every day everyone repeatedly query statistics and analysis.

In view of the explosive growth of business data, MySQL has appeared a bottleneck in reading and writing. For example, disk space can be split by database and table, but it is not a small workload for business and DBA; The most painful is just like the modification of these large tables. The cost of implementing DDL on a large table is very large. In general, MySQL has been unable to meet the needs of netshield business, which puts forward new selection requirements for the 360 cloud platform technical team responsible for supporting the underlying data platform.

360 cloud platform is responsible for providing service support to major business lines of 360 group. The database support schemes involved include mysql, redis, mongodb, elasticsearch, Greenplum and pika. After sufficient market research, the 360 cloud platform team decided to introduce tidb to meet the business demand.

360 x tidb improves performance by 10 times, how can 360 easily resist double 11 traffic

The overall architecture is as above. There are two main services using tidb:

  • Migration of original MySQL service. Due to the limitation of single instance disk, single instance disk cannot support the explosive growth of data volume. Data is more important and needs to be backed up and supported for 7 * 24 hours of recovery. This kind of business realizes barrier free migration through DM suite. The import time of 1TB is 16 hours. If it is a relatively large data source and tidb is a new cluster, tidb lightning can be used for data import, and the speed can reach 100g / hour.
  • New business. At present, all the new services will be put into tidb, and the data volume of this service is generally large. At present, there are many tables of netshield service that are over 1 billion, and one of them has reached 10 billion +.

Smart business

360 smart business relies on Internet products covering all scenarios of users to provide enterprises with full life cycle services. Through multiple business layouts such as intelligent marketing, enterprise service and innovation platform, we can meet the multi-dimensional growth needs, fully connect users and enterprises, and create a symbiotic and growing smart business ecology. Internet advertising is one of the important ways to realize the commercial flow, and is also an important revenue source of 360 group, which involves enterprise service platform, advertiser delivery, algorithm strategy Data engineering and other directions. In the process of advertising, real-time / offline report business and advertising material delivery are the most important and core business for advertisers.

Advertiser real time & offline report service

Advertiser’s real-time report business process: business data enters Kafka. One processing link obtains Kafka’s data through druid to provide real-time analysis. The other link aggregates through Flink and writes it into MySQL sub database and sub table database, and then provides real-time query requirements through advertiser dimension.

Advertiser’s offline report business process: every morning, the data warehouse extracts all the data of the previous day from the MySQL sub database sub table into hive, and writes the result data into the MySQL result table after statistical aggregation by spark or business program, which is provided to the offline report platform or console platform for query.

As the core business of the advertising platform, reporting faces the following problems:

  • Large amount of data: the total amount of data is 100 billion, and the amount of single table data is 120-150 million.
  • Low response delay: it needs real-time feedback to the user’s query of any period and keywords.
  • Complex query: time dimension, region, industry, keywords and so on, while meeting the diversified display.
  • The structure is complex: MySQL based sub database and sub table can not perform global statistics, but can only generate detailed reports based on advertiser uid, and Druid should be introduced to assist global statistics; Offline report needs hive data warehouse to extract full amount of data.

Database selection: MySQL or tidb?

360 x tidb improves performance by 10 times, how can 360 easily resist double 11 traffic

Before the deployment of tidb, 360 had tried single instance Mysql to meet business requirements. After the test, it was found that single instance MySQL was under great pressure. In order to disperse the writing pressure, it was necessary to follow the old path of MySQL database and table splitting. In addition, for database and table splitting with large amount of data, splitting rules often need to be changed, and each rule change may involve the relocation of data, And the business side also needs to invest a lot of manpower to maintain the routing rules, and to meet the advertiser’s report needs, it needs to introduce other databases. The extraction of MySQL by offline ETL in the early morning of every day causes the network card full load, which also affects other business operations in the early morning.

360 x tidb improves performance by 10 times, how can 360 easily resist double 11 traffic

After the deployment of tidb, the good scalability of tidb completely solves the problem of sub database and sub table. At the same time, through the performance pressure test, the whole system load fully meets the business requirements with 150 million data storage in 2 hours (TPS: 2W / s). By combining with tiflash (the real-time analysis engine plug-in of tidb), we can carry out real-time analysis of various dimensions and details of the merged single table, In addition, it realizes the online statistics of offline reports, eliminates the T + 1 timeliness and synchronization process of offline data warehouse, and provides strong consistency guarantee of financial level.

Advertising material data business

For advertisers, in search promotion, based on the safe, accurate and reliable new generation search engine 360 search, through keyword technology matching, target netizens are located, and enterprise promotion information is accurately displayed. The role of material creativity is to help advertisers attract potential customers, and then generate transformation behaviors, such as registration, online order submission, telephone consultation, and so on Door to door visits and so on.

At present, the material platform of 360 advertising will carry the information of pictures, texts, videos, etc. produced by customers, and support the upload, download, import, export, add, edit, delete and other operations of materials at all levels of promotion account, promotion plan, promotion group, keyword, promotion idea and advanced style.

Before using tidb, the bottom layer of the focus material data used the MySQL architecture of 16 sub databases * 4. Each set of sub database mysql single table has reached 1 billion, and the single table data scale is 370g. For SQL requests with single database QPS over 10000, the MySQL table has reached the performance bottleneck, and the peak jitters frequently. In addition, new businesses want to add new fields to large tables, because of the lack of hard disk space, It also can’t support new business online. If you continue to use mysql, you need to split the current 16 sub databases into 64 sub databases (about hundreds of new servers are needed). In addition to new servers, the cost of migration and operation and maintenance is also very high.

The technical team of 360 commercialized business line finally determined to use tidb as the underlying database of the material platform through technical selection research.At present, the tidb cluster supporting the material platform has 63 nodes, with more than 7 billion SQL requests per day. In the past double 11 QPS, the maximum is 25 W / s, and 99% of the SQL requests in working days are less than 15 ms. the fast response, stability and scalability all meet the expectations.

The benefits of tidb deployment are as follows:

  • Cost saving
  1. Compared with the previous MySQL deployment mode, it saves 40% of the server cost.
  2. It used to be a routing rule of sub database and sub table for business maintenance, but now it is a table for business, which improves the efficiency of business development and makes R & D pay more attention to business.
  3. Operation and maintenance costs are reduced. Tidb provides a rich tool chain ecology, covering data migration, synchronization, backup and other scenarios, which improves operation and maintenance efficiency.
  • Performance improvement
  1. 618, double eleven QPS up to 25 W / s, 99% of the working day SQL requests are in 15 ms.
  2. Compared with MySQL master-slave architecture based on VIP switching, the availability of tidb is more than 99.95%.
  3. Pure distributed architecture, with good scalability, supports flexible expansion and reduction, throughput and storage can be expanded online smoothly, and the database expansion ability has been improved by 1-2 data levels.

In the future, the 360 technical team also plans to push tidb onto 360 Hulk cloud platform to provide stable and reliable database services for more business lines such as 360 group security brain, 360 games and core security service platform.

Go with customers and believe in the power of openness

Every time the database architecture is improved and implemented, whether it is TB level or Pb level, efforts need to be made, but it is also worth every enterprise to practice. In this era, no matter what the scale of the enterprise is, we should learn to use the power of open source to avoid making wheels repeatedly.
Behind each seemingly easy, there are unknown efforts, behind each seemingly bright, there are unknown efforts. Tidb is willing to work hand in hand with 360 and every customer to do a good job.