Summary of activities
Influenced by the trend of retail innovation, when the annual 618 is approaching, how can we provide users with a stable and smooth shopping process in high-frequency, high-volume and high-density trading scenarios? How to create brand and partner infrastructure service providers through technological innovation?
In the recently concluded Beijing-East Cloud Technology Salon Shanghai Station event, many Beijing-East technology cafes and developers face-to-face on the retail industry and social e-commerce core technology and successful solutions.
There are plenty of seats at the salon.
More than a hundred developers on the site participated enthusiastically in the exchange and interaction, especially in many technical fields such as database services under the “big promotion”, design and practice of enterprise-level monitoring, various aspects of social e-commerce and intelligent traceability of block chains.
We have sorted out and summarized some contents of salon and put them into this article, hoping to provide reference for relevant practitioners. (Public number reply “PPT0526” can obtain salon video & PPT)
How does the Jingdong database service respond to the “big promotion activities”?
Guo Lijing, Senior Director of Jingdong Cloud Product Research and Development Department
As we all know, there are many businesses in Jingdong Mall. Many businesses don’t want to choose too many kinds of databases in the initial stage of R&D and design. Because they don’t know the size of the business, it’s simplest to start using MySQL only. Considering the speed of R&D, sometimes they choose to write logs and large field data. With the increase of data accumulation and the expansion of user scale, data switching will be involved in the future. Therefore, some data architecture optimization work should be done for these “later-in-the-world” businesses.
Beijing Dongyun database has some flexible database services, which can configure multiple databases on one physical server at the same time, and raise the utilization rate of single machine to a higher level. At the same time, the flexible scheduling system can complete the data migration between two databases with little pressure. Help customers to control cost effectively. In addition, on the basis of using MySQL, performance is inevitably limited to machines, and distributed database services such as Jproxy are also used in Jingdong Mall. Guo Lijing said that in the practice of Jingdong Mall, two computer rooms were selected for the online system, namely, the production environment and the disaster preparedness environment.
With regard to intelligent analysis, in the process of assisting e-commerce, Jingdong Yun started SQL log analysis and capacity analysis on the one hand, such as whether the number of connections has been growing, IO pressure, CP pressure, etc. At the same time, it also tried to do some fault self-healing and disk cleaning.
In terms of “Jingdong Cloud Service Practice”, we provide rich database services in the cloud, such as MySQL, Percona, MariaDB, SQL Server, MongoDB and so on. From the statistical point of view of Jingdong Cloud, domestic developers can use MySQL up to 95%, and the other 5% will use Percona or MarinDB.
According to overseas survey data, more than 80% of overseas database users are using MySQL, while less than 20% of users will choose MariaDB and Perona. The reason why there is such a big gap may be the obvious difference between the official publicity efforts at home and abroad. In addition, Jingdong cloud also provides a wealth of functions in security, auditing, logging and many other aspects, as well as Binlog backup and download.
In terms of monitoring engine, Jingdong Yun’s attempt is also more meticulous, including monitoring services, alarm services and so on. It is worth mentioning that all monitoring items can set rules, such as more than 70% CPU memory, can set up a SMS alarm, and all of the indicators are connected to the cloud monitoring system. “We also support the creation of read-only instances. We need to create 8 read-only instances with maximum read-only support. Each read-only instance is assigned a separate domain name, which can be accessed by the domain name.” He added.
Talking about data operation and maintenance, how to achieve automatic fault switching? Beijing Dongyun has designed a relatively perfect automation system for this purpose. At first, it is manual handover. The main reason for choosing such handover mode is to worry about the occurrence of mistake. After all, many services link to the database through domain names, and the handover of domain names will cause some delays, which is unacceptable for the service. Now database failures have evolved to automatic switching.
As for data security, cloud hard disk encryption is used. As long as the data encryption function is activated, cloud RDS can fully enjoy its security services. For which IP can access a specific database, the whitelist can be set flexibly.
Emphasis is laid on the point that DRDS is also involved in the Beijing-East Cloud Database Service. The so-called DRDS is a distributed relational database. The traceability origin is to transplant the Jproxy technology from the Beijing-East Mall to the Beijing-East Cloud. The main function is to complete the sub-database and sub-tables on the premise that the single computer can not support the large amount of data. It’s more convenient.
DRDS service mainly solves the following difficulties:
1. With the increase of data volume and access volume, stand-alone database will encounter great challenges. Depending on hardware upgrade can not completely solve the problem. That is to say, the bottleneck of stand-alone database capacity needs to be taken seriously urgently.
2. The expansion of traditional database capacity often means service interruption, which makes it difficult to achieve business imperceptibility or less perception. It is considered as the dilemma of expansion.
3. More importantly, the cost of using traditional databases is high. When the amount of business data access increases to a certain extent, traditional databases need to rely on specific high-end storage and minicomputer equipment. For users, the cost is high. But the use of DRDS can help customers to control cost effectively.
Specifically, DRDS services provide middleware, such as for user tables, which can split IDs from user tables. When you need to query users, you don’t need to modify the SQL statements. DRDS parses the statements and discovers the accessed user data tables, user IDs and other information. In fact, DRDS plays a role in parsing the SQL statements, distributing the SQL statements and summarizing the SQL statements to the users, so that users can focus more on their own business logic development. At the same time, it can effectively improve the whole performance.
At the same time, DRDS can be deployed more than one, but it is worth noting that the performance of the whole DRDS system is limited by the number of back-end instances. Simply put, the more instances in the back end, the better the performance of DRDS. Through actual measurement, the delay caused by DRDS middleware to SQL is very low. DRDS optimizes the parsing speed of SQL statements, so the delay caused by middleware can be ignored.
DRDS itself does not store data, which is stored in the back-end RDS MySQL database. In order to ensure the stability of the system, the MySQL version of each sub-library must be consistent. DRDS will create eight sub-libraries on each RDS instance, and a separate sub-library 0 will be created to store non-splitting data.
The maximum processing capacity of the system depends on the number of sub-libraries, so the capacity and capacity of the whole system should be planned after it needs to be created. It is better to use small-sized MySQL instances in the early stage according to the preliminary planning, but enough sub-libraries, so that sufficient capacity can be reserved.
In technology sharing, Guo Lijing also systematically introduced various aspects of capacity expansion, such as two-core 4G RDS, which has relatively small performance. If the back-end RDS is upgraded to a higher configuration, it is called vertical capacity expansion and easy to operate. Whether horizontal or vertical expansion, although the specific operation methods are different, the process is smooth and common at any time and anywhere.
Although DRDS has greatly improved its scalability compared with traditional single-machine databases, it also brings some limitations, such as ensuring transaction consistency in the same sub-library, but not supporting distributed transactions; supporting join operations in a single sub-library only, not cross-library join; supporting UTF8 character set only, because Not all MySQL-based applications can undoubtedly be migrated to DRDS, and some modifications are needed.
Design and Practice of Enterprise Monitoring System
Zheng Yongkuan, Director of Jingdong Cloud Product Research and Development Department
Zheng Yongkuan, Director of Jingdong Cloud Product Research and Development Department, as the second guest to share with the salon, delivered a technical speech on the theme of “Design and Practice of Enterprise Monitoring System” for developers.
What is monitoring? Monitoring is the lifeline of operation and maintenance.
What problems does monitoring solve? It’s far more than finding problems!
Monitoring also needs to solve the problem of location and stop loss after location, so as to reduce the MTTR, that is, the average recovery time of the fault. For large-scale e-commerce like Jingdong, the loss caused by unexplained downtime is very heavy. In order to effectively solve the problem of reducing the recovery time of the fault, it is very effective to make the recovery plan of the corresponding fault in advance before the fault occurs, and to identify the fault type and locate the fault accurately after the fault occurs.
In order to solve the problem of rapid response, the monitoring system usually needs to have a wealth of data acquisition means, real-time aggregation calculation of multi-dimensional data, accurate anomaly detection, timely alarm notification, customizable dashbord, root cause location and assistant decision recommendation, as well as pre-plan platform and other functions to achieve fast through the above functions. Stop Loss, High Availability/Extensibility of Business, etc.
According to the above function positioning, Jingdong Yunnan will provide fault recovery plan recommendation according to the different business characteristics of each system to ensure the efficient operation of the system. For monitoring system, it is very important to know the system state of cyclic iteration, whether it is to find problems, locate problems or solve problems. Cloud monitoring platform can provide reliable and effective feedback of system operation and maintenance based on abundant monitoring means. At the same time, it can recommend various plans based on monitoring data and business analysis to help business recover quickly.
Specifically, we first define a set of unified monitoring standards: monitoring needs to cover the four levels of basic – survival – Performance – business, so as to ensure the comprehensive collection of data, and thus avoid monitoring omissions. That is, basic monitoring covers the basic indicators of machines, such as cpu, memory, hard disk, etc. Survival monitoring focuses on the survivability of machines and service processes; Performance monitoring focuses on performance indicators of external performance of services, such as PV, ping, error codes; Business monitoring is from the user’s perspective, whether the monitoring system provides normal services. 。 Further, according to different alarms, we can locate the corresponding problems, and then supplemented by the corresponding plan platform, thus ensuring the first time fault location and recovery.
Monitoring system needs to ensure that data acquisition means are enriched, computing aggregation is diversified, storage is efficient and data display is effective. Among them, data storage and data visualization display are the most basic requirements of the monitoring system, no matter how large the operation and maintenance system is.
Jingdong Cloud uses a variety of acquisition methods, including Agent acquisition, API push, external detection points. Through the detection points all over the country, we can simulate and initiate access to various protocols, and provide API push interface to support users to send data themselves, which can further display the data users need. In addition, our self-developed TSDB sequential data storage adopts the scheme based on Cassandra + self-developed cache + esmeta. We put the last three hours’data into self-developed cache, which increases the timeliness and reliability of queries. Among them, self-developed cache is based on the implementation of Facebook paper gorilla, pursuing the ultimate compression ratio. By these means, the storage space can be reduced to 1/10, and the query performance can be guaranteed.
In addition, the monitoring CMDB of Jingdong Yun has performed well. First, CMDB can manage resources and services simultaneously, which is the first problem to be solved by monitoring CMDB. On the other hand, when a machine in a large-scale cluster has a problem, it can quickly locate the affected business through JNS name service, and provide business/application/cluster monitoring configuration recommendation based on CMDB. This is monitoring. CMDB highlights the key points.
Zheng Yongkuan concluded that first, monitoring visualization is the key link. Multidimensional aggregation calculation makes the data more meaningful, which is based on years of operation and maintenance experience of Beijing Dongyun. Beijing Dongyun’s monitoring system can locate various causes of failure, such as whether a business has changed, whether it has corresponding alarms, whether the network and other resources are abnormal, which business problems arise, etc. Through the above monitoring, we can know the implementation of a business’s fault recovery operation. This method is very effective for fast fault location and fault source search. Based on the above ability to identify the cause of the failure, Jingdong Cloud has developed a change visualization product by rapid iteration and upgrading. It monitors all the problems that may lead to the failure, from single-task monitoring of online operation and configuration changes to global event monitoring of the platform, and further helps the positioning of Jingdong Cloud Monitoring System. Improvement of problem-solving ability.
After positioning the problem, in order to enable users to receive meaningful alarms and quickly respond to the problem, we analyzed and merged the abnormal problems. Jingdongyun supports the merging of various strategies. It can alarm according to the strategies of fixed receiver alarm, different business scope alarm and fixed time alarm, so as to reduce the alarm storm. In the docking of downstream notification, we have realized many ways of alarm, such as mail, short message, micro-message, telephone and so on.
In the future, Jingdong Cloud will also start to optimize the configuration management function to better support automatic event recovery. “Combined with monitoring CMDB and alarm settings, we can really link the failure and alarm. For the related alarm events, the monitoring system can automatically call the corresponding plans, so as to achieve the goal of self-healing system failure.”
SaaS product technology architecture for social E-commerce
Zhao Jinyong, Director of Social E-Commerce, Beijing Dongyun Application Research and Development Department
From the whole development of e-commerce, it can be summarized as “four modernizations”, which are decentralization, scene, fragmentation and personalization. We find that the cost of traffic is getting higher and higher, and the cost of “leading” customers and the operating cost of brand companies are also keeping high. This is probably the reason why brand companies do not choose other ways of operation themselves. This is the so-called “de-centralization of brand image propaganda”.
The so-called social e-commerce is bound to have a close relationship with “social” and social tools are naturally indispensable. In addition to a variety of push tools, Jingdong cloud social e-commerce has also achieved successes by virtue of its inherent mass of goods and superior supply chain. Based on the ability of Jingdong big data marketing, the precise portraits of mass users, as well as the stable and reliable basic services of commodity marketing, intelligent selection, etc., to build the system.
Jingdong social e-commerce supports three-tier Commission automatic settlement, flexible design of category Commission points, custom membership promotion rules, automatic statistical team sales performance and rich marketing promotion methods.
In terms of technology architecture, Jingdong cloud merchant’s micro-service architecture is database, hard disk and unified operation and maintenance. Among them, public platform, open platform, decoration service, market, data handling, traffic statistics and other systems are based on the micro-service architecture. Business layer provides a unified, atomized API interface, and upper business is built on these interfaces.
Jingdong social e-commerce is divided into three parts: buyer end, supplier end and PC end. The system of social e-commerce has been connected to the huge commodity pool in Jingdong. Everyone can choose their own products in the whole commodity pool and build a common benign ecology.
In addition, we also provide Magpie Bridge plug-in based on Beijing-East big data, Magpie Bridge connects people and goods to complete accurate marketing. After the completion of the transaction, all the transaction data and the source data as well as the source behavior data are aggregated. The data platform of Jingdong University carries out real-time monitoring and analysis of all the data. On this basis, it highlights advertising optimization, customer clustering, thousands of people, thousands of faces and automated marketing based on data-driven.
“The whole system is based on Beijing-East cloud technology. As a window of Beijing-East’s external empowerment, Beijing-East cloud provides some reliable basic services, including technical services and system support. This part mainly involves e-commerce cloud.”
For example, Beijing Netwarehouse Information Technology Co., Ltd. is a domestic B2B e-commerce service company based on single-product supply chain. Its mission is to build a whole-industry genuine purchasing platform for small and medium-sized enterprises. It pays attention to the application of online supply chain for superior items of enterprises. Its services include online purchasing, online wholesale distribution and online distribution. Customization, etc.
Since its establishment, the company’s service platform has accumulated more than 18 million user data for SMEs, registered nearly 2 million members of its B2B platform for SMEs, provided basic B2B e-commerce application services for more than 300,000 SMEs nationwide, and constructed single-product e-commerce with more than 2400 enterprises with single-product advantages. Trading platform, in this process, has carried out close cooperation with Jingdong social e-commerce.
Technical Practice of JD Chain Based on Enterprise Block Chain Framework System
Zhang Zuoyi, Director of Product Innovation in Jingdong District Chain
The brand name of Jingdong Block Chain is Jingdong Zhizhen Chain. The architecture of Jingdong Block Chain is divided into two parts: JD Chain and JD BaaS (Blockchain as a Service). As the core engine, JD Chain introduces new research results and engineering architecture in data account book, consensus protocol, cryptographic algorithm and storage to solve processing performance and extension. Foundation and key technical issues such as scalability, scalability, security, etc. Create innovative technical architecture and application solutions. JD BaaS is a service platform, which provides flexible, easy-to-use and scalable block chain system management capabilities. It seamlessly integrates the deployment and management of multiple block chain systems, including JD Chain and Fabric. It provides rapid deployment capabilities of public, private and hybrid cloud environments for enterprise users, reduces the cost of enterprise use and speeds up the application landing.
Speaking of service layer, we pay more attention to identity, authority, resource scheduling management, certificate management, version control, log records and so on. “In this year’s”White Paper on the Practice of Block Chain Technology in the East of Beijing 2019”, information of five categories and 14 application cases are listed.
If BaaS can be considered as the “chassis” of automotive engine, other parts can be configured around the chassis with the chassis. JD Chain is the “engine” inside, which is suitable for different scenarios of block chain application scenarios. BaaS helps companies or developers take the “free ride” of block chains, and Jingdong also integrates many application services based on BaaS platform.
“JD Baas and JD Chain are the two major technical architecture systems of Jingdong block chain. Based on block chains to match a variety of application scenarios, the theme is how to use block chains to create real value, which can also be called sustainable value.
The important feature of Jingdong JD BaaS platform is identity authentication service, which is also the difference between block chain and alliance chain. Alliance chain is a licensing chain. Beijing-East identity authentication services, such as identity registration, enterprise license audit and a series of content audit, are added to the block chain to ensure the legitimacy, relevance and security of the main bodies of the alliance chain.
In this process, the identity authentication information of both enterprises and individuals will be stored in the block chain, making a synergy between the nodes of the block chain, including external display, is based on a series of relationships of identity authentication, which will be very different from the relationship without authentication, and it is also a promising direction in the future.
JD Chain, the bottom framework of Jingdong independent intellectual property block chain, officially declared open source in March 2019. It meets the definition and requirements of storage and computation in classical block chains in terms of “ten-thousand-level transaction processing, second-level transaction confirmation, massive accounts and transactions, dynamic scaling, account storage, multi-core parallelism and cryptographic algorithm”. The Key Value structure is also used to maximize processing efficiency with existing recycles, which is why higher speeds can be supported.
Retail, logistics, digital technology and other fields are the main service directions of Jingdong block chain. For example, we participated in the construction of “Netcom Law Chain” of Guangzhou Internet Court, the pilot project of value-added tax special invoices for block chains, and the pilot project of opening electronic licenses with Suqian, etc. We have actually enabled many practical applications such as digital deposit certificates, electronic invoices, electronic licenses and so on.
For example, block chain digital certificates. We participated in the Block Chain Project of Beijing Internet Court and Guangzhou Internet Court. We need to set up a judicial alliance chain with the content of judicial service. What can we achieve? For example, signing a contract, notarizing in the notary office is one of the necessary processes. If there is this alliance chain, all operations can be achieved in the judicial “circle of friends”. Judicial alliance chain can be understood as a digital signature technology to judge the validity of contract signature. At the same time, the signed contract will be backed up to this alliance chain in time, so that the courts, notaries and judicial appraisal departments, including the signatories of both sides of the contract, can get the information in time.
In addition, Jingdong block chain has done the most in anti-counterfeiting traceability. The platform of anti-counterfeiting traceability in Jingdong block chain has been tested four or five times. The rate of user repurchase and user conversion have been significantly improved. The target customers in Jingdong have highly recognized the upgrading of consumer experience and the guarantee of commodity quality.
These are the general contents of this salon event.
Pay attention to the Wechat public number “Jingdong Cloud Developer Community”.
Reply to “PPT0526” for course video & PPT.
In addition to promoting technology, you can also click on “Jingdong Yun” to learn about 618 promotional activities!
The whole game is as low as 1% discount.