Kyligence + Amazon cloud technology – realize fine operation and digital command on the cloud


Recently, kyligence co-founder and CTO Li Yang attended the “Amazon cloud technology innovate Data Driven Innovation Conference” and delivered a speechKyligence + Amazon cloud Technology | realizing fine operation and digital command on the cloudIn his keynote speech, combined with practical application cases, he gave kyligence’s answers to the difficulties faced by enterprises in the process of digital transformation in simple terms.

The following is the transcript of Li Yang’s speech at the conference:

The dilemma of enterprise digital management

After interviewing many enterprises, Gartner summed up the main anxiety of many CIOs in today’s digital transformation:

  • Unable to find the most valuable data;
  • Spend a lot of time looking for data rather than analyzing data;
  • There is a lack of unified data semantics in cross departmental global data analysis.

The main reason for these anxieties is thatIn the actual business construction, the enterprise adopts the vertical chimney construction, which is lack of horizontal alignment。 This leads to the lack of governance of enterprise business data, which will cause many problems, such as too complex data, unable to find data and so on.

So how to solve these anxieties?

Digital command starts with “business digital modeling”

Adding a digital modeling platform in the process of data source to data application is the best way to solve these anxieties.With the business digital model as the center, every business personnel can use data to optimize their daily work, rather than directly dealing with technical data.

The logical relationship is similar to the “farm management game”. In the game, as a digital commander of the farm, the farmer can control the daily operation of the whole farm from the perspective of God in the control panel, which is the logic of business digital modeling.

Abstracting the real farm into the digital world requires the process of digital modeling of business objects, business processes and business rules. Once the transformation from reality to the electronic world is completed, we can command and operate in the electronic world and optimize the work efficiency in reality. The optimized reality will give us the latest feedback, business object status and process status, and arrange the next round of command, thus forming a positive flywheel effect.

However, this is somewhat difficult in the real environment. From the technical perspective, these enterprises do have unified data platforms such as data warehouse and data lake, but when these data are connected to the business, they present a vertical “chimney construction”. In this construction mode, the business model is usually located at the top of the chimney. After the business analyst outputs the business model in his mind, the data engineers complete the transformation from data to business model, that is, a labor-intensive data analysis method of one analyst + a group of engineers.

Using this analysis method, the business model will not be precipitated. This means that when an enterprise needs to build a new business line, the previous business experience is difficult to reuse, there is no horizontal alignment, and it is unable to generate a global perspective, let alone global command.

Kyligence intelligent data cloud base

Kyligence + Amazon cloud technology - realize fine operation and digital command on the cloud

1. Data governance must rise from the technical level to the business level

To change this situation, data governance must rise from the technical level to the business level. On top of the digital platform of technology, there needs to be a digital platform of business, which can be calledBusiness digital model layer, it can also be called business semantic layer. As a platform layer, its role is to sort out, precipitate and reuse the business digital model of the enterprise step by step. When a business line is vertically constructed, it can keep the business digital model in this level. In this way, when we do the next vertical construction, it can achieve horizontal alignment and connection, and effectively reduce repeated construction. Its value also lies in the ability to form a digital baton with a unified overall caliber. Enterprises can use this unified standard (such as KPI standard) whether at the top, middle, grass-roots and cross departments. In the process of daily operation of the enterprise, it can show a state of high alignment and one hole.

The idea of business digital model (i.e. business semantic layer) is mentioned in the classical data warehouse theory and the way of Huawei data. The classical theory also recommends us to precipitate it. At present, many enterprises only pay attention to the technical level in the process of digital construction, and often ignore the level of business digital modeling.

2. Provide services based on business model to make everyone an analyst

Once business digital modeling is done, it will produce a series of immediate advantages. The most intuitive point is:Providing data services based on business model can realize data democratization, and everyone can become an analyst。 For analysts, if they can directly interact with the data service layer, they can no longer rely on engineers to collect data for them, but directly conduct data analysis, so as to work independently.

The core of kyligence is aMultidimensional database, it is different from the traditional relational database. It is a place that can reflect the business model. The so-called business model is to provide data services with concepts that can be understood by traditional business personnel such as “indicator”, “label”, “dimension” and “measurement”, which ordinary people can use without database foundation. Compared with the previous relational data assets, it gives ordinary business personnel the possibility of self understanding and self operation.

In terms of using data, kyligence supports exploration, minimally invasive innovation and even collaborative data sharing. For example, a site of a bank has produced some data exploration and innovation. The bank hopes that these results can be precipitated into the business model and even shared with the lobby managers of other parallel branches. This goal can be achieved at the model level of multidimensional database.

As another specific example, a bank intends to add 5000 analysts in 2019. If these 5000 analysts are trained by teaching them to use traditional relational databases, they are unlikely to produce ideal results. If they use the business data services provided by the multidimensional data model to enable them, they can directly use the data without too much extra learning. When they use the PivotTable function of Excel to connect to kyligence’s multidimensional data model service, they can directly use the data by themselves. In this way, the productivity of data can be greatly released, and 5000 ordinary salesmen can directly use data to optimize their daily work.

3. AI enhanced data service and management platform capabilities

Another layer of capability provided by kyligence is the enhanced data service and management platform capability of AI, which can reduce costs and increase efficiency through AI automation。 Once the business model is established, the data index can be automatically optimized according to the high-frequency dimension and measurement combination in the business model. The most valuable data and low-frequency data can be quickly found through statistics at the business model layer. We can also help you roll out the business model from some traditional SQL through reverse engineering. In addition, there is automatic data preparation, which can automatically raise the technical level data to the business level from the definition of the business.

Kyligence has open technical standards, and the whole system can seamlessly connect with various mainstream Bi platforms. This means that no matter which Bi tool is used, it can be connected to the unified standard data semantic layer, that is, the standard business model. The data interfaces provided by the platform are also standard and open, including SQL, mdx and rest APIs. The bottom layer has open source technology engines, including kylin, Clickhouse and spark.

The following is a case of kyligence providing customers with automated data services and management in Amazon cloud technology environment:

Practice 1: Practice of global large auto companies on Amazon cloud technology

A car company mainly producing smart cars hopes to complete the establishment of business model at the initial stage of entering the Chinese market. To complete this construction, we need to deal with many challenges. For example, in the early stage of platform construction, we need to face the problems of complex computer room approval process, high technology learning threshold, high construction cost and low utilization rate. As a large automobile enterprise, it will produce a large number of batch data and real-time data processing every day, and has high requirements for concurrency analysis.

Kyligence successfully helped the car company deliver their first data Lake platform on Amazon cloud technology in only three months.

Kyligence + Amazon cloud technology - realize fine operation and digital command on the cloud
There are many business scenarios we intend to solve, including several examples:

1. Based on predictive fault identification, it is possible to actively improve quality

The traditional data analysis mode can only analyze the real-time data status, that is, when an alert is encountered on the callcenter, it can only be found when there is a problem according to the current status. With the ability of data analysis, we can do early warning and identification in advance. Because there are many sensors installed inside the car, which can generate enough data accumulation, problems can be found seven days, a month, or even earlier. With the continuous advancement of technology, the time to find problems can also be continuously advanced.

2. The data Lake brings more scenarios: Vehicle networking data analysis

The car is like a small mobile room. Customers not only need the ability to move in the car, but also need services during the journey, such as catering, hotels, etc. These additional value-added services are actually a typical experience optimization or retention scenario for Internet service customers, which can be optimized through accurate data analysis.

3. Data Lake brings more scenarios: analysis of intelligent charging service

Battery life has always been a big problem for electric vehicles. Data analysis can greatly optimize the battery life of electric vehicles. For example, the most efficient charging equipment can be set up nearby by analyzing the places most frequently visited by customer groups; Help the user plan his driving route in advance, so that he can have endurance all the way.

The scope of such application scenarios is very wide, because on the middle stage of a unified data lake, kyligence’s core mission is to collect the data of various entrances from the upstream omni-channel data, and take this unified platform as the starting point to serve all downstream scenarios, Moreover, these scenarios can be expanded with the continuous improvement of the platform’s digital operation command ability.

Kyligence + Amazon cloud technology - realize fine operation and digital command on the cloud

The architecture implementation diagram based on Amazon data lake is relatively complex. Its main significance is that Amazon’s technology can well support both streaming data inflow and batch data inflow in the modules of monitoring, security, storage, computing and metadata.

Kyligence + Amazon cloud technology - realize fine operation and digital command on the cloud

This is a more simplified architecture diagram, which is the most common diagram of data Lake construction. Its value points are similar to the previous figure, but it removes the infrastructure below. In this kind of architecture diagram, we can easily focus on the technical level and ignore the business data model, which is the core ignored in today’s enterprise digital transformation.

Kyligence directly applies to the data of the upper layer, which may be a report or a self-service data use. Kyligence hopes to give users the ability of fast data innovation closed loop in the data Lake scenario of large automobile enterprises, that is, when enterprises need to analyze and collect data and innovate some ideas, they can give answers directly on the business model layer given by kyligence, Instead of requiring business personnel to find a lot of data engineers to help them solve problems. Therefore, this new structure gives self-service data innovation ability. Business personnel can use data themselves and give full play to the value of data. In addition to the core ability of fast closed-loop data innovation, it also produces other equally important values, such as improving customer interaction and operating efficiency.

Practice 2: SaaS customer operation on Amazon cloud technology

This is a typical SaaS service digital operation practice on the cloud. Our customer is the site building SaaS service provider, strikingly, which helps customers establish individual / individual company websites within a few minutes. At present, it has served more than 200 + countries around the world and more than one million users.

Kyligence + Amazon cloud technology - realize fine operation and digital command on the cloud

The scenarios and pain points behind the strikingly business are classic. Because its scenario is website traffic analysis, its business model is relatively stable. There are many existing and reference models in the business model layer, but its technical challenges are relatively large. As early as 2017, strikingly used Apache kylin as its tool called analytics platform. Its capabilities include click stream analysis, PV, UV, access devices and sources of web pages, such as classic customer traffic, website behavior, as well as retained analysis scenarios and models,The difficulty is that it needs sub second response ability and high concurrency, because it has a large number of customers worldwide.

For the management of strikingly, it is very difficult to provide the operation and maintenance of such analysis services, because its services cannot be interrupted and need to last for 7 years × 24 hours. Open source kylin’s tools and services will be more challenging in terms of reliability, and its overall cost (TCO) will always be high, not only the resource cost on the cloud, but also the cost of big data technicians, that is, a lot of data engineers are needed under the traditional chimney construction.

The ability we gave strikingly is in itAfter migrating to kyligence cloud platform, give it ai enhanced modeling and automatic optimization capabilities, so as to release its IT productivity

This includes automatically optimizing the business model through SQL queries. Maybe many companies are more technical. Take strikingly as an example. Their personnel composition is still dominated by technicians. Therefore, many innovations can start from SQL, including dynamic model adjustment ability. At any time during the model use process, you can manually and flexibly adjust the model design, such as increase / decrease relationship table or analysis dimension, indicator, etc. This is a capability that open source kylin does not have.

After comparing the overall operating cost of the traditional deployment method (i.e. Hadoop + kylin on the cloud) of strikingly with the updated kyligence cloud, it can be seen that only the hardware part (resources on the cloud) will be greatly improved. The main source of improvement here is Hadoop. This cluster has been optimized. There is a cloud native architecture on the new kyligence cloud, which no longer needs the traditional big data layer of Hadoop.This not only reduces a lot of hardware costs, but also reduces a lot of operation and maintenance costs.

Let’s look at the effect of high concurrency. Under the automatic model tuning and index optimization of AI, we can easily achieve the business goal of 100t / s customers at the entrance of a single query node. And with the growth of data, the performance of our query can remain stable. This is the support provided by the precomputing capability under the multidimensional model behind Apache kylin or kyligence cloud. Most of our query calculations are completed in advance. The amount of calculation during online service can remain stable, and is almost independent of the original amount of data. The advantages of kyligence in strikingly can be summarized as follows:

To sum up, kyligence has given the enterprise business digital model capability on Amazon cloud technology platform. It is an AI enhanced data service and management platform. Through centralized business data model management, kyligence can unify the business caliber, so that each salesperson can use data and data innovation by themselves. In addition, AI enhanced automatic data preparation can also conduct automatic system tuning, which greatly reduces the overall TCO by saving manpower and material resources.

About kyligence
Kyligence, founded by the founding team of Apache kylin, is committed to building a next-generation intelligent data cloud platform to realize automatic data service and management for enterprises. Based on machine learning and AI technology, kyligence identifies and manages the most valuable data from cloudy data storage, and provides high-performance and highly concurrent data services to support various data analysis and applications, while continuously reducing TCO. Kyligence has served a number of banking, insurance, manufacturing, retail and other customers in China, the United States and the Asia Pacific, including China Construction Bank, Shanghai Pudong Development Bank, China Merchants Bank, Ping An Bank, Bank of Ningbo, Pacific Insurance, China UnionPay, SAIC, FAW, Anta, yum, Costa, UBS, MetLife, appzen and other global well-known enterprises and industry leaders. The company has passed ISO9001, ISO27001 and soc2 type 1 certification and audit, and has many ecological partners all over the world.