Hologres: the core principle of high performance native acceleration maxcompute


Hologres (Chinese Name: interactive analysis) is a one-stop real-time data warehouse developed by Alibaba cloud. This cloud native system integrates real-time services and big data analysis scenarios. It is fully compatible with PostgreSQL protocol and seamlessly connects with big data ecology. It can support real-time writing, real-time query and real-time offline federated analysis with the same data architecture. It simplifies the business architecture, provides real-time decision-making ability for business, and makes big data play a greater business value. From the birth of Alibaba group to the commercialization of cloud, with the development of business and the evolution of technology, hologres is constantly optimizing its core technology competitiveness. In order to let you know more about hologres, we plan to continue to launch a series of revealing the underlying technology principles of hologres, from high-performance storage engine to high-efficiency query engine, from high-throughput write to high QPS query, etc, All round interpretation of hologres, please continue to pay attention!

Past highlights:

In this issue, we will bring the technical principle analysis of hologres high-performance native accelerated query maxcompute.

With the continuous enrichment of data collection means and the accumulation of industry data, the data scale has grown to the level of massive data (TB, Pb, EB) that cannot be carried by traditional software industry. Therefore, maxcompute (formerly known as ODPs) emerges as the times require. It is committed to the storage and calculation of batch structured data, and provides solutions and analytical modeling services for massive data warehouse Fully managed EB class data warehouse solution.

Hologres naturally and seamlessly integrates with maxcompute in the offline big data scene. It can speed up the query of maxcompute without data import and export. It is fully compatible with various maxcompute file formats, and realizes the millisecond level interactive analysis of Pb level offline data. Behind all this, SQE (s query engine) is the executor behind hologres. Through SQE, the native access to maxcompute is realized, and then combined with the processing of HQE, a high-performance distributed execution engine of hologres, the ultimate performance is achieved.

The main advantages of hologres to speed up the query of maxcompute are as follows:

  • High performance:It can directly speed up the query of maxcompute data, and has the query performance of sub second response. It can directly query ad hoc in OLAP scenarios, meeting the vast majority of analysis scenarios such as reports.
  • Low cost:After years of development, users have stored a large amount of data on maxcompute, which can be accessed directly without redundant storage; On the other hand, users can only migrate the data of some high-performance scenarios to SSD, and the data of analysis scenarios such as reports can be stored in maxcompute to further reduce the cost.
  • More efficient:Realize native access to maxcompute, without data migration and import, you can access all kinds of maxcompute file formats with high performance and full compatibility, as well as hash / range clustered table and other complex tables, so as to reduce the cost of users.

    Introduction to SQE architecture

    Hologres: the core principle of high performance native acceleration maxcompute

As shown in the figure above, the overall architecture of SQE is very simple. The data of maxcompute is uniformly stored in Pangu. When hologres executes a query to speed up the query of maxcompute data, on the hologres side:

  • Hologres frontier requests meta and other related information from SQE master through RPC.
  • Hologres Blackhole requests specific data related information from SQE executor through RPC.
  • SQE consists of processes with two roles:

    • SQE master is responsible for processing meta related requests. It is mainly responsible for obtaining table, partition metadata, authentication, file fragmentation and other functions.
    • As the core of SQE, SQE executor is responsible for specific data read requests, involving block cache, pre read, UDF processing, expression push down processing, index processing, metric, meter and other functions.

    Core technology innovation of maxcompute engine

    Based on the SQE architecture, it can speed up the data query of maxcompute with high performance, mainly based on the following technological innovation advantages:

1) Abstract distributed appearance
Combined with the distributed characteristics of maxcompute, hologres abstracts a distributed appearance to support access to maxcompute distributed data. At present, it can access the maxcompute distributed Pangu file across clusters, and read the cluster nearby according to maxcompute.

2) Seamless interworking with maxcompute meta, supporting metadata caching with version
The meta of SQE and maxcompute are seamlessly interworking, which can achieve the real-time acquisition of meta and data. It supports the automatic synchronization of maxcompute metadata to the appearance of hologres through the import foreign schema command, realizing the automatic creation of appearance and the automatic update of structure.

3) Support UDF / expression push down
SQE implements user-defined UDF calculation by supporting UDF / expression push down; Pushing down the expression can reduce the cost of useless data transmission and further improve the performance.

4) Asynchronous Orc reader
At present, most of maxcompute data is in Orc format. In hologres v0.10 and above, hologres updates the execution engine, uses asynchronous reader for more efficient asynchronous reading, and supports asynchronous readingprefetchTo further reduce the read delay; In addition, hologres supportsIO merging, lazyread, lazy decodingAnd other column optimization techniques to reduce the IO latency in the entire query, in order to bring the ultimate performance.

5) Support block cache
In order to avoid using IO to fetch data from files every time you read data, SQE also uses blockcache to put commonly used and recently used data in memory, so as to reduce unnecessary IO and speed up reading performance. In the same node, consistent hash is used to share the same accessed data with a block cache. For example, in the scan scenario, it can bring more than 2 times performance improvement, greatly improving the query performance.

6) Resident process, reduce scheduling overhead
Traditional process model and other architectures need dynamic and real-time scheduling operations such as creating process, which brings large scheduling overhead. SQE adopts the resident process mode to avoid unnecessary scheduling overhead. In addition, it can greatly improve the hit rate and effective utilization rate of block cache.

7) Network shuffle to reduce the cost of stock taking
Network shuffle needs to provide a fast and stable fault tolerance mechanism. In order to complete the data shuffle, network shuffle must ensure that the sender and receiver processes are alive at the same time. Similarly, if the traditional way of disk dropping is used to retry the network shuffle, although the stability can be guaranteed, the disk IO may introduce a relatively large performance overhead in the retry process. In order to solve this problem, we optimize the staged scheduling to solve the fast and stable fault-tolerant problem.

Upgrade maxcompute engine to HQE

As mentioned above, we use SQE to speed up the query of maxcompute. The performance can be very good through SQE query. However, when we interact with hologres, there will be a layer of RPC interaction in the middle. When the amount of data is large, there will be a certain bottleneck in the network.

Therefore, based on the existing capabilities of hologres, we have optimized the execution engine in version v0.10 and above of hologres to supportHologres HQE query engine reads maxcompute table directlyThe performance is further improved, which is more than 30% higher than SQE reading.

This is mainly due to the following aspects:

1) It saves the interaction of RPC between SQE and hologres, which is equivalent to saving one time of data serialization and deserialization, and further improves the performance.
2) The block cache of hologres can be reused, so that there is no need to access the storage during the second query, so as to avoid storing IO and access the data directly from the memory to speed up the query.
3) The existing filter push down ability can be reused to reduce the amount of data to be processed.
4) Read ahead and cache are implemented in the underlying IO layer to further accelerate the performance of scan.

The following is the performance data of an actual online business query of a customer:

E2E running time SQL Num SQE query performance (average response) HQE query performance (average response) Performance improvement
2-10s 547 4,956 ms 2,609 ms +47.34%
10-30s 207 16,757 ms 5,457 ms +67.43%
More than 30s 63 78,686 ms 12,666 ms +83.90%
Total 817 13,631 ms 4,106 ms +69.87%

Execute 817 SQL, the overall performance is improved by 70%, and the long query is improved by more than 80%.

Note: the optimization has been launched in hologres v0.10, welcome to view the document.

Maxcompute accelerates scene selection

There are two ways to speed up the query of maxcompute in hologres

1) Create appearance (data is still stored in maxcompute), the performance will be 2-5 times higher than the query in maxcompute
2) Imported into the inner table, the performance is about 10-100 times higher than the appearance

The principle of creating appearance is that foreign data wrappers in PostgreSQL can access the data stored outside through external access interface.It is suggested that you use the more convenient import foreign schema method to create the appearance, which can better simplify the synchronization of metadata without paying attention to the field type mapping.

In fact, the way of directly building the appearance is to improve the efficiency by using the optimization ability of the query engine, but it does not take advantage of the indexing ability of hologres. Therefore, when importing the outer surface to the inner table, the index structure of the inner table can be specified according to the query mode, which can bring higher query performance. This is the reason why the performance of the inner table is better when the outer table is imported. It can give full play to the index optimization ability of the data warehouse.

At present, the two methods are mainly compared as follows:

Scene / dimension performance Storage cost Data volume Indexes Convenience
Hologres internal table very nice high
(SSD) It can support Pb level It can support bitmap, clustering and other indexes Need to import data
Hologres appearance preferably low
(HDD) 200GB limit for single query Only ODPs index is supported No need to migrate and import data

As can be seen from the above comparison:

  • If you have a large amount of data and high performance requirements (such as within 100ms), you are sensitive to query latency and have SLA requirements for queries. It is recommended that you import the data into the table in hologres for query access.
  • For temporary exploratory analysis or delay insensitive internal business, maxcompute can be used to reduce data movement.
  • In addition to the above scenarios, you can choose the appropriate scenarios according to the specific business situation.

The combinatorial relationship between maxcompute and hologres

The above describes a lot of scenarios of how the hologres appearance query engine can speed up the query of maxcompute, but it does not mean that all types of queries are suitable for execution on the hologres appearance engine.
Hologres is a synchronous query engine designed for interactive analysis scenarios. It is designed for big data in and small data out scenarios. It is typically used in serving and analytics scenarios. Maxcompute is an asynchronous data processing engine designed for massive data processing scenarios. It is designed for big data in and out scenarios, and is typically used in ETL scenarios. In the ETL scenario, jobs are submitted asynchronously and IO interfaces are optimized for scan. The redundant design of nodes is needed to support high availability in the calculation process, and the computing state is needed to drop the disk, so that it can automatically try again in case of failure. All these are capabilities that hologres does not have. Therefore, the combination of maxcompute and hologres forms a one-stop experience of data processing and service, reduces the isolation and redundancy of data, provides a reasonable solution architecture for big data warehouse, and supports the real-time offline integrated development experience.
Hologres: the core principle of high performance native acceleration maxcompute


Hologres makes full use of the advantages of hologres and maxcompute through the deep integration of SQE and maxcompute. With the ultimate performance as the goal, it can directly speed up the query of maxcompute data, make users more convenient and efficient for interactive analysis, reduce the huge analysis cost, and realize the integration of offline data warehouse services.

About the author: Wang Qi (Hua Ming Huiqing), a technical expert of Alibaba, is now engaged in the research and development of interactive analysis engine hologres.

In the future, we will launch a series of revealing the underlying principles of hologres technology. The specific plan is as follows. Please keep your attention!