Author: Shanghai Electric SmartOPS team
Small T Introduction: Shanghai Electric Group Co., Ltd. is a world-class comprehensive high-end equipment manufacturing enterprise, focusing on three major business areas of smart energy, smart manufacturing and smart infrastructure, providing customers with industrial-grade green intelligent system solutions. Its business covers the world, mainly including new energy, integrated energy, environmental protection, medical equipment and industrial automation.
The company builds the core competitiveness of energy storage around the "one-core 3S" core product chain. As a key component of the "SmartOPS energy storage intelligent operation and maintenance system", based on technologies such as the Internet of Things, big data, and machine learning, the energy storage system can open up the complete technology from information collection to cloud access, to data storage and data analysis. Chain, build a smart energy storage management and control system platform, realize advanced applications such as comprehensive monitoring, predictive maintenance, thermal management analysis, etc., to help customers achieve optimal configuration and efficient use of energy storage equipment.
1. Application background
SmartOPS supports cloud deployment and local deployment.
The cloud deployment is based on the current unified architecture of Shanghai Electric Group, using the cloud time series database, various resources that can be expanded and flexibly configured, which can easily meet the usage requirements, efficiently and smoothly.
However, in local deployment, it is necessary to focus on the limitations of local hardware resources, such as the memory, CPU, and read and write performance of the station-end system. The current configuration of the station-end system is shown in the figure below.
Therefore, we need to consider the time series database suitable for deployment in the station system, and then we have the next technology selection.
2. Technical selection
The overall selection consideration will involve many dimensions, including read and write performance, compression rate, industry recognition, product vitality, service support, security, compatibility, maintainability, scalability, functionality, reliability, ease of use properties such as sex.
Our team focused on evaluating the following databases:
- OpenTSDB: With HBase as the underlying storage, it encapsulates its own logic layer and external interface layer upwards. This architecture can make full use of the features of HBase to achieve high data availability and better write performance. However, compared with the native time series database, the OpenTSDB data stack is longer, and there is still room for further optimization in terms of read and write performance and data compression.
- InfluxDB: Currently the most popular time series database, data is stored in columns, which can efficiently process, store, and query time series data, and provides a feature-rich Web platform for visual display and interactive analysis of data.
- Apache IoTDB: A distributed time series database specially designed for the Internet of Things. Data is stored in columns, with excellent write performance and rich data analysis functions, and can effectively handle out-of-order data.
- TDengine: A distributed time series database specially designed and optimized for IoT scenarios. Data is stored in columns, with extreme write performance and rich query functions. It also provides common functions of big data platforms such as caching, stream processing, and message queues. The installation package mentioned on its official website is less than 10MB and the performance improvement of 10 times is indeed very attractive.
- ClickHouse: A powerful OLAP database, data is stored in columns, with extremely high data compression ratio, high write throughput and extreme query performance, and provides a wealth of data processing functions to facilitate various data analysis.
Based on the fact that localized deployment on the site requires lightweight resource consumption, we first excluded OpenTSDB and Apache IoTDB. OpenTSDB is based on HBase and is relatively heavy, while Apache IoTDB is not friendly to edge lightweight devices in terms of resource consumption; ClickHouse advantage It is fast for a single table, and it is weak in other aspects, including join, management and operation and maintenance, which are more complicated and give up.
The R&D team finally settled on testing options in InfluxDB and TDengine.
3. Preliminary comparative test
The technical team actually tested InfluxDB and TDengine.
Let's first look at the test situation of InfluxDB.
The installation package size of InfluxDB is 60.2M, and the resource usage after running is shown in the following figure:
Open the data access of the test site. At the current moment, the number of fields written per minute is less than 3000. At this time, the resource consumption is shown in the figure below. Currently, the CPU consumption of InfluxDB has increased, and the memory resources have not changed much.
Test the query function with the following command:
The resource consumption is shown in the following figure:
Meanwhile, the current query gets stuck, so there is no result. It can be seen that under the current resource configuration conditions, the resource consumption of InfluxDB privatization deployment is high. If the local application service (SmartOPS) is enabled at the same time, the required functions cannot be provided.
Let's look at the test situation of TDengine.
We are using TDengine version 184.108.40.206. Check the size of the installation package TDengine-server-220.127.116.11-beta-Linux-x64.rpm and find that it is only 9.42MB. The backend developers deployed it to the test nodes.
The resource consumption when just deployed is as follows:
When simulating data access at a site, the resource consumption is as follows:
Then test the query function. Under the same conditions, TDengine has a small increase in the CPU for a short time, and at the same time, the query return results are obtained in milliseconds.
From the above preliminary tests, it can be found that in the case of limited local deployment server resources, compared with InfluxDB, TDengine has obvious advantages in the application of station-end systems. TDengine can better deal with the resource limitation of low-profile servers at the site. For energy storage scenarios of large-scale applications, it provides a cost-effective solution.
TDengine is aimed at the scenario characteristics of IoT applications, such as few data updates or deletions, no transaction processing of traditional databases, more writing and less reading than Internet applications, etc., and through "one data collection point, one table" and "super table" ” concept, which simplifies the data storage structure, and these optimizations really fit our scenario very well. In addition, through some pre-computing functions, the efficiency of aggregation query is improved, which fully meets the needs of energy storage scenarios when our site resources are limited.
So we tried to implement TDengine in the system.
Fourth, TDengine data model design
The data format returned by the energy storage station equipment is basically fixed, with one timestamp and one value. So we built a hypertable for one station. The sub-table is based on the information of the point, and one sub-table per point is used to distinguish the information collected by different devices. Combined with business requirements, the tags are set to 5: point identification, station id, sub-station id, unit id and device id.
The supertable table creation statement is:
create table ops (ts TIMESTAMP,value FLOAT) TAGS (name NCHAR(10),sid NCHAR(20),sub NCHAR(10),unit NCHAR(10),dev NCHAR(20))
The subtable creation statement is:
CREATE TABLE escngxsh02_g01_e03h01_c1 USING ops TAGS ("C1","ESCNGXSH02","G01","E03","E03H01")
After using InfluxDB for a week, use the SQL statement
select * from ops WHERE ts>1629450000000 and ts<1629463600000 limit 2;
To execute the query, the memory usage rate reached 80%, and no results came out after ten minutes, so it was completely unsuitable for business use.
After using TDengine for nearly a month, using the same SQL statement, the query takes only 0.2 seconds. The performance is excellent.
Five, the use of TDengine
At present, the technical team has adopted TDengine as the core time series database of the SCU (Station Control Unit) architecture to realize the comprehensive information perception, local operation control and coordinated protection functions of the energy storage system; Data analysis and operation optimization protect the safety of energy storage power stations in all aspects.
TDengine's high-performance writing and aggregation query functions can respond to power station operation information monitoring in milliseconds.
In terms of compression, TDengine also performs very well. In the case of the same number of collection points, before using TDengine, we used InfluxDB, and the amount of data in one day was about 200 MB. After using TDengine, the data in one day was less than 70MB, which is 1 of InfluxDB. /3.
In follow-up projects, we will continue to expand its distributed cluster applications, build digital archives of the operation of energy storage power stations, and combine the developed analysis algorithms, prediction algorithms, and data mining technologies to achieve power station stability analysis, efficiency and loss analysis, Advanced analysis and diagnostic capabilities such as failure prediction, life prediction, location of performance shortcomings, and thermal management analysis.