[open source] csharpflink (net 5.0 Development) distributed real-time computing framework, PC 100000 data point second level computing test description

GitHub address: https://github.com/wxzz/CSharpFlink
Gitee address: https://gitee.com/wxzz/CSharpFlink

reference resources:[open source address] give up Flink NET5. 0 development csharpflink, brief design, deployment and secondary development instructions.

 1. Computer hardware configuration

CPU: 4-core i5-7400, 2.7ghz, memory: 16g, random data point time window and calculation operator, CPU and memory usage of master node: 15% – 35%, 1500mb-2048mb, CPU and memory usage of work node: 0.1% – 2.5%, 18mb-30mb. The operation effect is shown as follows:

2. Computing node and task configuration

One master node and 10 compute nodes are deployed locally. The master node generates a calculation task of 100000 data points. Each data point generates a new data in one second, and calculates the maximum, minimum, average or sum value of the time window.

Aggregate calculation type:

Calculation statistics: csharpflink Core. Window. Operator. Avg:25003
Calculation statistics: csharpflink Core. Window. Operator. Max:24892
Calculation statistics: csharpflink Core. Window. Operator. Sum:25133
Calculation statistics: csharpflink Core. Window. Operator. Min:24972

Data point time window statistics:

Window statistics: 60 seconds window: 25015
Window statistics: 5 seconds window: 24976
Window statistics: 3600 seconds window: 25130
Window statistics: 300 seconds window: 24879

As shown below:

3. Configuration document parameters

As shown below:

(1) Maxdegreofparallelism: task parallelism. The master node generates tasks and the work node processes tasks, depending on this parameter.

(2) Masterlistenport: the listening port of the master node, which is used for active connection of work nodes.

(3) Masterip: master node IP, which is used for active connection of work nodes.

(4) NodeType: node operation mode, including master, slave and both.

(5) Remoteinvokeinterval: interval between remote calls to work nodes, unit: Ms.

(6) Repeatremoteinvokeinterval: the interval between calling the work node again after the failure of calling the work node, unit: Ms.

(7) Slaveexcetecalculateinterval: the interval between calculation tasks executed by the work node, unit: Ms.

(8) Maxframelength: the maximum data length transmitted between the master node and the work node, in bytes.

(9) Workerpower: work node capability coefficient, greater than 1, will send multiple tasks continuously.

4. Deployment structure

“Mastercache” is the cache of computing tasks of the master node, which can be consumed by the computing node immediately after completion. The deployment structure is shown as follows:

 5. Test demonstration

Test demonstration, as shown in the following figure:

 Internet of things & big data technology QQ group: 54256083

 Internet of things & big data cooperation QQ group: 727664080

 Website:http://www.ineuos.net

 Contact QQ: 504547114

 Cooperation wechat: wxzz0151

 Official blog:https://www.cnblogs.com/lsjwq

 INeuOS industrial Internet operating system official account