GitHub address: https://github.com/wxzz/CSharpFlink
Gitee address: https://gitee.com/wxzz/CSharpFlink
1. Computer hardware configuration
CPU: 4-core i5-7400, 2.7ghz, memory: 16g, random data point time window and calculation operator, CPU and memory usage of master node: 15% – 35%, 1500mb-2048mb, CPU and memory usage of work node: 0.1% – 2.5%, 18mb-30mb. The operation effect is shown as follows:
2. Computing node and task configuration
One master node and 10 compute nodes are deployed locally. The master node generates a calculation task of 100000 data points. Each data point generates a new data in one second, and calculates the maximum, minimum, average or sum value of the time window.
Aggregate calculation type:
Calculation statistics: csharpflink Core. Window. Operator. Avg:25003 Calculation statistics: csharpflink Core. Window. Operator. Max:24892 Calculation statistics: csharpflink Core. Window. Operator. Sum:25133 Calculation statistics: csharpflink Core. Window. Operator. Min:24972
Data point time window statistics:
Window statistics: 60 seconds window: 25015 Window statistics: 5 seconds window: 24976 Window statistics: 3600 seconds window: 25130 Window statistics: 300 seconds window: 24879
As shown below:
3. Configuration document parameters
As shown below:
(1) Maxdegreofparallelism: task parallelism. The master node generates tasks and the work node processes tasks, depending on this parameter.
(2) Masterlistenport: the listening port of the master node, which is used for active connection of work nodes.
(3) Masterip: master node IP, which is used for active connection of work nodes.
(4) NodeType: node operation mode, including master, slave and both.
(5) Remoteinvokeinterval: interval between remote calls to work nodes, unit: Ms.
(6) Repeatremoteinvokeinterval: the interval between calling the work node again after the failure of calling the work node, unit: Ms.
(7) Slaveexcetecalculateinterval: the interval between calculation tasks executed by the work node, unit: Ms.
(8) Maxframelength: the maximum data length transmitted between the master node and the work node, in bytes.
(9) Workerpower: work node capability coefficient, greater than 1, will send multiple tasks continuously.
4. Deployment structure
“Mastercache” is the cache of computing tasks of the master node, which can be consumed by the computing node immediately after completion. The deployment structure is shown as follows:
5. Test demonstration
Test demonstration, as shown in the following figure:
Internet of things & big data technology QQ group: 54256083
Internet of things & big data cooperation QQ group: 727664080
Contact QQ: 504547114
Cooperation wechat: wxzz0151
INeuOS industrial Internet operating system official account