The introduction of 5g network increases the demand for data volume and speed, which brings pressure to the traditional data architecture. The demand for absorbing data traffic is growing unprecedentedly. At the same time, intelligent and dynamic decisions should be made across multiple data streams to promote execution.
Current data stream processing architectures are usually sufficient to construct processing pipelines, but they can not meet the requirements of application critical task programs. Low latency and multi-step responsive decision-making highlight the requirements of these task-based applications.
In addition, it is the opposite of the traditional few central hub data centers. Data and processing will be decentralized through several edge data centers as the density of 1 million connected things per square kilometer is expected to increase and the low latency per millisecond is required.
In the incomplete information confluence, traditional and modern processing stream data selection will fail. In order for interactive low latency applications and pipelined pipelines to coexist, they must use the same data to drive cross functional consistency.
The first four parts of incomplete information are as follows:
1. Microservice architecture requires separation of state and logic
What is missing is an understanding of the logic of business types and where they should exist. Although the application flow control logic can be retained in the application layer to make the computing container truly stateless, the data business logic must be driven together with the existing data.
2. Network bandwidth utilization efficiency
When you store the state in the NoSQL data store and the instance container has to move 10 to 25 KB of data payload for each interaction (for example, read an object from the store, modify it, and send it back to the data store), the application will soon start consuming a lot of network bandwidth. In a virtualized or containerized world, network resources are like gold. People shouldn’t waste it on trivial data movement.
3. Basic premise of stream processing
Today’s stream processing is based on the concept of time windowing: the event time window or one of the processing time windows. This is not the real situation. Organizations need to continuously process events, whether they arrive individually or in context. This approach will avoid problems such as missed events, because they will only be late, without having to inflate the database to wait for the last known event that is late.
4. Cross polling multiple data streams to construct complex events that drive decisions
An event driven architecture is a message flow, each of which is associated with an event to drive certain operations. The challenge of architecture is to build complex events from multiple data streams, or to change a single data stream to multiple states based on complex business logic.
- The intelligent flow processing architecture is operable
- Absorbing incoming event data into the state machine
- Building context entity state from multiple ingestion streams
- Applying the rule set of business rules to drive decisions
- These rules are enhanced and enriched by iteratively fusing the new knowledge obtained from the machine learning plan
- Let decisions spread to drive implementation
- Once the data that does not need context completion / processing in real-time processing is migrated to archive storage
Intelligent stream processing architecture consists of a unified environment for ingestion, processing and storage.
This integrated method with built-in intelligent function can analyze the data in the location. It utilizes the fast in memory relational data processing platform (imrdpp) to not only make streams “smart”, but also provide linear scaling, predictable low latency, strict acid and much lower hardware space edge that can be easily deployed in the following locations.
With built-in analysis functions such as aggregation, filtering, sampling and correlation, as well as stored procedure / embedded supervised and unsupervised machine learning, all elements of stream processing for real-time decision-making can be obtained on an integrated platform.
If you are interested in voltdb’s big data low latency solution for industrial Internet of things and real-time data platform management for the whole life cycle, you are welcome to chat in private and discuss with more small partners.