Flutter dynamic framework thresh

Time:2022-5-24

Original link:The dynamic flutter framework “thresh” of manbang is now open source

1、 Foreword

Since the birth of mobile terminal technology stack, its dual terminal development cost and release efficiency have been widely criticized. In order to solve these problems, the front-end cross end technology has been trying constantly, hoping to develop at one time, run at multiple ends and release quickly. It has experienced several stages of technological development.

The first stage: represented by H5, rendering based on WebView

Only one development can run on both ends, which solves the problem of low development efficiency. However, WebView has serious performance problems, and there is a significant gap in user interaction experience compared with native rendering.

The second stage: represented by RN and weex, front-end technology stack development and native rendering

These schemes are developed using front-end technology and finally mapped to native component rendering. The user experience has been greatly improved compared with H5 scheme. However, there are also deficiencies in the scheme at this stage. Because the rendering of the framework ultimately depends on dual end native components, there are problems of inconsistent dual end experience and platform compatibility. In extreme cases, the development cost is even higher than that of dual end native development.

Stage 3: fluent, self drawing engine rendering

Based on the skia rendering engine, Google launched the fluent cross platform framework, which supports three platforms: Android / IOS / Web (especially the release of 2.0 supports the whole platform).

Based on the self drawing engine, fluent smoothes the differences of various platforms, and truly realizes one development and multi terminal operation. The industry also has high hopes for flutter to completely solve the problem of cross end development. However, flutter is not perfect. Its dynamic capability is insufficient and cannot be released as quickly as H5, RN and other technologies.

In order to solve the problem of insufficient dynamic capability, the front-end team of manbang University began to explore the dynamic capability of flutter from 2019, developed the dynamic flutter framework, continuously optimized and iterated internally, and has launched 20 + pages, including core page order details, owner source details, navigation map, etc., and opened the source at the end of 2020.

2、 Thoughts on the dynamic of flutter

The original intention of thresh project is to provide a complete cross end dynamic scheme based on flutter, with performance reaching or even better than react native. In addition, its multi end rendering consistency and the default development language of Google Fuchsia system to be launched soon are flutter, which show that thresh will be full of imagination in the future.

2.1. Common dynamic solutions

The following points should be considered to realize the dynamics of the flutter:

  • Replacement of flutter compilation products

Google originally planned to launch the code push scheme in 2019, but later abandoned it for two main reasons: violating the regulations of the app store and security considerations; But at present, Android can be dynamic through product replacement, while IOS cannot.

  • Component construction

Define some core general components through dart, distribute the page JSON assembled by the existing component list on the platform, and then parse and render it into a page on the end. This scheme can meet light interaction scenarios, but can only support limited dynamics.

  • Custom dart transformation + dynamic logical mapping

By customizing a set of dart specifications and generating JSON through the converter to achieve dynamic update, the performance loss is small, but the logic dynamics needs to be embedded in advance, and the front-end development students need a certain learning cost.

  • Custom DSL + depends on the dynamic execution of JS engine

Similar to RN / weex, the user-defined dynamic UI description + the interpretation and operation transformation idea of JS engine are used to finally build the page and execute dynamic logic. This scheme is very friendly for front-end development and has zero learning cost, but there will be some performance loss due to running in JS engine.

2.2 selection of thresh

Manbang’s actual use scenario and rapid business iteration require both Android and IOS to support dynamics, so the idea of product replacement can not completely solve the problem. Then we consider using the idea of componentization to splice multiple business components. Although it can build pages, the disadvantages are also obvious, which can not be realized in complex interaction logic. In addition, although the user-defined dart description UI scheme meets the requirements of dynamic update, the logic dynamics is still not strong, and dart development has a certain learning cost for front-end development students.

Finally, considering the factors such as development efficiency, learning cost, multi terminal performance and consistency, we chose custom JS to describe the interpretation, operation and transformation idea of UI + JS engine, the like react syntax structure, and the development language uses JS / ts.

3、 Implementation principle

3.1 principle of constructing dart page

The basic unit for describing the composition of views in fluent is widgets. Each widget only contains the configuration information of the current component. It is a lightweight data structure that can be created and destroyed efficiently. Many widgets are combined to build a widgettree containing all the information of the view. Then, fluent will generate elementtree from widgettree, and then generate renderobjecttree from elementtree. The element in the elementtree will hold its corresponding widget and renderobject at the same time.

Flutter dynamic framework thresh
Among the three trees, widgettree will be frequently created and destroyed, but elementtree and renderobjecttree will only change when the state changes. Elementtree is responsible for updating and diff of elements, and renderobjecttree is responsible for actual layout and drawing.

The core idea is to construct the widget, the first of the three trees in the page rendering logic of fluent, through JS. Among them, it is necessary to complete the basic component mapping between JS and fluent layer, and then generate the UI description through JS engine and pass it to uiengine of dart layer. Uiengine converts the UI description into fluent control and finally renders it into a page.

Thresh framework completes the definition and development of common basic components, which can support the access of more than 95% of business scenarios. The syntax definition rules support react and provide zero cost access to front-end developers. The list of supported components and some of their properties are as followsFlutter dynamic framework thresh
Flutter dynamic framework thresh

3.1.1. Flutter initialization

Flutter is executed by the program starting with the main() function. It mainly completes the following tasks:

  • Establish a communication channel methodchannel with native to ensure that all communications can be received and sent;
  • Establish the distribution channel of all processing methods when receiving the message, so as to ensure that all legitimate communications can be processed correctly in fluent, and send the media data of the current device to JS through methodchannel;
  • Register the interception function to convert JSON into widget after receiving rendered JSON data;
  • Finally, establish the initial hosting page of the fluent app, which will be in a waiting state until receiving the message sent by JS to display the page; At the same time, send a ready message to JS, indicating that the fluent environment is ready and the page can be displayed.

3.1.2. Generate widgettree

According to all intercepting functions registered for widgets in fluent, JS will provide a set of corresponding atomic components to convert components between two different DSLs. In JS, the construction of UI is realized through JSX, and the writing method of react is used for reference.

By building the description layer of UI in JS, the UI description is converted into JSON format string and sent to fluent through native. After parsing the JSON string, fluent creates the corresponding widgettree and performs subsequent rendering operations.

Flutter dynamic framework thresh

3.1.3 communication between JS and flutter

Before the execution of JS code, native will register two communication methods in the execution environment of JS code, one is the channel for JS to deliver messages to fluent, and the other is the channel for fluent to deliver messages to JS. Through these two channels, you can realize the flow of all data between JS and fluent (which will be described in detail in Chapter 3.2 later).

3.1.4. Building a fluent page

When the data conversion of all links is completed, the modeltree & widgettree will be obtained. The modeltree will hold and cache the widgettree, and finally build a widget page and render the display. The process of page construction and rendering mainly includes:
Flutter dynamic framework thresh
After receiving the rendered JSON data, fluent will start from the bottom layer by recursive traversal and parse each independent rendered data node into a model object. The model will hold all the rendering data and associate its own parent node; At the same time, the model will carry all the rendering data, generate its corresponding widget instance through the widget interception function, and hold the widget instance.

For example, in JS<Container />The component will be created as a widget instance named dfcontainer after intercepting the function in fluent. Widgets such as dfcontainer are a set of custom components encapsulated by atomic components provided by fluent.

When you create a widget through model, if you find itsisStateful = true, a statefulwidget will be wrapped in the outer layer of the widget instance, and the model will hold the statefulwidget and its state for updating later. That is, if a model hasisStateful = true, it will also have the feature of widget & statefulwidget & state.

During the traversal, the original JSON data will be transformed into two trees – modeltree & widgettree. Each node in the widgettree will be held by the corresponding node in the modeltree.

For the first displayed page, the created widgettree will be used to directly replace the content of the hosting page created during initialization; Instead of the home page, it will be directly through navigator Push (), use widgettree to create and display a new page. The whole process is as follows:
Flutter dynamic framework thresh

3.2 communication mechanism

JS and fluent depend on native and completely independent ends: the data operation and flow in JS will not directly affect the rendering of fluent pages; The rendering process of fluent will not block the code execution of JS.

In order to connect the completely independent two, we found a medium that can not only connect with JS, but also transmit messages with fluent – native By passing a message from one end to native, and then from native to the other end, the communication between JS and fluent is realized.

The dynamic fluent framework is mainly composed of these three parts. Each part handles different logic and binding event communication to update the rendering page and event response. Its core rendering communication process: fluent ⇋ native ⇋ JS.

3.2.1 build a three terminal communication link

When the flutter is initialized, the flutter will establish a communication relationship with the native through the methodchannel. The methodchannel is a two-way communication link, which can not only receive the native message in the flutter, but also actively send the message to the native.

At the same time, native will inject a method into the JS context before executing JS code. We name this method methodchannel_ js_ call_ Flutter is used to enable JS to deliver messages to flutter. Therefore, the communication link in flutter dynamics is shown in the following figure.
Flutter dynamic framework thresh
From the above two links, it can be found that the message from JS to native can reach fluent smoothly; However, there is no direct communication link between fluent and JS, which is interrupted in native. To solve this problem, JS will expose a channel named methodchannel in the context_ flutter_ call_ JS, and the parameters of this method are the message content, so native can directly call this method to pass the message to JS.

3.2.2 “half duplex” communication process

In thresh, almost all three terminal communication requirements are “half duplex”. “Half duplex” here refers to that when one party is the message sender, it is unable to obtain the feedback of the message receiver through the channel where the message is currently delivered. This means that when the sender sends a message, they will end their communication behavior. They don’t need to care whether they will get feedback, and in fact there will be no feedback.

Based on the above situation, all communication links in thresh will use this mode for communication: the message sender only needs to pass data without caring about callback, and the message receiver only needs to process data without returning processing results. This mode is easier to manage and restrict the communication across three terminals, and also makes native a complete data transfer station. Otherwise, native needs to process the feedback of results in addition to transmitting data. That is, data transfer Party – > data transfer Party – > data receiver is one-way.

Flutter dynamic framework thresh
However, not all communications do not need feedback. For example, the two terminal communication link bridge communicating with native needs to obtain the processing results of native after sending communication messages to native. In this case, simple and rough one-way communication will not directly meet the needs. However, if it is replaced by “full duplex” communication with callback, so that the result can be received on the same communication channel, it will destroy the original communication mode and increase the difficulty of communication management.

In order to solve the problem of communication feedback in the “half duplex” communication mode, we add an identifier to each communication requiring feedback on the transmitting party, and then cache the feedback processing method through the identifier; After the receiver completes the processing, it carries the identifier and passes the processing result to the original sender as a new message through another communication channel (in this new channel, the original data delivery and receiver will exchange identities), and the sender will find the processing method in the cache according to the identifier and execute the processing logic.

Flutter dynamic framework thresh

3.2.3 establish reliable message channel

The communication between JS and fluent is the cornerstone of the dynamics of fluent, and the success of the first communication is the primary condition for the successful establishment of communication.

Since all cross three terminal communications are “half duplex”, and the environmental preparation of JS and flutter are completely independent, if either party sends a message before the environmental preparation is completed, the party whose environment is not completed will not receive the message, which will affect all subsequent communications and lead to communication interruption or disorder.

In order to solve this situation, JS and flutter adopt the following strategies to ensure the smooth implementation of the first communication (hereinafter, a / b refers to either of JS and flutter):

  • A will send a notice to B immediately after the environmental preparation is completed;
  • If B is ready, it will immediately reply to a notice. After receiving the reply notice, a will mark that the environment of both parties has been established and can carry out subsequent communication;
  • If B is not ready, a will not receive any reply until B is ready. At this time, a / b identity exchange will return to step 1.

Flutter dynamic framework thresh

3.3 component update and event transmission

3.3.1 JS event triggering and transmission

After converting the event function in JS into ID, this ID will also be carried into the flutter together with the page name and node ID of the node. Finally, these three information will be packaged as an event function in the flutter.

When an event is triggered in fluent, this function will be triggered first. This function will send a message to JS carrying the page name, node ID, event ID and event parameters. After receiving the message, JS will first find the node that triggered the event according to the page name and node ID, then find the corresponding event in the node event pool through the event ID, pass in parameters and execute the event.

3.3.2. JS component update

The purpose of triggering events is mostly to update the content on the page. In JS, the basic unit of component update is user-defined components.

When a custom component triggers setstate(), it will be pushed into the update queue to wait for updates. The node will be de duplicated before entering the queue. 16ms after entering the first component in the queue, the queue will perform the update operation. Other components to be updated in the queue within 16ms will trigger the update together.

If the parent node does not exist in the queue, it will be updated from the parent node in turn. If the parent node does not exist in the queue, it will be updated first. This is because as long as the parent component exists in the queue, the child component will be updated; The purpose is to perform the least number of operations, but to update as many components as possible.

The component update refers to the component update diff algorithm of react, but due to the introduction of the concepts of fluent statefulwidget and statelesswidget, compared with the diff algorithm of react, thresh JS diff algorithm is coarse-grained.

The same thing between the two is that each node will be compared to ensure that the state of each node is correct and finally updated correctly.

The difference is that in addition to merging the attributes and states of nodes of the same type, react will also insert or delete newly created or deleted nodes in the old node array. The basic unit of operation and update is atomic components; And thresh JS will only focus on those nodes of the same type that remain before and after the update. After merging the attributes and states, it will directly discard the old node and retain the new node. Finally, the new node will replace the old node in the custom component to be updated, and send an update message to flutter using the data of the updated custom component – the basic unit of update is the custom component.

Flutter dynamic framework thresh

3.3.3. Flutter component update

The update message sent by JS consists of two parts: the page name to be updated, the update node ID and the JSON data of the update node. When flutter receives the update message sent by JS, it will first repeat the step of converting JSON to model to create modeltree & widgettree Then find the model to be updated in the cache through the updated page name and node ID.

Since the update takes the custom components in JS as the minimum unit, and each custom component will be created as statefulwidget in fluent, the following operations will be carried out after obtaining the old and new models:

  1. Merge the rendering data of newmodel, child node models and its newwidget into oldmodel;
  2. Update the oldwidget wrapped in the statefulwidget to newwidget through the state held by the oldmodel;
  3. After updating components through state, fluent will diff and re render the updated components to ensure that the page can display new content.

#4、 Engineering

##4.1 thresh architecture
The overall engineering architecture of thresh is shown in the figure below:
Flutter dynamic framework thresh

As shown in the above figure, from bottom to top, CI / CD + basic services + monitoring and reporting support thresh business, and the top is the architecture diagram.

  • X-ray is a production release platform developed by the company, which supports the construction, distribution, operation and maintenance of bundle packages.
  • At the top is the architecture flow chart of the overall thresh, including page development, DSL transformation, communication, etc., which is used to build pages and logic.

Although the thresh dynamic cross platform scheme has the advantages of high-performance rendering, consistency, development efficiency and zero cost access of front-end students in design, considering the future multi-party access and improving the efficiency of development and debugging, it has promoted the construction of infrastructure around thresh. The following is a brief introduction to the development period, debugging period and release period.

  • Development period
    Support plugin access, and provide a set of template projects for business access, which can quickly enter business development; In addition, thresh is compatible with TS and can integrate front-end development at a lower cost.
  • Commissioning period
    By supporting hotreload mode and second level compilation, the development and debugging efficiency can be greatly improved. In addition, the debugging panel + dynamic debugging ability can also greatly help to improve the debugging efficiency.
  • Release period
    Relying on the self-developed X-ray gray publishing system of manbang, it has the ability of minute level dynamic publishing, and can quickly support business and problem repair.

Flutter dynamic framework thresh

4.3. Thresh development integration

Thresh’s development and integration has formed a complete set of processes, including tripartite integration, multi business module access, development and debugging, etc., which involves many details. This is described in detail in the open source warehouse.

So far, thresh’s architecture design and development integration capabilities have been basically completed. Compared with other dynamic cross platform development frameworks, thresh has the following advantages:

  • Self defined DSL based on JS has strong expansibility and low learning cost
  • Multi terminal consistency, with a unified self rendering engine skia, better cross terminal compatibility and adaptation
  • It supports hot reload, which is convenient for development and debugging and second level compilation
  • Support component level UI refresh, excellent experience

    • Provide debugging panel during development to facilitate development

5、 Concluding remarks

The basic principle of building a fluent application through JS is not complex, mainly including data processing in JS, data conversion in fluent, and realizing the flow channel of data in JS and fluent. Such schemes are all similar, such as mxshuttle and meituan takeout mtshuttle. However, this scheme seems to be relatively weak at present, which deviates from the original intention of flutter’s cross platform involvement.