From native to black Technology: what’s the experience of flutter image optimization( Welfare at the end of the article)

Time:2021-5-10

Introduction:Alimei Guide: image loading is the most common and basic function of app, and also one of the factors influencing user experience. There are many technical problems behind the seemingly simple image loading. This paper introduces the technology team of Xianyu’s attempt to optimize the image of flutter, and shares the technical details of the typical image processing scheme of Xianyu, hoping to bring some inspiration to you.

those years

As early as when Xianyu used flutter, the image is our core concern and key optimization function. The quality of image display experience will have a huge impact on the user experience of idle fish. Have you ever met:

  • Too much memory for picture loading?
  • After using flutter, the local resources are duplicated and the utilization rate is not high?
  • Is the loading efficiency of native image of flutter low in hybrid scheme?

In view of the above problems, since the first version of flutter was launched, the optimization of image frame by Xianyu has never stopped. From the original optimization at the beginning to the external texture of black technology at the back; From memory occupation to packet size; The text will be introduced one by one. I hope the optimization ideas and means can give you some inspiration.

native mode

From a technical point of view, in fact, simply speaking, the goal of image loading is to maximize the efficiency of loading – to load as many images as possible as quickly as possible with as little resource cost as possible.

The first version of the idle fish image is basically a pure native solution. If you don’t want to change a lot of underlying logic, the native solution is definitely the simplest and most economical one. The functional modules of the native scheme are as follows:

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

If you do nothing directly, then you may find that the effect is not as good as you expected. So if we start with the original scheme, what specific optimization methods do we have?

Set picture cache

That’s right. It’s cache. For image loading, the most imaginable solution is to use cache. Firstly, the components of native image support custom image caching, and the specific implementation class is imagecache. The setting dimension of imagecache has two directions:

  • Number of cached pictures. Set by maximumsize. The default is 1000.
  • The size of the cache space. Set by maximumsizebytes. The default value is 100m. Compared with the limit of the number of sheets, the size setting method is more in line with our final expectations.

By reasonably setting the size of the image cache, we can make full use of the cache mechanism to speed up the image loading. Not only that, idle fish has also made two additional important optimizations at this point

Low end mobile phone adaptation

After going online, we received feedback from online public opinion and found that it is not optimal to set the same cache size for all models. Especially when the large cache is set on the low-end machine, not only the experience will become worse, but also the stability will be affected. Based on the actual situation, we implement a flutter plug-in that can obtain the basic machine information from the native side. Through the information obtained, we set different cache strategies according to the configuration of different mobile phones. Reduce the size of the image cache on the low-end machine and enlarge it on the high-end mobile phone. In this way, the optimal cache performance can be obtained on different mobile phones.

Disk cache

Students familiar with app development know that mature image loading frameworks generally have multi-level caching. In addition to the common memory cache, a file cache is usually configured. From the loading efficiency, it is through space for time to improve the loading speed. In terms of stability, this will not take up too much valuable memory resources, resulting in oom. Unfortunately, the image loading framework of flutter does not have an independent disk cache. So we extend the disk cache capability on the basis of the native scheme.

In the specific implementation of the architecture, we do not completely own a disk cache. Our strategy is to reuse existing capabilities. First of all, we expose the disk caching function of native image loading framework through the interface. Then, the native disk cache capability is grafted to the flutter layer by bridging. When the image is loaded on the flutter side, if the memory fails to hit, it goes to the disk cache for a second search. If they all miss, they will go through the network request.

By increasing the disk cache, the efficiency of image loading is further improved.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Set up CDN optimization

CDN optimization is another very important image optimization method. The main efficiency improvement of CDN optimization is to minimize the size of transmitted pictures. Common strategies include:

Crop based on display size

In short, the real size of the image you want to load may be larger than the size of your actual display window. So you don’t have to load the full image, you just need to load an image that can cover the size of the window. In this way, the size of the transmitted image can be minimized by cutting out unnecessary parts. From the end side point of view, it can improve the loading speed and reduce the memory consumption.

Compress image size appropriately

Here is mainly according to the actual situation to increase the proportion of image compression. Without affecting the display effect, the image size is further reduced by compression.

Picture format

It is recommended to give priority to webp format, so the image resources are relatively small. Flutter natively supports webp (including dynamic graph). In particular, we emphasize that webp map is not only much smaller than GIF, but also has better support for transparency. Webp is an ideal alternative to GIF.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Based on the above reasons, the idle fish image framework implements a set of CDN size matching algorithm on the flutter side. Through this algorithm, the request image will automatically match to the most appropriate size and compress appropriately according to the actual display size. If the image format allows, the image should be converted to webp format for distribution. In this way, the transmission of CDN images can be as efficient as possible.

Other optimizations

In addition to the above strategies, flutter has some other means to optimize the performance of images.

Image preloading

If you want to display the image as fast as possible, the official also provides a set of Preloading Mechanism: precache image. Precache image can load the image into memory in advance, and it can be used in seconds.

Element reuse optimization

In fact, this is a general optimization scheme of flutter. Duplicate didwidgetupdate scheme, by comparing the description of the image before and after the two widget is consistent, to determine whether to re render the element. This can avoid the same image, unnecessary repeated rendering.

Long list optimization

In general, listview is the most common scrolling container of flutter. The performance of listview directly affects the final user experience.

The implementation idea of flitter’s listview is different from that of native. Its biggest feature is the concept of a viewport. Parts beyond the viewport will be forcibly recycled.

Based on the above principles, we have two suggestions:

1) Cell splitting

Try to avoid the appearance of large cells, which can greatly reduce the performance loss in the process of frequent cell creation. In fact, the impact here is not just the image loading process. Text, video and other components should also avoid the performance problems caused by too complex cell.

2) Rational use of buffer

Listview can set the size of the preloaded content by setting the cacheextent. The speed of view rendering can be improved by preloading. But this value needs to be set reasonably, not the larger the better. Because the larger the preload cache, the greater the pressure on the page‘s overall memory.

The shortcomings of the scheme

Here we need to point out objectively: if it is a pure flutter app, the native solution is perfect and sufficient. However, from the perspective of hybrid app, there are two defects as follows:

1) Unable to reuse native image loading capability

There is no doubt that the original image scheme is a completely independent image loading scheme. For a hybrid app, the native scheme and the native image framework are independent of each other, and their capabilities cannot be reused. For example, CDN cutting & compression capabilities need to be built repeatedly. In particular, native’s unique image decoding capabilities make it difficult to use flutter. This will result in inconsistent support for image formats within the scope of the app.

2) Insufficient memory performance

From the perspective of the app as a whole, in the case of adopting the native image scheme, we actually maintain two large cache pools: one is native image cache, and the other is the image cache on the flutter side. The two caches cannot communicate with each other, which is undoubtedly a huge waste. In particular, the peak memory performance of memory is under great pressure.

Get through to native

After several rounds of optimization, the original solution has achieved great performance improvement. However, the memory water level of the entire app is still relatively high (especially the IOS side). The pressure of reality forces us to continue to optimize the picture frame more deeply. Based on the analysis of the disadvantages of the native scheme, we have a bold idea: can we fully reuse the image loading ability of native?

External texture

How to get through the image capabilities of flutter and native? We think of external textures. External texture is not flutter’s own technology, it is a common performance optimization method in the field of audio and video.

At this stage, we implemented the texture extrapolation of flutter and native based on the shared context scheme. Through this scheme, flutter can get the loaded images from native image library and display them by sharing textures. In order to realize the texture sharing channel, we made deep customization to the engine layer. The detailed process is as follows:

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

This scheme not only gets through the image architecture of native and flutter, but also optimizes the performance of image loading in the whole process.

The external texture is a big leap of the idle fish image scheme. Through this technology, we not only realize the local ability reuse of image scheme, but also realize the texture external connection of video ability. This avoids a lot of repeated construction and improves the performance of the entire app.

Multi page memory optimization

This optimization strategy is really forced out. After analyzing the online data, we find that the flutter page stack has a very interesting feature: in the case of multiple page stacks, the underlying pages will not be released. Even in the case of very tight memory, recycling will not be performed. This leads to a problem: as the number of pages increases, memory consumption increases linearly. The highest proportion here is the proportion of image resources.

Is it possible to recycle the images in the page directly when the page is at the bottom of the page stack?

Driven by this idea, we have carried out a new round of optimization on the image architecture. The pictures in the whole picture frame will monitor the changes of the page stack. When the party finds that it is not at the top of the stack, it automatically recycles the corresponding image texture to release resources. This scheme can make the memory size occupied by the image will not continue to grow linearly with the number of pages. The principle is as follows:

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

It should be noted that in this stage, the page stack (specifically, the hybrid stack) needs to provide additional interfaces to determine the position of the page. The coupling between systems is relatively high.

Windfall: package size

After getting through the native and flutter side image frames, we found an unexpected gain: native and flutter can share local image resources. In other words, we no longer need to keep one copy of the same image resource on the flutter side and one copy on the native side. This can greatly improve the reuse rate of local resources and reduce the overall packet size. Based on this scheme, we implement a set of resource management functions, the script can automatically synchronize the local image resources of different ends. In this way, the utilization of local resources is improved and the packet size is reduced.

Other optimizations — placeholder enhancement

The native image has no function of placeholder. If you want to use native solutions, you need to use fadeinimage. We have a lot of customization for the scene of idle fish, so we have implemented a set of placeholder mechanism ourselves.

In terms of core functions, we introduce the concept of loading state, which is divided into two parts

  1. uninitialized
  2. Loading
  3. Loading complete

According to different states, the display logic of placeholder can be fine-grained controlled.

Overall structure

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

The shortcomings of the scheme

After all, I changed the engine

With the continuous development of idle fish business, the cost of engine upgrade is something we must consider. Whether we can achieve the same function without changing the engine is our core requirement (PS: I admit we are greedy).

There is still room for channel performance optimization

The scheme of external texture needs to communicate with native’s ability through bridge. This includes the transmission of picture request and the synchronization of various states of picture loading. Especially when listview is sliding rapidly, the amount of data sent through the bridge is considerable. In the current scheme, the bridge will be called separately when each picture is loaded. In the case of a large number of pictures, this will obviously be a bottleneck.

Too much coupling

When implementing the picture recycling scheme, the current scheme needs the stack to provide the interface whether it is at the bottom of the stack. It is difficult to abstract an independent and clean image loading scheme.

Clean & Efficient

The time has come to 2020. With the gradual deepening of our understanding of the basic capabilities of flutter, we have achieved a better picture framework for the overall scheme.

Noninvasive external texture

Can the external texture not need to modify the engine? The answer is yes.

In fact, flutter provides the official external texture scheme.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Moreover, the texture of native operation and the texture displayed on the flutter side are the same object at the bottom, and no additional data copy is generated. This ensures that texture sharing is efficient enough. So why did Xianyu implement a set based on shared context before? Before version 1.12, the external texture scheme of official IOS had performance problems. Cvpixelbuffer will be acquired frequently in every rendering process (whether the texture is updated or not), resulting in unnecessary performance loss (locking loss in the process). This problem has been fixed in version 1.12 (official commit address), so that the official solution is enough to meet the requirements. In this context, we re enable the official scheme to achieve external texture function.

Independent memory optimization

As mentioned before, the old version of image resource recovery based on page stack needs interfaces that strongly depend on the stack function. On the one hand, there are unnecessary dependencies, and more importantly, the overall scheme cannot be independent into a general scheme. In order to solve this problem, we study the bottom layer of flutter deeply. We found that the layer layer of flutter can sense the change of page stack stably.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Then, each page reorganizes all the image objects in a page by using the router object obtained from context as the identifier. All the identifications obtained from the same router object form the same page. In this way, we can manage all the pictures by page. On the whole, the algorithm of LRU is used to simulate the virtual page stack structure. In this way, the image resources of the bottom page of the stack can be recycled.

Other optimizations

High multiplexing of channels

First, we aggregate the image requests in a frame, and then pass them to native’s image loading framework in a channel request. This avoids frequent bridge calls. Especially in the scene of fast rolling, the optimization effect is particularly obvious.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Efficient texture reuse

After using external texture for image loading, we find that multiplexing texture can further improve the performance. Take a simple scene. We know that in the e-commerce scene, there are often pictures such as labels and base maps on the product display. This kind of picture often appears a lot of repetition on different commodities. At this time, the rendered texture can be directly reused to different display components. This can further optimize the GPU memory consumption and avoid repeated creation. In order to manage texture accurately, we introduce reference counting algorithm to manage texture reuse. Through these schemes, we realize the efficient reuse of texture across pages.

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

In addition, we move the mapping relationship between texture and request to the flutter side. In this way, texture reuse can be completed in the shortest path, which further reduces the communication pressure of the bridge.

Overall structure

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

Optimization effect

As the latest version is still grayscale, the specific data will be described in detail in the following article. The subordinate data is mainly scheme 2.

Memory optimization

By opening up native, compared with the first online version, the above rate of IOS is reduced by 25% without changing the display effect, and the user experience is significantly improved.

Memory optimization of multi page stack**

The memory optimization of multi page stack has obvious effect on memory optimization in multi page scenario. We did a limit test, and the results were as follows (test environment, non idle fish APP)

From native to black Technology: what's the experience of flutter image optimization( Welfare at the end of the article)

It can be seen that the optimization of multi page stack can control the memory occupation of multi flutter pages better.

Packet size reduction

By accessing the external texture, the local resources are reused better, and the packet size is reduced by 1m. In the early stage, when idle fish access to flutter, they will take the transformation of the existing page as the starting point. Resource duplication is serious, but with more and more new business of flutter. There are fewer and fewer duplicate resources for flutter and native. The effect of external texture on packet size has gradually weakened.

Follow up plan

This is an endless journey, and our optimization of idle fish images will continue. In particular, our latest program, limited space, this article is only a preliminary introduction. More technical details, including test data, we will write a special article later to continue to introduce you. After the improvement of the scheme, we will gradually open source.

More related content:click here
Benefits:
1.click here (e-Book)
2.click here (physical books and around idle fish)

Recommended Today

What is “hybrid cloud”?

In this paper, we define the concept of “hybrid cloud”, explain four different cloud deployment models of hybrid cloud, and deeply analyze the industrial trend of hybrid cloud through a series of data and charts. 01 introduction Hybrid cloud is a computing environment that integrates multiple platforms and data centers. Generally speaking, hybrid cloud is […]