Xianyu’s Flutter Image Optimization: From Native Code to Advanced Technology

Image loading is the most common and basic feature of apps, but more importantly, it is also one of the determining factors of user experience. This function although seemingly simple, actually introduces many technical challenges. This article introduces the attempts made by the Xianyu technical team at Flutter image optimization, and shares the technical details of Xianyu’s typical image processing solutions. Hopefully, this article can provide some inspiration for dealing with this function.

Xianyu in the Old Days

  • Image loading consumes too much capacity.
  • Repeated local resources occur after Flutter is introduced, but their utilization is low.
  • The efficiency of Flutter’s native image loading is low under a hybrid solution.

In response to these problems, Xianyu has been optimizing the image framework since the initial version of the Flutter service. Factors from the optimization of native code in the beginning to the advanced technology of external textures later on, and from memory usage to package size, will all be described in this article. I hope the ideas and methods about the optimizations can bring you some inspiration.

The Native Mode

The initial version of Xianyu image processing is a purely native solution. If you do not want to modify much of the underlying logic, the native solution is definitely the simplest and most economical one. The functional modules of the native solution are as follows:

However, if you modify nothing before using it, you may find that its performance is not as expected. Then, what optimizations can be done at this early stage?

Set Up an Image Cache

  • The number of cached images, which can be set through maximumSize, with a default value of 1,000.
  • The size of the cache, which can be set through maximumSizeBytes, with a default value of 100 MB. Compared with setting a limit on the number of images, setting the cache size is more in line with our final expectations.

By setting the ImageCache size, you can make full use of the caching mechanism to accelerate image loading. In addition, Xianyu has made two major optimizations:

Adapt to Low-end Mobile Phones

The Disk Cache

In this effort, we did not create a disk cache on our own for the specific architecture implementation. Instead, we chose to reuse the existing capability. First, we exposed the disk caching function of the native image loading framework through an interface. Then, we grafted the native disk caching capability onto the Flutter layer by means of bridging. When an image is loaded to Flutter and no hits can be found in the memory, another search attempt is made in the disk cache. If this attempt also has no hits, a network request is raised.

By increasing the disk cache capacity, the efficiency of Flutter image loading can be further improved.

Configure CDN Optimization

Image Resizing According to the Size of Display Window

Compress the Image Appropriately

Image Formats

For the preceding reasons, the Xianyu image framework implements a set of CDN size matching algorithms on the Flutter side. With these algorithms, the requested image will be automatically matched to the most appropriate size and compressed according to the actual display window. If the image format allows, you should always try to convert the image format into webp before delivering it, which maximizes the efficiency of transmitting CDN images.

Other Optimizations

Image Preloading

The Reuse of the Element

Long List Optimization

The implementation idea of Flutter ListView is different from the implementation idea of the native solution. Its most significant highlight is that it introduces the viewPort concept. In this concept, any part that goes beyond viewPort will be forcibly reclaimed.

Based on this principle, we have put forth two suggestions:

1) Cell splitting

In this practice, avoid large cells as much as possible to minimize the performance loss during frequent cell creation. This is because, the cell size affects not only the image loading process, but also affects other components such as text processing and video processing which are prone to performance problems caused by overly complex cell lines.

2) Rational use of the buffer

With ListView, you can set the size of the preloaded content by setting cacheExtent. Preloading can accelerate view rendering. However, the cacheExtent value must be appropriate. A larger value does not suit all cases, because the workload of the overall memory of the page increases as the preloading cache size expands.

Shortcomings of the Native Solution

1) Unable to reuse the native image loading capability

2) Poor memory performance

Integrate the Native Capabilities

External Textures

In the current phase, we have developed texture plug-ins for Flutter and the native solution based on the shared-context solution. By adopting this solution, Flutter can obtain and display images loaded by the native image library through sharing textures. To implement the channel for texture sharing, we made in-depth customizations on the engine layer. The detailed process is as follows:

This solution not only integrates the native and Flutter image architectures, but also optimizes image loading performance throughout the process.

The application of external textures is a big leap forward in the image solution of Xianyu. With this technology, we can reuse the local capabilities of the image processing solution and also implement external textures for the video capability. This avoids heavy repetitive construction and improves the performance of the entire app.

Multi-Page Memory Optimization

So, we are considering whether images on the page at the bottom of the page stack can be directly reclaimed?

Driven by this thought, we launched another round of optimization on the image architecture. All images in the entire image framework monitor the changes of the page stack. When an image finds itself not at the top of the stack, it automatically reclaims the corresponding image textures to release resources. This solution prevents the memory usage of images from increasing linearly with the number of pages. The following figure shows how it works:

Note that the page location judgment at this stage requires the additional interface provided by the page stack (hybrid stack specifically). The coupling degree between systems is high.

Unexpected Gain: The Package Size

Other Optimizations — PlaceHolder Enhancement

In terms of core functions, we introduced the following loading states:

  • Not initialized
  • Loading
  • Loaded

For different states, you can control the display logic of PlaceHolder in a fine-grained manner.

Architecture

Shortcomings of the Integrated Solution

The engine has been changed

Channel performance still has room for optimization

Excessive Coupling

Clean & Efficient

Non-intrusive External Textures

In fact, Flutter provides an official external texture solution.

In addition, textures used by the native solution and textures displayed on the Flutter side point to same objects at the underlying layer and do not generate additional data copies. This ensures highly efficient texture sharing. Then, you may be wondering, why did the Xianyu team implement an independent external texture solution based on shared-context? Prior to version 1.12, the official iOS external texture solution had performance bugs. In earlier versions, during each rendering process regardless of whether textures were updated or not, CVPixelBuffer was frequently obtained, resulting in unnecessary performance loss due to lock-related loss in the process. This problem had been fixed in version 1.12 (with the official commit address on GitHub) so that the official solution is now sufficient for the needs. In this context, we used the official solution again to implement the external texture function.

Independent Memory Optimization

Each page uses the router object obtained from context as the identifier to reorganize all image objects on the page. Images that obtain the same router object are identified as the images on the same page. In this way, you can manage all images on a page basis. Overall, we simulated the virtual page stack structure by using the LRU algorithm, so that image resources for the page at the bottom of the stack can be reclaimed.

Other Optimizations

Highly Reuse of Channels

Efficient Texture Reuse

Moreover, we migrated the mapping between textures and requests to the Flutter side so that the reuse of textures can be done on the shortest path, further reducing the communication workload of the bridge.

Architecture

Optimization Results

Memory Optimization

Multi-Page Stack Memory Optimization

As you can see, the optimization of the multi-page stack enables enhanced control of the memory usage of multiple Flutter pages.

Package Size Reduction

Follow-up Plan

Original Source:

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.