One Solution to Improve Flutter Memory Utilization (Knowledge Sharing)

By Jingshu, from Xianyu Technology Team

Background

The images Xianyu uses are based on a proprietary external texture:

• For Android, SurfaceTexture is created and registered to the Flutter engine with FlutterJNI. Then, the texture ID is returned to the Flutter application layer, which uses Texture Widget and the texture ID to display the image texture.
• Texture data is on the Android side. The image texture is written into SurfaceTexture through OpenGL. Then, it is passed into the application layer through the shared memory in the Flutter engine. Finally, it moves to Skia for rendering.

Existing Problems

The texture data in the Flutter application layer is not cached, thus requiring Bitmap data to be rendered into texture data and delivered to the Flutter application layer each time. Since the loaded Native images and image library of Flutter are cached separately in memory, plenty of memory is consumed. The Flutter image caches refer to stored local resource images. However, most of the images in Flutter pages are external texture images downloaded online. Thus, the cached resource utilization is very low.

Analysis

Put aside the technical implementation, what is an ideal solution to solve the preceding three problems?

• For the lack of texture cache, texture memory cache can be added to the application layer.
• When the upper application layer has cached textures, the Native-side Bitmap memory cache can be removed, with only a disk cache of image resources remaining.
• The texture cache and the cache of Flutter ImageCache are merged as one memory cache in the app to avoid wasting memory resources.

An ideal solution would see one memory cache exist in the app that could cache the textures and image data loaded by the Flutter Image Widget.

Solution

ImageCache is officially attached and can’t be removed. Image Widget is also used in the Xianyu app. Now, the ideal solution would see cache texture data in ImageCache, from which textures are acquired.

The following figure shows the current Flutter image loading logic and the process of image caching.

The chart above shows the Flutter image loading using ImageCache.putIfAbsent to retrieve the cache. If it fails, the system uses the passed-in loader to establish the corresponding ImageStreamCompleter for the image loading.

If ImageCache.putIfAbsent works, putIfAbsent returns ImageStreamCompleter directly, which contains imageInfo. Then, Image Widget renders the ui.Image of imageInfo.

Solution 1: Extend the ImageCache Features for Texture Caching

For ImageCache, putIfAbsent is the only way to retrieve caches for externals.

In the beginning, it is assumed to establish the corresponding key, loader, and ImageStreamCompleter according to the parameters of ImageCache.putIfAbsent, and then use putIfAbsent to obtain the cache.

The attempt failed. As shown in the following figure, when the image is successfully downloaded and decoded, the listener method is called back. With the listener method, images will be stored in the cache queue of ImageCache.

The listener callback has two parameters with ui.Image stored in ImageInfo.

It’s impossible to establish ui.Image at the application layer because it is set to the application layer after the underlying layer of the Flutter engine completes image decoding. Besides, the application layer cannot actively set values. As a result, the imageSize value cannot be computed in the listener or be stored in the cache.

Solution 2: Customize ImageCache

Since the cache queue of ImageCache is private, putIfAbsent is the only way to store data in the queue. Therefore, it's feasible to start with the source code of ImageCache. ImageCache can be customized by modifying the code, and its functions can be extended.

ImageCache Replacement

The code of Flutter ImageCache cannot be modified directly, so the source code of ImageCache is copied. Then, ImageCache of PaintingBinding is replaced with a customized ImageCache.

As is shown, the createImageCache method can be observed in Flutter PaintingBinding. The method can be rewritten to get a customized ImageCache by inheriting WidgetsFlutterBinding. It is also feasible to set various cache sizes for ImageCache.

Function Extensions of ImageCache

A new texture caching method is defined to avoid modifying the ImageCache code as much as possible, which aligns the logic of putIfAbsent. The core code logic is listed below:

This method is mainly implemented by referring to the logic of putIfAbsent. You need several key extensions to cache textures into ImageCache:

1. TextureCacheKey is the only key to identify textures based on their width, height, and URL.
2. TextureImageStreamCompleter belongs to texture management, which inherits ImageStreamCompleter and internally stores callbacks of texture data and successful downloads. When the cache is hit, the hit cache is returned to the application layer, and texture ID is obtained and rendered by Texture Widget.
3. If the cache is not hit, the passed-in loader establishes TextureImageStreamCompleter and executes the logic of texture loading. At the same time, a listener callback is established and registered in TextureImageStreamCompleter.
4. When the texture is loaded successfully, listener callback will be executed. By doing so, the texture size is calculated and put into the cache queue for cache size check. If the size exceeds the maximum value, the longest unused texture will be eliminated.

Note: Images are normally Dart objects and will be automatically recycled by Dart VM. However, the real data of a texture object is located in the shared memory of the Flutter engine. Therefore, it is necessary to manage the release of the texture manually. Based on the reference counting of textures, textures will be truly released; only when no widget holds the texture and the reference counting is 0.

Similarly, when the upper-layer Texture Widget is disposing, the interface provided by ImageCache will also be called to see if the current texture is cached or being used. Only when no texture is cached or being used, will the texture be released.

Effects

Take the search results page as the test page. Many large pictures and various duplicate small label pictures are displayed on the page. The Huawei Glory 20 is used here to test the physical memory usage before and after optimization.

The test procedure is listed below:

1. Open the app and search for the same keyword to enter the search results page
2. Wait for 10 seconds and browse 100 pieces of data
3. Stop the operation

During this period, the physical memory is sampled in second and lasts for 100s total. The following data are obtained.

The blue curve represents the memory usage before optimization, and the orange curve represents the memory usage after optimization. In the beginning, the memory usage is generally the same. The decrease in memory usage during browsing is caused after GC recycles the app memory. Generally, the total memory usage after optimization is less than before because glitches caused by GC are reduced.

Prospects

The solution above implements the goal of one memory cache in one app. It also stores the textures and Flutter images, saves memory space, and improves memory usage. However, it still intrudes into the ImageCache source code. Thus, additional work is necessary to upgrade the Flutter engine and maintain the code.

Since Flutter uses putIfAbsent to load native images with the original size, one image may take up several MB in the app. As a result, the function of large image monitoring is added to putIfAbsent. When the size of the loaded image exceeds 2 MB, the data, including the URL, usage information, and size of the image, will be reported. By doing so, several cases of improper image use are found. For example, original images are loaded using Image.network, or Image.asset loads a large image from local resources.

Original Source:

One Solution to Improve Flutter Memory Utilization (Knowledge Sharing)