Storing Massive Image Metadata of Tmall and Taobao
By Wangde and Zhiqian
1) A Storage Cost Surge of Image Space Databases
Image Space is a free service provided by Taobao intelligent image center for merchants to store and manage images. Due to the huge number of images accumulated by Taobao and Tmall users (just imagine how many images are uploaded to Taobao and Tmall every day by merchants and consumers) as well as the astonishingly increasing speed, Image Space undergoes immense pressure in terms of storage capacity and write performance. Especially before every Double 11, merchants update a large number of stock-keeping units (SKUs) that leads to a sharp increase in image data.
Every day, products and comments on Taobao and Tmall contribute to adding a large number of new images.
Once, before the Double 11 Shopping Festival, when most databases of Alibaba still ran on the InnoDB engine, we were evaluating the potential risks related to the shopping festival. When an Image Space developer asked whether the databases had sufficient disk capacity, we confidently said, “Take it easy. We doubled the storage four months ago.” However, we were soon proved wrong. In less than five months, the amount of data accumulated was more than what we estimated for six or seven years. Also, as the daily growth soared, the expanded storage was increasingly becoming insufficient.
2) Solution: Scale-out or Change the Engine?
Why Change the Engine?
To address the aforementioned challenge, the most simple and direct way is to scale-out the storage, which is less risky. But, this measure doesn’t solve the challenge once and for all. With the current speed of data growth, it is inevitable to scale out the storage again and again in the future. Additionally, it is unacceptable to double the cost whenever space runs out.
The other way to address this challenge is to change the engine. At that time, Alibaba had just developed a proprietary engine named X-Engine. X-Engine focuses on high performance and low cost. Compared with the B+ Tree-based storage, such as InnoDB that suffers from a high space waste on data pages, Log-Structured Merge-Tree-based (LSM Tree-based) X-Engine features compact data storage and effective space utilization. In addition, prefix-compression technology is introduced to further reduce the space usage of compact data.
In-place updates are not necessary for data blocks in X-Engine. Therefore, X-Engine conveniently compresses data using universal compression algorithms, such as zlib, zstd, and snapy. All the data in lower levels of LSM-Tree is compressed by default. Based on the results of many comparison tests, X-Engine chooses the zstd compression algorithm as the default, while retaining support for other algorithms. Additionally, backend compaction keeps deleting invalid entries and reclaiming space. LSM Tree update and deletion operations write new entries, and old entries are regarded as invalid when they are no longer needed.
Owing to the preceding technical features, X-Engine supports such enormous space-saving that 7 times less space was saved after migrating Image Space from InnoDB to X-Engine as shown in the following figure:
How to Reduce Costs by 7 Times?
Now, you may wonder how we can save so much on costs through data migration from InnoDB to X-Engine.
- Firstly, InnoDB uses B+ Tree to index data. Along with data writes, nodes of B+ Tree are frequently split and merged. As a result, fixed-length data pages are often half-full, resulting in space waste. Comparatively, update and deletion operations in X-Engine are all append-writes to the memtable in the memory, which does not modify any data on the disk. Therefore, this static data can be stored compactly, and no space reservation is required for future writes. Consequently, space is well utilized. Although append-writes produce redundant data of different versions, backend compaction in X-Engine timely clears the data of useless versions.
- Secondly, the Image Space database stores a large amount of image metadata, for example, user IDs and image URLs, among which adjacent data has high similarity. For example, the same user ID may correspond to multiple image URLs, and in such a case, the prefixes of the URLs would be very similar. The prefix compression mechanism of X-Engine ensures that only one copy of the same prefix is stored for adjacent keys. Therefore, after prefix compression, the secondary index that contains the image metadata occupies a small amount of space.
- Lastly, although keys in master tables cannot be compressed by using prefix compression, universal compression algorithms can also achieve an ideal compression performance when handling image metadata that contains a lot of similar characters, such as URLs. Though InnoDB supports data page compression and has a good compression performance for static data, B+ Tree keeps splitting and merging written data so that the used space quickly expands. By contrast, static data pages in X-Engine eliminate this problem.
Excellent Performance
Image Space is a frequently used application at Taobao and Tmall. For this reason, if the performance does not meet the requirements, X-Engine will not be adopted. Thanks to the lightweight write mechanism of LSM, the advantageous write performance of X-Engine in conjunction with the introduction of the group commit and pipeline transaction processing mechanisms greatly enhanced the concurrency of write processing. On the other hand, LSM is weak in processing read requests, while the tiered structure and the multi-version data generated by append-writes extend the query path of read requests. To address this problem, X-Engine has made many optimizations, such as multi-granularity caches (memtable, block cache, and row cache), bloom filter, and range scan filter (Surf and SIGMOD’18). These improvements effectively reduce the number of point queries, range scans, and prefetches of asynchronous I/O. In a few words, we strove to make X-Engine a storage engine with balanced read-write performance and outstanding cost-effectiveness. For read-write optimizations of X-Engine, read this article: Detailed Explanation of X-Engine SIGMOD.
Verified by database administrators (DBAs) and business developers, X-Engine fully meets the business requirements for read-write performance and latency. Shortly, Taobao Image Space databases were migrated to X-Engine, leading to reduced storage costs.
3) X-Engine: Suitability for Businesses Scenarios
X-Engine with tiered storage architecture is ideal for business workloads with the following characteristics.
- Businesses having large data volumes and are cost-sensitive: After the migration from traditional InnoDB to X-Engine, the used storage space can be reduced by 2 to 10 times, which varies according to data characteristics. Many businesses do not need to split databases and tables after the migration as a single database can hold nearly 10 TB of data. For example, the X-Engine application in DingTalk.
- Businesses showing distinct characteristics on access timing: For example, most of the read and update operations are performed on recently written data while historical data is seldom accessed, such as the Taobao transaction database. On the one hand, X-Engine caches newly written data by using efficient memory-based indexing, which provides extremely high access performance. On the other hand, it stores rarely accessed historical data in disks, which provides slightly inferior access performance. For example, the X-Engine application in the Taobao transaction database.
