Who Said All Redis Data Must Be Stored in Memory? Explore Redis Hybrid Storage Instances

Image for post
Image for post

By Zhang Youdong (Linqing)

The Redis hybrid storage instance is a cloud database product independently developed by Alibaba Cloud and is compatible with Redis protocols and features. The hybrid storage instance breaks the convention that all Redis data must be stored in the memory. Now, use disks to store full data and cache hot data to the memory for balancing access performance with storage costs.

Architecture and Features

Image for post
Image for post

Command Compatibility

Hybrid storage is compatible with the vast majority of Redis commands. Compared to the native Redis, the following commands are not supported or their functions are limited. These commands are not supported mainly for performance considerations. If your business does need to use these commands, submit a ticket to apply for them.

Image for post
Image for post

Selection Guide — Scenarios

Image for post
Image for post

Selection Guide — Specifications

While selecting hybrid storage instances, select an appropriate memory configuration and disk configuration. The disk determines the total amount of data that can be stored, and the memory determines the total amount of hot data that can be cached. When an instance is created, an appropriate CPU resource configuration is selected based on the storage specifications. Currently, it doesn’t allow specifying the number of CPU cores.

For example, an instance with 64 GB memory and a 256 GB disk stores up to 256 GB of data (subject to the total size of the physical files of the KV storage engine), 64 GB of which can be cached in the memory.

Memory Selection Recommendation: To ensure maximum compatibility with the Redis native access protocol, Redis hybrid storage requires that all keys reside in the memory, while values are automatically stored in the memory or disk according to their access frequency. Therefore, the memory space must be sufficient to store all keys and the corresponding metadata.

Image for post
Image for post

Disk Selection Recommendation: Since Redis data is stored in the KV storage engine, each key will have additional metadata, increasing the required storage space. Therefore, reserve a proper margin while selecting the disk space. We recommend choosing a disk capacity of 1.2 to 1.5 times greater than the actual storage needs.

Performance Metrics

The performance of Redis hybrid storage is related to the memory-to-disk ratio and the service access level. According to the selected specification and service access mode, the performance of simple SET and GET methods may fluctuate between several thousand and tens of thousands. In the best case, all access requests hit the memory, and the performance is basically consistent with that of the Redis memory edition. In the worst case, all access requests need to be read from the disk.

Test Scenario: There are 20 million keys with values of 1 KB, and 25% of hot keys are stored in the memory. The data of the GET request test is as follows:

Image for post
Image for post

Scenarios

Live Streaming

Live streaming generates a large amount of hot data, most of which comes from popular live-streaming rooms. Use ApsaraDB for Redis hybrid storage instances to store data from popular live-streaming rooms in the memory and data from unpopular rooms in disks. In this way, make full use of the limited memory space.

E-commerce

E-commerce applications have a large volume of commodity data, but newly released products are the most frequently accessed, while older products are less frequently accessed. By using Redis hybrid storage instances, easily break through the memory capacity limit by storing a large amount of commodity data on disks. During normal business requests, active commodity data is gradually cached into the memory to meet business needs at the lowest possible cost.

Online Education

In online education scenarios, data such as courses, question banks, and conversation records between teachers and students are frequently accessed only for popular courses and the latest question banks. By using Redis hybrid storage, users store a large amount of course information on disks. Active course and question bank information is transferred to and stored in the memory. This ensures the access performance of frequently accessed data and balances performance and storage costs.

Other Scenarios

Other scenarios with significant access frequency variations and lower performance requirements may also use Redis hybrid storage to reduce storage costs.

Frequently Asked Question

The disk is not full, but the system reports an out of memory (OOM) error when the memory is full.

ApsaraDB for Redis

ApsaraDB for Redis is a stable and reliable database service with remarkable performance and support for auto-scaling. It is built on Apsara Distributed File System and full SSD high-performance storage and supports the primary-secondary and cluster-based high-availability architectures.

Original Source:

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store