Secret Weapon | How AnalyticDB Strongly Supports Double 11

Preface

AnalyticDB: Cloud-Native Data Warehouse

AnalyticDB Fully Supports Double 11

The Challenges of AnalyticDB

1. Alibaba Group’s Core Transaction Procedure

2. More Production and Operation Stages

3. More Import Tasks

4. More Writing Service With High Throughput

5. Online and Offline Hybrid Loads

The Latest Key Technologies of AnalyticDB

Compute-Storage Separation
Hot/Cold Data Hierarchy

Definitions of Hot and Cold Storage

create table event(
id bigint auto_increment
dt datetime,
event varchar,
goods varchar,
package int
...
) distribute by hash(id)
partition by value(date_format(dt, '%Y%m%d')) lifecycle 365
storage_policy = 'MIXED' hot_partition_count = 7;

Automatic Migration of Hot and Cold Data

  • When data has accumulated to a certain extent, the build task is automatically scheduled internally. Snapshots are created for real-time data, and the data is collated. Then, new historical partitions are created and written to the hot and cold areas according to the hot and cold policies.
  • While the build task is scheduled, historical partitions are automatically migrated from the hot area to the cold area according to the sliding window of the hot and cold policies. In the following figure, there are three defined hot partitions. On November 4, hot partitions were 11–04, 11–03, and 11–02. On November 5, the new 11–05 data was written, and the latest hot partitions were 11–05, 11–04, and 11–03 according to the sliding window. Therefore, the build task triggered the migration from hot partitions to cold partitions. As shown, partition 11–02 was automatically migrated to the cold area.

Query Acceleration of Cold Data

  • SSD Cache Entry with Different Granularities: This ensures that the random search and throughput-based data scanning of indexes can be done at the same time.
  • Metadata Pre-Heating: After the build task ends, the metadata of the cold partition is automatically generated to accelerate access.
  • Unlocked Access Queues of Hot and Cold Data: This prevents frequently accessed data from being swapped in and out frequently.

Hot and Cold Storage Usage

create table t1(
id int,
dt datetime
) distribute by hash(id)
partition by value(date_format('%Y%m',dt)
lifecycle 12
storage_policy = 'HOT';
create table t2(
id int,
dt datetime
) distribute by hash(id)
partition by value(date_format('%Y%m',dt)
lifecycle 12
storage_policy = 'COLD';
create table t3(
id int,
dt datetime
) distribute by hash(id)
partition by value(date_format('%Y%m',dt)
lifecycle 12
storage_policy = 'MIXED' hot_partition_count=1;

Online-Offline Integration

Hybrid Computing Load

Online Analysis (Online/Interactive)

Batch

Complex Computing (such as Iterative/ML)

Multi-Tenancy in Resource Group (Pool)

-- Create a resource group.
CREATE RESOURCE GROUP group_name
[QUERY_TYPE = {interactive, batch}] -- Specify the execution mode of the resource group query.
[NODE_NUM = N] -- Number of resource group nodes.

-- Bind a resource group.
ALTER RESOURCE GROUP BATCH_RG ADD_USER= batch_user
-- Resize a resource group.
ALTER RESOURCE GROUP BATCH_RG NODE_NUM= 10
-- Delete a resource group.
DROP RESOURCE GROUP BATCH_RG
  1. Interactive Mode: The all-in-memory and pipelined methods are used for online analysis that requires low latency.
  2. Batch Mode: The stage by stage model is used. Intermediate results and operator states can be persisted. This mode is suitable for queries that require high throughput and low latency, featuring lower computing costs.

Scheduled Elasticity

Intelligent Optimization

Smart Adjustment

Optimizations for Dynamic Execution Plans

Dynamic Management of Materialized Views

Summary

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com