The Journey of an SQL Query in the MaxCompute Distributed System

MaxCompute: Mega-scale Computing for Enterprises

  • Fully managed, multi-tenant, and mega-scale platform
  • Enterprise-grade high-performance computing engine

Mega-scale Enterprise-grade SQL Engine: MaxCompute UniSQL

The Journey of an SQL Query in a Distributed System

SQL Features

  • Not Only SQL — Script Mode
  • Not Only SQL — Parameterized View
  • Not Only SQL — IF/ELSE
  • Not Only SQL — UDT and Storm

SQL Performance

  • SQL Engine for Huge Data — Adaptive Join
  • SQL Engine for Huge Data — Advanced Shuffle
  1. Use Greysort mode, in which the reducer sorts and the mapper does not. It increases downstream pipelining opportunities and eliminate sorting when downstream is transferred to Hash Join.
  2. Reduce I/O and cache miss by encoding and adaptive column compression.
  3. Optimize memory structure, reduce working set size, and eliminate pointer chasing.

Enterprise-grade Distributed Intelligent Scheduling and Execution Framework

An Enterprise-grade Distributed Scheduling Execution System

Enterprise-grade Distributed Computation Scheduling Framework

  • Dynamic and intelligent execution
  1. Adjust data statistics based on partitions to avoid increasing data skew due to concurrency adjustment. Both upward and downward adjustments are supported.
  2. Automatically split large partitions based on partition statistics. Dual adjustment eliminates data skew in partitions and supports data processing and merging to retain partition attributes.
  • Efficient Job Management
  • Multiple Computational Models

AliOrc: A Next-generation of Columnar Storage Engine

Deep Optimization Based on Apache Orc

Next-generation Columnar Storage Engine

  • Parallel Encoding
  • Asynchronous Parallel I/O
  • Lazy read and lazy decoding

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com