Integrating Serverless with SaaS-based Cloud Data Warehouses

1. Introduction to Serverless

The following figure shows the Serverless architecture of MaxCompute, which mainly includes modules of the ingest service, multi-computing environment, storage service and management.

  • Provide Load and Unload commands to import and export OSS data free of charge.
  • Exclusive computing resources support subscription billing mode and workload management, including load isolation, priority, and time-sharing scaling;
  • ETL, OLAP, ML, and other big data analysis scenarios are supported during runtime.
  • Charge the storage space in pay-as-you-go mode to reduce costs;
  • Perform analysis-oriented optimizations by default, such as column pressure and compression;
  • Support partitions, buckets, Zorder, and other optimization methods.
  • Require no user O&M on the platform, which reduces costs.
  • Scenarios of fast-growing business and fast-changing demands;
  • Scenarios of routine demands with emergencies;
  • Scenarios of test or development demands.
  • Computing resources are required for business in the initial stage or the innovative business of the innovation department.
  • Conventional computing power and elastic computing power with job resource policies are scheduled or manually specified.
  • Non-critical jobs are processed at a cost as low as possible, without affecting critical jobs.
  • Refined workload management of fixed resources.

2. Serverless Benefits for Business Agility

Then, how does MaxCompute Serverless meet the demands in the above scenarios? For an enterprise with rapid business development and business change, on-demand computing resources of MaxCompute Serverless are recommended. From a management point of view, different projects can be created for isolation, such as development test environment and production environment.

  • Job priority: Procedure detection of critical dynamic jobs;
  • Time-sharing scaling: Daytime and nighttime resource allocation policies to maximize utilization.
MaxCompute Management
  • Personal setting: Computing resource and job-level resource routing can be set according to the actual needs.
  • Hourly computing power demand (unit: cu•h) in the day with the highest consumption by recent project jobs, based on information schema.

3. Summary

This article describes how to use Serverless to better manage resources and meet the resource demands of different businesses at low costs. It can be summarized as follows:

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store