Data Warehouse: In-depth Interpretation of Flink Resource Management Mechanism

1. Basic Concepts

1.1 Related Components

Figure 1 Resource management components of Flink

1.2 Logical Hierarchy

Figure 2 Logical hierarchy of components

1.3 Two-layer Resource Scheduling Model

Figure 3 Two-layer resource scheduling model

2. Current Mechanisms and Policies

2.1 Resources of a Task Manager

Figure 4 Resources of a task manager

2.2 Resources of a Slot

Figure 5. Slot resources

2.3 How Many Task Managers Does a Flink Cluster Have?

<FLINK_DIR>/conf/slaves
yarn-session.sh -n <num>
flink run -yn <num>

2.4 Process of Scheduling Resources from a Cluster to a Job

Figure 6. Process of scheduling resources from a cluster to a job

2.5 Process of Scheduling Resources from a Job to a Task

Figure 7. Process of scheduling resources from a job to a task

2.6 Resource Optimization

3. Future Development Directions

3.1 Fine-grained Resource Management

Figure 8. Limitations of slot sharing
Figure 9

3.2 Dynamic Slot Segmentation

Figure 10. Static slot allocation
Figure 11. Dynamic slot segmentation
Figure 12. Static slot segmentation
Figure 13. Dynamic slot segmentation

3.3 Resource Fragmentation Problem

3.4 Ease-of-Use Issues

3.5 Making Resource Scheduling Policies Plug-ins (FLINK-14106)

About the Author

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Building A Shazam Clone Part 5 (Discover Song Feature) — Building Android Apps Series

The Journey to Our New Authentication System

What is New in RDFox Version 5?

Early Experiences with Kubernetes: Debugging Unresponsive Nodes

Configuring AWS ASG and testing the policy

Splitting Git Repos

Auto-book COVID test appointment

Top 10 Java Coding Best Practices for Beginners

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

AWS re:invent 2021 Experience — A Data Engineer’s perspective

Advancing Apache Airflow Workflow Schedules

Neediness of MLOps for large-scale AI solutions using Azure

FluentD webhook with TLS Mutual Authentication