Metrics Principles and Practices: Flink Advanced Tutorials

What Are Metrics?

Metric Types

Metric Group

•TaskManagerMetricGroup
•TaskManagerJobMetricGroup
•TaskMetricGroup
•TaskIOMetricGroup
•OperatorMetricGroup
•${User-defined Group} / ${User-defined Metrics}
•OperatorIOMetricGroup
•JobManagerMetricGroup
•JobManagerJobMetricGroup

How to Use Metrics?

System Metrics

•CPU
•Memory
•Threads
•Garbage Collection
•Network
•Classloader
•Cluster
•Availability
•Checkpointing
•StateBackend
•IO
For more information, see: https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#system-metrics

User-defined Metrics

Inherited from RichFunction
•Register user-defined Metric Group: getRuntimeContext().getMetricGroup().addGroup(…)
•Register user-defined Metric: getRuntimeContext().getMetricGroup().counter/gauge/meter/histogram(…)

User-defined Metrics Example

•Counter processedCount = getRuntimeContext().getMetricGroup().counter("processed_count"); processedCount.inc();
•Meter processRate = getRuntimeContext().getMetricGroup().meter("rate", new MeterView(60));
processRate.markEvent();
•getRuntimeContext().getMetricGroup().gauge("current_timestamp", System::currentTimeMillis);
•Histogram histogram = getRuntimeContext().getMetricGroup().histogram("histogram", new DescriptiveStatisticsHistogram(1000)); histogram.update(1024);
•https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#metric-types

Ways to Obtain Metrics

Metric Reporter

•Flink offers many built-in reporters, which can serve as references for the selection of an external system model. For more information, see- https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#reporter. 
•Metric Reporter Configuration Example
metrics.reporters: your_monitor,jmx
metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 1025-10000
metrics.reporter.your_monitor.class: com.your_company.YourMonitorClass
metrics.reporter.your_monitor.interval: 10 SECONDS
metrics.reporter.your_monitor.config.a: your_a_value
metrics.reporter.your_monitor.config.b: your_b_value

Practice: Monitoring with Metrics

Automated O&M

Performance Analysis

Practice: How to Solve a Slow Task Problem

Q&A

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Note Taking in Runtime Configuration

101 on Kafka (beginners)

Word Search in a 2d matrix using DFS & backpropagation

Loading and Moving Data to Cloud Storage Using the Command Line

[Announcement] 2022 Upcoming updates

Backup & Restore Docker swarm

InfinityDB — A NoSQL Multi-Value Database you most know!

SURFING WEB: BEHIND THE SCENCE

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Better Developer Experience: Getting Started with YugabyteDB on Gitpod

Error handling with Apache Beam, Asgarde with Kotlin

An overview of message brokers

Setup HTTPs Authentication for Presto / Trino