Kubernetes Cluster Monitoring Using Prometheusv

By Anish Nath, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Alibaba Cloud Container Service for Kubernetes is a fully-managed service compatible with Kubernetes to help users focus on their applications rather than managing container infrastructure. There are two ways to deploy Kubernetes on Alibaba Cloud, one through Container Service (built-in) and another through an Elastic Compute Service (ECS) instance (self-built). If you are not sure which installation method suits your needs better, then refer to the documentation Alibaba Cloud Kubernetes vs. self-built Kubernetes.

In case you are new to Alibaba Cloud, you can get $10 worth in credit through my referral link to get started on an ECS instance.

In this article, we will cover the following topics:

  • Kube-prometheus vs Prometheus Operator
  • How to install Prometheus-operator
  • How to Install Kube-prometheus
  • Accessing Promethus Server UI (Grafana/Alert Manager)
  • How to perform Service Monitor Lookup
  • Conclusion

Before You Begin

You only need to have running Kubernetes cluster with deployed Prometheus. Prometheus will use metrics provided by cAdvisor via kubelet service (runs on each node of Kubernetes cluster by default) and via kube-apiserver service only.

Make sure your k8 cluster is up and running

Make sure your nodes is up and running

Prometheus-Operator vs. Kube-Prometheus

There are two ways to monitor K8s cluster service

  1. Kube-prometheus collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with PrometheusContainer
  2. using the Prometheus Operator.
  3. Prometheus-Operator: The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances

Prometheus Operator

The Prometheus Operator provides monitoring for k8s services and deployments besides managing the below components.

  1. Prometheus
  2. Alertmanager
  3. Grafana configuration

Operators were introduced by CoreOS as a class of software that operates other software, putting operational knowledge collected by humans into software. The below high reference diagram shows, various service monitor can be created to check various K8 service, the operator lookup to these monitor and reports to Prometheus server

Installing Prometheus-Operator

This step uses helm the kubernetes package manager. If you not setup the helm then do the below the configuration, otherwise move to next step.

Once the helm is ready and related titler pods is up and running , use the Prometheus chard from the helm repository

Lookup all the required pods are up and running

Installing Kube-Prometheus

To install kube-prometheus, first clone all the required repositories.

Apply the default Prometheus manifest

The default stack deploys the components like :

We have defined the following configurations:

  • Every things are built under the namespace of monitoring
  • 2 replica of Prometheus server deployment prometheus-k8s
  • 3 replica of Alertmanager deployment alertmanager-main
  • 1 replica of grafana
  • node-exporter pod per physical host
  • kube-state-metrics exporter
  • prometheus-operator pod, the core of the stack,

Note: The default configuration is not production ready, the TLS setting needs to applied and to re-work on which API needs to be exposed.

Accessing Grafana

Grafana dashboard has a datasource ready to query on Prometheus

Open web browser to http://localhost:300 , you will access the Grafana interface, which is already populated with some useful dashboards related to kubernetes (k8)
By default, Grafana will be listening on http://localhost:3000. The default login is "admin" / "admin".

Grafana Login

The Grafana interface already populated with some useful dashboards.

The K8s / Compute Resources / Cluster Dashboard.

The K8s / Compute Resources / Namespaces Dashboard

The K8s / Compute Resources / Pods Dashboard

Data Sources / Prometheus: Service lookup is achieved through kube-dns lookup URL http://prometheus-k8s.monitoring.svc:9090

Prometheus Server UI

Create the service pods which will expose the endpoint address

Wait for the prometheus-service-prometheus-0 to be up and running

Then Enable port forward configuration to access Prometheus server UI

Open web browser to http://localhost:9090, you will access the Prometheus interface.

These are the standard ports which will be helpful while dealing with Prometheus server and its associated service.

  • 9090 — Prometheus server
  • 9091 — Pushgateway
  • 9092 — UNALLOCATED (to avoid collision with Kafka)
  • 9093 — Alertmanager
  • 9094 — Alertmanager clustering

Performing ServiceMonitor Lookups

Until now, we have worked out with the example of monitoring the K8 cluster with prometheus-operator, what about the application which is loaded externally, well for that we have ServiceMonitor Lookups but before jumping to it, let’s revisit

  • CRD (CustomResourceDefinitions), API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify.
  • ServiceMonitor defines monitoring for a set of services.
  • Prometheus-operator uses a Custom Resource Definition (CRD), named ServiceMonitor, to abstract the configuration to target.

For example the below YAML configuration will select and monitor the nginx pod using the matchLabels selector. i.e app=nginx

serviceMonitorSelector which defines a selection of ServiceMonitors to be used.

Here’s another example of using ServiceMonitor. Let’s first deploy three instances of a simple example application, which listens and exposes metrics on port 8080

This serviceobject is discovered and monitor by a ServiceMonitor where app=example-app


That’s it! It is fairly easy to deploy Prometheus Operator and now I hope it’s easy to monitor all your services even if they are exist outside from your Kubernetes cluster. While running up this exercise, you might end-up with troubleshooting, as this is the beauty of kubernetes, In order to debug it correctly, you must have a good understanding of kubernetes deployment and related k8 services then proceed with Prometheus setup. There are many other thing Prometheus offer like third party integration and it’s not an event logging system, and attempting to use it that way will result in low performance and other issues.


Follow me to keep abreast with the latest technology news, industry insights, and developer trends.