Self-built Kubernetes on Alibaba Cloud

By Anish Nath, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Alibaba Cloud Container Service for Kubernetes is a fully-managed service compatible with Kubernetes to help users focus on their applications rather than managing container infrastructure. There are two ways to deploy Kubernetes on Alibaba Cloud, one through Container Service (built-in) and the other through an Elastic Compute Service (ECS) instance (self-built). If you are not sure which installation method suits your needs better, then refer to the documentation Alibaba Cloud Kubernetes vs. self-built Kubernetes.

For the most part, choosing Alibaba Cloud Container Service is the preferred choice as it saves time and reduces the complexity to maintain Kubernetes clusters. However, there may be cases where a manual installation is better suited to your needs. In this article, we will be setting up a self-built kubernetes on Alibaba Cloud Elastic Compute Service instances using Linux flavors (Centos7 and Ubuntu 16.04).

Prerequisites

  1. A valid Alibaba Cloud account. If you don’t have one already, sign up to the Free Trial to enjoy $300 worth in Alibaba Cloud products.
  2. An ECS instance running Ubuntu 16.04 or RHEL7 or Centos7. You can select your preferred region and configurations; this will not affect the outcome of the server setup.
  3. A sudo password for your server.

Kubernetes Security Group Setup

These required ports needs to be opened in the pod security group, to do this:

  1. Log on to the ECS console.
  2. Select a region.
  3. In the left-side navigation pane, select Networks & Security and select Security Group.
  4. Click Create Security Group.
  5. Set the Inbound rules for your pod network as shown below. For my example, 192.168.0.0/16 is the pod network IP address; you’ll need to replace it with your own.

Authorization ObjectProtocol TypePort RangeAuthorization Policy192.168.0.0/16TCP6443/6443Allow192.168.0.0/16TCP2379/2379Allow192.168.0.0/16TCP2380/2380Allow192.168.0.0/16TCP10250/10250Allow192.168.0.0/16TCP10251/10251Allow192.168.0.0/16TCP10252/10252Allow0.0.0.0/0TCP22/22Allow

Note:

  1. Alibaba Cloud Container Service and the pod CIDR block cannot overlap with the VPC CIDR block.
  2. The service CIDR block cannot overlap with the VPC CIDR block or pod CIDR block.
  3. The security group rules is enable communication between kubernetes master and cluster.

Connect to Your Alibaba Cloud Server

Locate the Internet IP address (Public IP address) associated with your Alibaba Cloud ECS Instance. There are other ways to connect to your ECS instance as well. Visit the official ECS documentation to learn more.

Before proceeding, stop and check whether:

  1. ECS public IP is enabled.
  2. ECS is up and running.
  3. ECS security group is pointed to pods-security group.
  4. ECS security group Authorization Object matches with ECS instance.

Setup Environment

This example utilizes two ECS instances running in the Alibaba Could environment and with these hostnames

Master Server nameMinion Clusterkube-masterkube-minion-1

Install Kubernetes-master and Kubernetes-minion

These are the minimum requirements to setup kubernetes ECS instance master and minion.

RequirementKubernetes-masterKubernetes-minionDisable system swap and SELinuxYYremove any swap entry from /etc/fstabYYnet.bridge.bridge-nf-call-iptables is set to 1YYInstall Docker & enable on restartYYInstall kubeadmYYInstall kubeletYNInstall kubectlYNConfigure docker cgroupsfsYNCreating Network Addons (flannel/Calico)YN

Disable system swap with swapoff -a. This will immediately disable swap and remove any swap entry from /etc/fstab

Disable SE Linux by setenforce 0

You should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctlconfig

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

If you are running CentOS or RHEL7, install Docker and enable on restart with these commands.

yum -y update
yum install -y docker

If you are running Ubuntu, install Docker and enable on restart with these commands.

apt-get update 
apt-get install -y docker.io

Start and enable Docker and check if docker service is running.

systemctl start docker
systemctl enable docker
systemctl status docker

Install Kubernetes

You will need to install these packages on all of your machines:

  1. kubeadm: the command to bootstrap the cluster.
  2. kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
  3. kubectl: the command line util to talk to your cluster.

Kubernetes Installation on CentOS7/RHEL7

Setting up Kubernetes yum repository

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Install Kubernetes in CentOS7/RHEL7

yum install -y kubelet kubeadm kubectl

Kubernetes Installation on Ubuntu

Setting up Kubernetes apt repository for Ubuntu

apt-get install -y apt-transport-https curl
apt-get install -y docker.io
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update

Install Kubernetes in Ubuntu

apt-get install -y kubelet kubeadm kubectl

Running Kubernetes

Enable & start kublet

systemctl enable kubelet  
systemctl start kubelet

Verify that your Docker cgroup driver matches the kubelet config: (kube-master node)

docker info | grep -i cgroup
Cgroup Driver: cgroupfs

Configuration for cgroup drive is right in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[Service]
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

Reload daemon and restart kubelet: (kube-master node)

systemctl daemon-reload
systemctl restart kubelet

Flush Reset Kubernetes component (This will wipeout all the cluster config if exist any do not run this after the config create)

kubeadm reset -f

Set up Kubernetes Network add on

  1. For flannel to work correctly, — pod-network-cidr=10.244.0.0/16 has to be passed to kubeadm init.
  2. For Calico to work correctly, — pod-network-cidr=192.168.0.0/16 has to be passed to kubeadm init.

Creating flannel Networks

Note apiserver-advertise-address is the IP of the kube-master

kubeadm init --service-cidr 10.96.0.0/12 --kubernetes-version v1.11.0 --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address 192.168.1.130

You should get information back on initiating commands as a normal user, as well as the network that you need to deploy as well as how to join worker nodes to the cluster.

To start using your cluster, you need to run the following as a regular user:  mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You can now join any number of machines by running the following on each node
as root:

Join the Cluster (kube-minnion)

kubeadm join 192.168.1.130:6443 --token 5m8qxr.46rpadiwt8fcka0v --discovery-token-ca-cert-hash sha256:b05a0b8849a57432247c06200864f5ce99d40ffdcae965293c0026204ef33da4

Run kubectl get nodes on the master to see this node join the cluster. It will few seconds to get your cluster in ready state

root@kube-master:kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master Ready master 2m v1.11.0
kube-minion Ready <none> 47s v1.11.0

Apply flannel Addons

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

You will get output like this

clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds created

Enable and restart kubelet engine

systemctl restart kubelet
systemctl status kubelet

Verifying the Installation

Verify the Cluster Information

root@kube-master: kubectl cluster-info 
Kubernetes master is running at https://192.168.1.132:6443
KubeDNS is running at https://192.168.1.132:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Verify the Services

root@kube-master:kubectl get services -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 2m

Create Tokens

[root@kube-master ~]kubeadm token create 
I0710 04:08:37.149017 8685 feature_gate.go:230] feature gates: &{map[]}
d49l0d.mheeem1dkrw3n43

That’s it! You have successfully configured a self-built Kubernetes cluster on Alibaba Cloud. To learn more about Kubernetes on Alibaba Cloud, visit Alibaba Cloud Container Service for Kubernetes.Reference: https://www.alibabacloud.com/blog/self-built-kubernetes-on-alibaba-cloud_594019?spm=a2c41.12067603.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store