Build a Machine Learning System Using Kubernetes

What is Kubeflow Pipelines?

  • A console for running and tracing experiments.
  • The Argo workflow engine for performing multiple machine learning steps.
  • A software development kit (SDK) for defining workflows. Currently, only the Python SDK is supported.
  • End-to-end task orchestration: You can orchestrate and organize a complex machine learning workflow. This workflow can be triggered directly at a scheduled time, or be triggered by events or even by data changes.
  • Easy experiment management: Scientists can try numerous ideas and frameworks and manage various experiments. Kubeflow Pipelines also facilitates the transition from experiments to production.
  • Easy reuse: You can quickly create end-to-end solutions by reusing pipelines and components, without the need to rebuild experiments from scratch each time.

Run Kubeflow Pipelines on Alibaba Cloud

  1. Pipelines must be deployed by using Kubeflow. However, Kubeflow includes many built-in components and it is complicated to use Ksonnet to deploy Kubeflow.
  2. In addition, with its heavy dependence on the Google cloud platform, Pipelines cannot run on other cloud platforms or bare metal servers.

Prerequisites

opsys=linux  # or darwin, or windows
curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest |\
grep browser_download |\
grep $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L
mv kustomize_*_${opsys}_amd64 /usr/bin/kustomize
chmod u+x /usr/bin/kustomize
  • For more information about creating a Kubernetes cluster in Alibaba Cloud Container Service, click Here.

Deployment Procedure

yum install -y git
git clone --recursive https://github.com/aliyunContainerService/kubeflow-aliyun
yum install -y openssl
domain="pipelines.kubeflow.org"
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout kubeflow-aliyun/overlays/ack-auto-clouddisk/tls.key -out kubeflow-aliyun/overlays/ack-auto-clouddisk/tls.crt -subj "/CN=$domain/O=$domain"
yum install -y httpd-tools
htpasswd -c kubeflow-aliyun/overlays/ack-auto-clouddisk/auth admin
New password:
Re-type new password:
Adding password for user admin
cd kubeflow-aliyun/
kustomize build overlays/ack-auto-clouddisk > /tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/regionid: cn-beijing/regionid: cn-hangzhou/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/zoneid: cn-beijing-e/zoneid: cn-hangzhou-g/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/gcr.io/registry.aliyuncs.com/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/storage: 100Gi/storage: 200Gi/g' \
/tmp/ack-auto-clouddisk.yaml
kubectl create --validate=true --dry-run=true -f /tmp/ack-auto-clouddisk.yaml
kubectl create -f /tmp/ack-auto-clouddisk.yaml
kubectl get ing -n kubeflow
NAME HOSTS ADDRESS PORTS AGE
ml-pipeline-ui * 112.124.193.271 80, 443 11m

Some Considerations.

  • Delete the components of the Kubeflow Pipelines.
kubectl delete -f /tmp/ack-auto-clouddisk.yaml
  • Click Release disks to release the two disks that store MySQL and Minio data.

Summary

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com