Deploying Kubeflow Pipelines on Alibaba Cloud

Overview

What Is Kubeflow Pipelines

  • A console for running and tracing experiments.
  • The workflow engine Argo for scheduling multi-step machine learning workflows.
  • An SDK for defining workflows. Currently, the SDK only supports Python.
  • End-to-end orchestration: enables and simplifies the orchestration of machine learning pipelines. Pipelines can be triggered directly, at a scheduled time, by event, or even by data changes.
  • Easy experiment management: makes it easy for you to try numerous ideas and techniques and manage your experiments. Kubeflow Pipelines also makes the transition from experiments to production much easier.
  • Easy re-use: enables you to re-use components and pipelines to quickly create end-to-end solutions without the need to rebuild experiments each time.

Deploy Kubeflow Pipelines on Alibaba Cloud

  1. Pipelines are deployed by using Kubeflow. However, Kubeflow has many built-in components and it is complex to use Ksonnet to deploy Kubeflow. 2. Pipelines depend on the Google cloud platform. They cannot run on other cloud platforms or bare metal instances.

Prerequisites

opsys=linux  # or darwin, or windows
curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest |\
grep browser_download |\
grep $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L
mv kustomize_*_${opsys}_amd64 /usr/bin/kustomize
chmod u+x /usr/bin/kustomize
  • For more information about creating a Kubernetes cluster in Alibaba Cloud Container Service, click here.

Procedure

yum install -y git
git clone --recursive https://github.com/aliyunContainerService/kubeflow-aliyun
yum install -y openssl
domain="pipelines.kubeflow.org"
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout kubeflow-aliyun/overlays/ack-auto-clouddisk/tls.key -out kubeflow-aliyun/overlays/ack-auto-clouddisk/tls.crt -subj "/CN=$domain/O=$domain"
yum install -y httpd-tools
htpasswd -c kubeflow-aliyun/overlays/ack-auto-clouddisk/auth admin
New password:
Re-type new password:
Adding password for user admin
cd kubeflow-aliyun/
kustomize build overlays/ack-auto-clouddisk > /tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/regionid: cn-beijing/regionid: cn-hangzhou/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/zoneid: cn-beijing-e/zoneid: cn-hangzhou-g/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/gcr.io/registry.aliyuncs.com/g' \
/tmp/ack-auto-clouddisk.yaml
sed -i.bak 's/storage: 100Gi/storage: 200Gi/g' \
/tmp/ack-auto-clouddisk.yaml
kubectl create --validate=true --dry-run=true -f /tmp/ack-auto-clouddisk.yaml
kubectl create -f /tmp/ack-auto-clouddisk.yaml
kubectl get ing -n kubeflow
NAME HOSTS ADDRESS PORTS AGE
ml-pipeline-ui * 112.124.193.271 80, 443 11m

FAQ

  • Delete the Kubeflow Pipelines components.
kubectl delete -f /tmp/ack-auto-clouddisk.yaml

Summary

Original Source

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Types-First the only supported mode in Flow (Jan 2021)

Web Video Cast | Browser to TV/Chromecast/Roku/+ Mod Apk 5.5.0 (Unlocked)(Premium)

Web Video Cast | Browser to TV/Chromecast/Roku/+ Mod Apk 5.5.0 (Unlocked)(Premium)

Website Vs Web Application

Abusing Docker Registry to gain access to a mounted Volume

Automated Observability in Pipedrive — the What, the Why, the How?

Cleaner Code = Better Quality

[PMPL — Milestone 3] Upgrade Python 2 to 3 and Django 1.11 to 2.2

Impact Lab — Smart Spaces: Making sounds visible

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Scaling Airflow Workers in EKS

Efficient fetching of data from Oracle database in Golang

Deploying Airflow in Local Kubernetes Cluster: Part II

Building Docker images that require NVIDIA runtime environment