Building Docker Enterprise 2.1 Cluster Using Terraform
By Sajith Venkit, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.
In this article, we will show how you can use Terraform to fully automate the build of a 3-node Docker Enterprise 2.1 cluster on Alibaba Cloud. If you are in the process of planning or setting up an Enterprise Docker Cluster for your containers, in any of the public cloud platforms, this article and the templates will be useful.
Even if you are not building a Docker Enterprise cluster, you may still find this article useful if you’d like to automate your infrastructure build in Alibaba Cloud.
As my objective was to show the art of possible using IaC (Infrastructure as Code) to automate build in Alibaba Cloud, I thought why not do that using the most popular platforms, hence decided to:
- Build a Containers-as-a-Service platform using Docker Enterprise Edition 2.1 and,
- Build that platform using Terraform.
Docker Enterprise Edition
Docker Enterprise 2.1 is a Containers-as-a-Service (CaaS) platform that enables a secure software supply chain and deploys diverse applications for high availability across disparate infrastructure, both on-premises and in the cloud. It is a secure, scalable, and supported container platform for building and orchestrating applications across multi-tenant Linux, Windows Server 2016, and IBM Z environments.
One thing that I always loved about Docker is their Simplicity and Customer Centricity. That’s exactly what they have done with the release of Enterprise 2.1 too. With Docker EE 2.1, you now have a freedom of choice as it can:
- Reliably support both Windows and Linux containers.
- Be hosted in any cloud platform or on-premise Data Center.
- Use both Docker Swarm and Kubernetes orchestration interchangeably.
So, if you are an Enterprise customer who is looking to embark on a project to either migrate your legacy applications to Containers or keen to embrace DevOps for development of new applications, I’d strongly recommend Docker EE 2.1 as your CaaS platform.
You can start small and scale your cluster as you grow your container base. You can start utilizing the much simpler Docker Swarm for initial orchestration and switch to Kubernetes later if you really need it.
Docker EE 2.1 cluster also comes with components such as:
- Docker UCP — which gives a single-pane-of-glass across your cluster.
- Docker Trusted Registry — to securely host your container images.
It also has enterprise security features like encrypted communication, application isolation, vulnerability scanning for images and much more.
Terraform is one of my favorite Orchestration/IaC tools out there. I just love the power and flexibility that Terraform offers for deploying new services to any public cloud platforms. You just define what you need and ask Terraform to Go and Build. It is that simple.
I chose Terraform for this automation as it is pretty much platform agnostic. Though you can’t use the same templates for any cloud service provider it is quite easy to customize to a different provider once it is developed for a specific Cloud platform.
To learn more about Terraform, visit the HashiCorp website or read this good summary by MVP Alberto Roura’s Tech Share article about Terraform.
Building the Docker Enterprise Cluster
For this demo, I chose to build a small 3-node Docker Enterprise 2.1 cluster.
1. One Alibaba Cloud ECS Linux server that hosts both Docker UCP and Docker Trusted Registry (DTR). Same node will also be configured as the Docker Swarm Manager and Kubernetes Master.
2. One Linux host which will be automatically joined as a worker node, in the Docker Swarm created by the UCP host.
3. One Windows host which will be automatically joined as a worker node, in the Docker Swarm created by the UCP host.
If you would like to get on with the cluster build right away, go to my GitHub repository and follow the instructions there.
Once you get the pre-requisites ready, you could get the cluster built in less than 30 minutes.
Terraform Files Explained
If you’d rather understand the templates first before jumping into action, the following sections will take you through the details of the Terraform files.
terraform configuration files in my GitHub repository, and the
powershellscripts, have adequate comments to help you understand them better. In this article, I would like to focus on explaining the purpose of each of those template or script.
Terraform, as you may be aware, helps to deploy the Infrastructure by letting users define what they need, using the High-level Configuration Language (HCL). Terraform will build a detailed execution plan based on this definition and deploy the infrastructure on the platform specified.
Terraform supports a wide range of Cloud service providers and certainly the major Cloud providers and VMWare vSphere for on-premise Data Center.
The complete definition of our Docker Enterprise Cluster could be specified in a single
.tfdefinition file but it is recommended to split into multiple
.tf files for ease of maintaining and readability. All
.tf files in the main folder form part of the
root (main) module.
Terraform will load all configuration files (
.tf files) in the folder, in alphabetical order. The order of variables, resources, etc. defined within the configuration doesn't matter. Terraform configurations are declarative, so references to other resources and variables do not depend on the order they're defined.
Input variables serve as parameters for the Terraform module. All variables must be declared in a
.tf file (e.g., variables.tf) and their values could be passed in command line during execution OR in a separate
.tfvars file (e.g., terraform.tfvars)
I recommend using
UPPER CASE for the input variable names so they stand out from the terraform local variables, defined using
locals and terraform
resource names. I would also suggest to give the values for variables in a separate
terraform.tfvars file, so you can choose to not send this file when sharing with others as it may contain your account secret keys, passwords etc. During execution, if terraform cannot find a
.tfvars file and the default values for variables are not defined, it prompts the user.
This file defines the keys to connect to Alicloud and the region where you want the resources created
Network and Security
This file defines the VPC, vSwitch, Security Group and the security/firewall rules to restrict access to the docker hosts.
The vSwitch and Security groups defined in this file will be mapped to our hosts, in the ECS (Elastic Compute Service) definition.
Few points to note:
- Priority value for the security roles range from 1–100. Smaller the value, higher the priority
- RDP, WinRM and SSH access are allowed only from a specific IP — your Public IP.
- Kubernetes, Docker and application access are allowed from anywhere
Compute for Docker UCP Manager
This file defines the main host where Docker UCP and DTR will be installed. As part of Docker UCP installation, it will also initiate creation of Docker Swarm to which the other hosts will join as worker nodes.
Few points to note:
internet_max_bandwidth_outattribute ensures a Public IP is assigned as part of VM creation.
- This Public IP is used in the
fileprovisioner to copy the installation scripts and for
remote-execprovisioner to run the setup automatically for Docker EE, UCP and DTR
ssh key pairfor the UCP host and the linux worker node is defined in
Compute for Docker Worker Nodes
This file defines the linux worker node. Quite similar to the
docker-host.tf file, except that this only has scripts to install Docker EE and join as worker node.
This file defines the windows worker node. Key differences from the Linux node definition are:
- Windows image instead of the Linux image.
passwordattribute instead of the
key_nameattribute used in Linux for ssh-key pair.
user-dataattribute which is used for bootstrapping the VM, including configuration of
WinRM, so further scripts can be executed using
- Connection type is
sshused in Linux host.
This file defines the values returned by the module. These values will be printed once Terraform successfully completes the execution.
Values defined in
output.tf can be printed anytime using the command
terraform output, after a successful execution.
To retrieve further details of the Terraform execution, use
Scripts to Automate Build of Docker Components
Scripts used to bootstrap Linux hosts are in the folder
lin-files. They are called using the
remote-exec provisioner in the ECS VM definition.
These scripts are used to setup Docker Enterprise, Docker UCP and Docker Trusted Registry as per the documentation in Docker website.
Scripts used to bootstrap Windows host are in the folder
win-files. They are called using the
remote-exec provisioner in the ECS VM definition.
These scripts are used to setup Docker Enterprise and also to join as worker node. Docker UCP and Docker Trusted Registry cannot be installed on Windows hosts. They can only be worker nodes.
Key point to note in the Windows scripts,
Invoke-RestMethod works fine when accessing the Docker API, following changes had to be done in the script prior to calling the Docker API.
- Change the default TLS version in powershell from 1.0. to 1.2
- Ignore the certificate errors when connecting to Docker API
This article was intended only to introduce you to the basic concepts of Docker Enterprise and how you can use Terraform to automate deployments in Alibaba Cloud. The definition files and scripts in the GitHub repository will help you setup a basic Docker Enterprise cluster and also provide you tips which you can use for other installations.
In subsequent articles, we will look at how to scale your cluster with a load balancer and additional nodes and also details on deploying containers using Docker Swarm or Kubernetes orchestration.
This section lists several useful references if you would like to learn more about Docker, Terraform, and Automation
1. Docker Enterprise
3. My GitHub repository