Creating Custom Environments for Batch Services

Image for post
Image for post

By Afzaal Ahmad Zeeshan, Alibaba Cloud Community Blog author and Alibaba Cloud MVP.

You must have realized the importance of concurrent programming and processing when working with microprocessors and their applications. Nonetheless, the concept of batch computing is an advanced topic, but the basis of the idea relates strongly to the above-mentioned example. In batch computing, you utilize several CPU cores and then have it perform a task that you require it to perform.

That being said, Alibaba Cloud’s Batch Compute is one of the most powerful products and toolset which one should explore for distributed processing of a massive number of datasets. On Alibaba Cloud, Batch Compute is used extensively for several jobs; running a background job every hour, processing the incoming data as the bucket becomes full, or even high-end computing tasks such as rendering computer graphics. All these tasks can be done with Alibaba Cloud Batch Compute, and since the underlying infrastructure is strong enough to support thousands of CPU cores, you are in safe hands.

Alibaba Cloud Batch Compute consists of the toolset that offers resource management, service to automate job scheduling and post-build actions, such as, resource releasing and cleaning. Moreover, you can get all these services in a pay-as-you-use model, which is undoubtedly a cost-cutting feature. Customers do not have to manage or customize the settings at all, they just need to select the kind of worker nodes they are looking for and their plan is ready to be executed.

Fundamentally, the program or task description which you want to compute is termed as your job. In Alibaba Cloud Batch Compute, you can submit your job either by CMD or console. Also, there is a way to submit and manage your query using Alibaba Cloud’s official SDK. Once you submit your job for batch computing, your program starts running on multiple VM instances which are launched by a specified image such as Windows Server machine or Ubuntu.

So far, we have explored the basic concepts of the batch computing and how Alibaba Cloud provides these services. But in this article, we will not explore how to create jobs rather we will take a look at how we can customize the underlying infrastructure as needed or required by our software packages.

Extending Features of Public Images

Regarding customization, we can modify VM specification and settings (I.e., network and sharing) as per our requirements, for this very fact, Alibaba Cloud provides us the facility to also import local images to launch VM instances. This can mean that we can have our own customized and proprietary image running on the cloud, with all the hardware and compute power rented from the cloud platform.

Official vs. Custom Batch Computing Image

Image for post
Image for post

If the above-listed default image options are not meeting your requirements then, then you should create your custom image on which you will be installing the software which you require as per your need. You can always find a bunch of officially supported and uploaded images on the portal,

Image for post
Image for post

In case you decide to go with the custom image creation then, firstly, you need to purchase a basic batch compute image and launch ECS instance on it. The values from that ECS instance would be used to create the custom image of yours, as you can see in the following image,

Image for post
Image for post

While creating the ECS instance, you can also select the size of the virtual machine resources that you might need. They must be officially supported, and currently you cannot modify the size of the virtual machines.

Image for post
Image for post

Secondly, you would be allowed to install whatever the software you need to use on your VM, then you after the creation of ECS image, you will be able to register your Batch Compute image to further go along with it, and at the end, you will be supposed to release the ECS instance.

Fundamental Features

Achieving the Optimal Performance

Easy Job Description and Submission

Image for post
Image for post

And here is the JSON editor, that you can use, yourself to write the job script or you can download the script written for the job that you created in the graphical window.

Image for post
Image for post

Moreover, you can also configure some other settings such as where log files should be created and placed, etc. Once you have a job description JSON file, you can submit your job to Batch Compute to run on the pre-configured VM with the pre-specified image instance — these settings can be controlled by double-clicking the task icon in the graphical window as shown in the image above.

Job Management

Benefits and Use cases

High-Resolution Audio/Video Rendering

Most of the times the rendering of the images is done on several hundreds (if not thousands) of CPU cores that can finish the overall job in a matter of hours with a couple hundred or thousand dollars, if it would take weeks on an on-premises setup with millions of dollars of investment.

Big Data Analytics

Concurrency and Scaling

Optimization for IO operations and Cost Management

At last, you do not need to much concern about cost and resource consumption, as this is a pay-as-you-use model, which means you will be charged only as per your business usage and requirements.

Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store