Using Alibaba Cloud Storage Gateway for Hybrid Solutions
Alibaba Cloud Storage Gateway (CSG) is a hybrid cloud solution from Alibaba Cloud that can help customers resolve multiple use cases pertaining to cheaper cloud storage solutions, data backups and data archiving. CSG comes in two flavors — file and block — and two deployment models — online and offline (i.e in Alibaba Cloud and customer premises).
You can find more information and details about Alibaba Cloud CSG in the official documentation.
Use Case of Cloud Storage Gateway
In this blog post, I’ll show you step-by-step how to create a file gateway and use Alibaba Cloud Object Storage Service (OSS) as the storage backend. This can be useful in scenarios where the customers don’t want to commit to a large onsite storage and instead can use Alibaba Cloud OSS as the storage solution for TBs or PBs worth of data. Though the same outcome can be achieved by an offline deployment as well as online deployment (barring certain limitations highlighted at the end of this post), I’ll use an example of online deployment for this post.
In this solution, Cloud Storage Gateway will be used to expose a folder created in the OSS bucket to the client server as a NFS mount. This mount can be then used to store the data and only the data equivalent to the gateway cache size will be stored locally whereas all the excess data will be available in the OSS. I’ll use an Elastic Compute Service (ECS) instance to mount the NFS share, however the same can be done on an on-premises VM.
Alibaba Cloud Products Used in the Solution
In this demo, I’ll use following Alibaba Cloud products:
- Elastic Compute Service (ECS) — This is one of the fundamental services that offers virtual computing instances on cloud to customers. We’ll use a basic Ubuntu server for this demo. More information can be found at https://www.alibabacloud.com/help/doc-detail/25367.htm
- Object Storage Service (OSS) — This is Alibaba Cloud’s cheap and durable storage service that can be used for various backup and archival scenarios or simply storing any amount of data at low cost. More information can be found at https://www.alibabacloud.com/help/doc-detail/31817.htm
- Cloud Storage Gateway (CSG) — A file gateway with NFS protocol enabled (required for an Ubuntu system).
1. On the Alibaba Cloud Storage Gateway console, create a gateway cluster.
2. Once the cluster is created, create a new gateway in the console.
Click on “Create” button in the right corner. You will see the following screen.
Location — Alibaba Cloud means the online deployment (this is what we are going to see in this example) and On-premise is the offline deployment (i.e in customer’s environment on a VM)
Type — File Gateway (NFS or SMB protocols that we are going to use in this example) and iSCSI Gateway is the block gateway
In the next screen, choose the gateway model. The gateway models vary in terms of the gateway bandwidth. Here you also attach the gateway to a VPC (Virtual Private Cloud) in your environment.
3. Once created you should be able to see the newly created gateway in the list on the gateway console
4. Next step is to create the gateway cache that will be used as the local storage on the client server.
For this, click on the gateway link on the console > click on the cache link in the left navigation menu > click on “create cache” button on the left right corner of the screen.
Chose the size of cache and disk type
5. Next step is to create the NFS share. For this, go the share link in the console and click “create” button
6. On the first screen, select the OSS bucket that you want the NFS share to be mapped to
To be able to map the NFS share to a specific folder in your OSS bucket — After selecting the bucket, make sure you check “Path Prefix” and then type in the exact folder name that you want to map to the bucket. You can also map sub-folders like Test1/test2 in the Path prefix
7. Next, provide the details like share protocol (NFS or SMB) and available cache (that we created in the step#4)
Note that an already mapped cache disk can’t be mapped to another share (they won’t even appear in the drop down). So for every share you create, there must be dedicated cache disk
8. Next is the advanced configurations.
You can leave the advanced settings as default. Check the explanation for each of them before changing the configuration.
9. On the last screen, you will be shown the summary of configuration just made
The new share will appear in the list of shares on the console. You can now manage the share in the console
10. Now perform the following steps on the client Ubuntu ECS server (assuming you already know how to create one). This is the server instance where you want to mount the NFS share
root@test-NFS-share:~# mkdir /var/nfs/new -p
root@test-NFS-share:~# chown nobody:nogroup /var/nfs/new
root@test-NFS-share:~# mount.nfs 172.27.48.156:/test-NFS-share-1 /var/nfs/new
You can see the newly mounted NFS share in the screenshot
To test, create a file on the server inside the NFS share that you just mounted
The same file should appear in the OSS bucket inside the folder that you mapped while creating the share
Please keep in mind the following consideration when adopting a Cloud Storage Gateway based solution:
- For an online deployment, if you are trying to mount the share on an ECS instance, it needs to be in the same VPC as the gateway, or if in different VPCs, then both VPCs should be interconnected/peered.
- If you are trying to mount the share on a local on-premises server, there needs to be a direct connectivity like an Express Connect or Cloud Enterprise Network (CEN) link between on-premises and Alibaba Cloud VPC.
- Currently there is no way to connect to the gateway over the internet because this will open potential security holes (although things may change in the future). If a customer wants public internet access, on-premises gateway (offline deployment) might be the preferred choice because online gateway only works using Private IP.
Author: Rohit Gupta, Solutions Architect