11 Security Recommendations for Production Instances on Alibaba Cloud

12 min readMay 7, 2019

By Anish Nath, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Recently, I have successfully hosted my first three-tier web application, which includes two Elastic Compute Service (ECS) instances, one [ApsaraDB for RDS MySQL] database, a Server Load Balancer, and used Elastic IP and security group to secure them. This is the most common scenario for most applications hosted on the web. Although my system is functioning well, this type of deployment is deficient in terms of cyber security. This is especially true for servers used in production scenarios.

As with many popular cloud providers, Alibaba Cloud adopts a shared responsibility model that puts data and application security on the hands of the customer and infrastructure and hardware security on the hands of the provider. Alibaba Cloud offers many solutions and guidelines to protect user account and resources, and this article is all about that. Since there are many resources in Alibaba Cloud, I will be focusing on fundamental resources (SLB, ECS, OSS, RDS) and their security hardening requirements, which are easy to follow.

Prerequisites

Before we begin, you’ll need an Alibaba Cloud account. In case you are new to Alibaba Cloud, you can get $10 worth in credit through my referral link to get started.

You must have a basic familiarity with Alibaba Cloud’s IaaS service and resources like RAM, OSS, ECS, security groups, firewall, Action Trail, and just to name a few.

1. Identity and Access Management

A root account has unrestricted access to all resources in an Alibaba Cloud account, so it is NOT recommended to use this account on a regular basis. Instead, create subaccounts using Alibaba Cloud’s Resource Access Management (RAM) service.

Ensure multi-factor authentication (MFA) is enabled for all RAM users that have a console password (Scored). Perform the following to determine if a MFA device is enabled for all RAM users having a console password. To Audit your security setting visit the RAM console at https://ram.console.aliyun.com/overview

Set multi-factor authentication (MFA) for the root user

Proceed to the next screen.

Identity Verification screen.

MFA Activated after the Identity verification

From now Alibaba cloud login ask for MFA tokens

Ensure credentials unused for 90 days or greater are disabled: Audit your RAM user on daily basis, on time user and groups may increase as people come an leave the organization, or the account created for some automation/testing purpose left in Alibaba RAM console, therefore audit of these account needs to be validated on regular basis to avoid any backdoor entry.

Ensure access keys are rotated every 90 days or less: Access keys should be rotated to ensure that data cannot be accessed with an old key which might have been lost, cracked, or stolen.

Ensure RAM policies are attached only to groups or grants

Create groups and assign users to the groups
Create custom policies following the least privilege principle and assign the policies to groups rather than individual users
Audit built-in policies and custom before assigning it.

Don’t embed RAM access keys directly into code use the encrypted form utilize KMS. Use different RAM access keys for different applications, never use the same key for more than one application, for devops perspective your dev/test/qa/stag/prod keys should differ. Finally, rotate RAM access keys periodically.

You can find more tips about account security on this guide Best practice of primary account security.

2. Enable ActionTrail

Alibaba Cloud ActionTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your Alibaba Cloud account. With ActionTrail, you can log, continuously monitor, and retain account activity related to actions across your Alibaba infrastructure. ActionTrail provides event history of your Alibaba account activity. This event history simplifies security analysis, resource change tracking, and troubleshooting.

If you are not sure where to start, refer to this guide for Creating your ActionTrail

Important cloud resources to be trailed for audit and compliance

VPC flow
OSS Bucket
Service Load Balancer logs
Traffic flows
Key Management Services

3. KMS and Encryption Setup

Alibaba Cloud Key Management Service (KMS), which is a secure and easy-to-use management service provided by Alibaba Cloud. KMS allows you to use keys securely and conveniently and focus on developing encryption and decryption functions, without having to spend a great deal in protecting the confidentiality, integrity, and availability of keys.

Key protection with Scheduled key rotation: Every organization has their key rotation, based on organizational needs. It is advisable to rotate the key as per the key rotation policy.

Benefits of KMS Encryption:

Through the envelop encryption technology, users can store the Customer Master Key (CMK) in KMS and deploy only the encrypted data key, and users can call KMS to decrypt the data key only when they need to use it.
Protect the HTTPS certificate on the server
perform local encryption and decryption

4. Protect Data Stored in OSS Buckets

Typically, an OSS bucket name is not a secret, and there are many ways to figure it out. Once an attacker knows it, there are multiple misconfigurations that can be used to either access or modify information stored in the OSS bucket policy. Since many companies store sensitive data in OSS buckets, any data leaks could be devastating.

Before creating any bucket, you need to audit it to detect the following vulnerabilities:

OSS bucket allows for full anonymous access
OSS bucket allows for arbitrary file listing
OSS bucket allows for arbitrary file upload and exposure
OSS bucket allows for blind uploads
OSS bucket allows arbitrary read/writes of objects
OSS bucket reveals ACP/ACL

To protect of any accidental leakage each bucket policy needs to be thoroughly checked

Any OSS bucket name can be reached through using https url using public IP, what makes the security important here is to review the ACL permissions of it, for example in the below screen shot a private bucket is created, developer might assume private means not reachable through public IP, the moment a bucket is created it’s assigned with a DNS name which can be publicly accessible, if it’s ACL permission is manipulated

Logging

Logging MUST be enabled for the OSS buckets and it’s classification should mapped to data classification as per your organization policy.

Lifecycle

Backups and logs are two types of data which is mostly stored in the OSS bucket, it’s important to have the regulatory requirement of managing the lifecycle of OSS buckets.

Encryption

When creating an object in the OSS bucket, you only need to add the HTTP header, x-oss-server-side-encryption, to the Put Object request and specify its value as AES256 to encrypt and store the object on the server. This method is ideal for bulk data's encryption and decryption.

Learn more about storage security in these guides:

5. TLSv1.2 on Server Load Balancer

A better TLS security can be applied to your exposed only with right TLS policy in place which in large would be able to mitigate the following vulnerability from your exposed endpoint.

CCS injection vulnerability
renegotiation vulnerabilities
CRIME vulnerability
BREACH vulnerability
POODLE (SSL) vulnerability
FREAK vulnerability
BEAST vulnerability
LOGJAM vulnerability

SLB tls_cipher_policy_1_2 provides good compatibility and high security and support only cipher suites like and it's required compliance for PCI-DSS,HIPPA and related standard.

ECDHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA256, AES256-SHA256, ECDHE-RSA-AES128-SHA, ECDHE-RSA-AES256-SHA, AES128-SHA, AES256-SHA and DES-CBC3-SHA

Whenever you apply or change HTTPS listener policy to SLB, test it for the compliance perspective.

6. Reduce External Exposure of Alibaba Cloud Resources

As part of the design of your offering, you should have controls in place to ensure an ECS can only access their data and resources they are authorized to access. What controls can you put in place? What assurances can you offer to your ECS instance that their data can’t be accessed by another ECS? This all bubbles down to the least privilege principle: the less you expose the offering to external world, the more secure design can be achieved.

In Alibaba Cloud this can be achieved through security group and network segregation of your offering.

Security group plays important role here, to segregate traffic,

Operates at the instance level (first layer of defense)
Supports allow rules only
Is statefull: Return traffic is automatically allowed, regardless of any rules
Evaluate all rules before deciding whether to allow traffic
Applies to an ECS instance only if someone specifies the security group when launching the instance, or associates the security group with the instance later on

Network segregation: After creating a VPC, the next logical construct is the vSwitch. vSwitch in Alibaba cloud are sub-networks within a VPC and are analogous to the subnets. One can add one or more subnets in each availability zone; however, each subnet must reside exclusively within one AZ and cannot span AZs. Here is one example how the production env is segregated in two AZ operating on different vSwitch and one staging environment which is on complete different vSwitch

7. Secure Bastion Hosts

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet.

Here is the architecture diagram

First, create a security group that will be used to allow bastion connectivity. This SG should only accept SSH or RDP inbound requests from your bastion hosts across your Availability Zones (AZ)

Use this bastion hosts for your existing private instances. Apply this group to all of your private instances that require connectivity.

However, once you have connected to your bastion host, logging in to your private instances from the bastion would require having their private keys on the bastion, to overcome this situation, utilize SSH agent forwarding in the ssh config file, for example the below ssh config file uses ForwardAgent capablity

Host 8gwifi
    HostName 47.88.57.195
    Port 22
    User alicloud
    IdentityFile /root/.ssh/id_rsa
    IdentitiesOnly yes
    ForwardAgent yes

And then connect to your production instances without storing private keys in the bastion hosts.

ssh root@47.88.57.195  --> Bastion server
ssh 192.168.1.3   --> From Bastion to production server

Some security points to be considered:

Bastion hosts is single point of accessing the private ECS instance; its security should be considered high enough.
Utilize SSH Identity based login to connect to bastion hosts.
Harden the bastion by limiting the access from given public IP or IP range.
Monitor the logins.
Disable root logins on bastion hosts.
Have different bastion hosts to connect to production vs development environment.

8. Hardening ECS OS Images

By default ECS provisioned OS images are open to world, it’s security posture is not much as it’s required for the production environment . This choice is left to end-user to take this responsibility of the OS hardening which is provisioned in the Alibaba ECS. The requirement should be approved by the hardening process or by leveraging an approved custom images.

The end user can benefit from the CIS benchmarking of the OS hardening process. Once the ECS OS is process by CIS benchmarking tool, the end user will be presented with the action to be taken off.

For example the given screen-shot is example of CIS hardening requirement of the Linux kernel image.

9. Vulnerability and Penetration Testing of ECS Instance

Before starting this activity, you need to request for permission from Alibaba to your root account. Once you permission is granted, then only you can start the Vulnerability Testing(VT) and Penetration Testing (PT) activity. Otherwise any suspected traffic spikes will be block by Alibaba Cloud Security Intelligence.

VT and PT is must perform activity of your production service, this ensure compliance and security governance of your managed resource.

Vulnerability management process ensure all the critical patch applied on the timely manner and all the required asset particularly ECS are covered.

Continuous Penetration testing process ensure, application exposed on Alibaba cloud service, are safe from black hat hackers and a continuous effort is gone to achieve the web security.

10. Monitoring

There are many resources in the Alibaba cloud which needs to be monitored performing any manual scrubbing on the logs are error prone tasks so take help of monitoring solution.

The ActionTrail events should be filtered so that only certain important logs can be seeded to the monitoring system like splunk, elasticsearch etc .

The filter should be revised over time as new and different attack patterns are discovered.

Some of the list of events which can be actively monitors for Alibaba cloud resources

Root sign-in on console
RAM Accounts without MFA
Production security group changes
User, Group, and Role Membership Events
Billing
KMS key changed/deleted
OSS bucket API keys reads/writes
Casual Logins (Login after certain period )

Some of the list of events which can be actively monitors for Alibaba ECS Linux Instances

Monitored from syslog events:

Nginx/httpd or apache web servers errors logs
SSH too many failed authentication
SSH successful logins
sudo commmand invocations

Monitored from system processes:

Processes with listening sockets
Open connection
CLOSE_WAIT
Zombie process

Monitored from the system:

User and group list
Group membership changes

Monitored from the file system:

SSH authorized_keys for users
SSH known_hosts for users
APT /YUM GPG keys changes/added

Learn more about monitoring on Alibaba Cloud through these guides:

11. Incident Management and Response

This is last point I would cover and plays an important role in maintaining your Alibaba Cloud security posture referred as Incident management and response. Even by following all of the above best practices, errors may occur when you are deploying your solution on Alibaba Cloud. To ensure business continuity, you need to plan for incident management and response in advance.

The incident can occur during your VT and PT testing of your managed resources which is considered as internal incident and incident which is reported from outside your organization considered as external incident management. To prioritize and remediate those incident a separate task force needs to be assigned whose main function is to be take the closure.

Automation can help automate and co-relate incident events with ongoing incidents, it’s important to responds to the incident on the timely manner, failing to do so will result in risking on business and resources. Alibaba Cloud Log Service can also be used as on the feeder to security information and event management (SIEM) software products like splunk

Conclusion

Root account shouldn’t be use for any provisioning, this account is having highest level of privilege, ensure you have protected with MFA.
For any endpoints that is exposed publicly, build a security control around it, utilize the least privilege and defense of depth principle.
Security is continuous, and this continuity can be achieved through proper approved automation and well defined process.
Never ignore any security events or alerts.
Care and due diligence is the key of maintaining the Alibaba Cloud security posture healthy.
Alibaba Cloud is committed towards data protection for all customers. Visit our Security and Compliance Center to learn more.

Reference:https://www.alibabacloud.com/blog/11-security-recommendations-for-production-instances-on-alibaba-cloud_594743?spm=a2c41.12842847.0.0