How Fast Is Alibaba Cloud CDN: An In-Depth Performance Analysis
By Valerie YJ Lee, Solutions Architect
The purpose of this study is to evaluate the performance factors of Alibaba Cloud’s CDN service by region by measuring object and web page load time. We will be running two types of tests, as described below.
Object Load Test
This test is that measures the NETWORK PERFORMANCE by periodically loading certain objects from distributed CDN nodes.
Test URL examplehttp://www.abc.com/img/product/test.png (QT011_100K)Test toolCatchPointTest nodesKR (Seoul), HK (Hong Kong), TW (Taipei), JP (Tokyo), SG (Singapore), CN (Beijing, Shanghai), VN (Hanoi), ID (Jakarta), MY(Kuala Lumpur), TH (Bangkok), US (New York, San Jose), AU (Sydney), CA (Toronto) — 15
The test was performed twice for comparing the improvements before and after CDN tuning.
Page Load Test
This test is similar with object load test, but this measures the whole page response time in various locations. Therefore, this evaluates the UX LEVEL PERFORMANCE rather than network environment.
Test URL example 1http://www.abc.com/img/Prd.mall?spm=a2c41.11461447.0.0.3fea52e8Lmsfng&method=getPDtail&prdNo=1(QT021)
http://www.abc.com/img/Search.mall?spm=a2c41.11461447.0.0.3fea52e8Lmsfng&method=getSDtail&isGnb=N(QT022)Test toolCatchPointTest nodesKR (Seoul), HK (HongKong), TW (Taipei), JP (Tokyo), SG (Singapore), CN (Beijing, Shanghai), VN (Hanoi), ID (Jakarta), MY(Kuala Lumpur), TH (Bangkok), US (New York, San Jose), AU (Sydney), CA (Toronto), Dubai, GM (Frankfurt), PH (Manila) — 18Test URL example 2 http://www.xyz.com/index.html?spm=a2c41.11461447.0.0.3fea52e8LmsfngTest toolGomezTest nodesHK (HongKong), JP (Tokyo), CN (Beijing, Shanghai), TW (Taipei)
Object Load Test
Before reading, make sure that you remember these factors that indicate good performance
- A smaller gap between median(green line) and the 75 percentile (khaki line)
- Smaller occurrences below median (converges to zero)
- Less frequent occurrences of over 75 percentile
Before Tuning
- Median-75 percentile distance: around 700
- 0–75 percentile: around 50%
- The proportion of 75 percentile or more: around 25%
After Tuning
- Median-75 percentile distance: around 600
- 0–75 percentile: around 40%
- The proportion of 75 percentile or more: around 17%
A/B test comparison
- Median-75 percentile distance decreased from around 700 to around 600. This shows that “Local deviation” slightly decreased.
- 0–75 percentile increased from 50% to around 40%. This means that “good performance area” increased.
- The proportion of 75 percentile or more decreased from 25% to 17%. This describes that reduction of “bad performance area”
Summary of Results
- Result table by testing nodes
- A means “Before tuning” and B means “After tuning”
- Response = DNS + Connect + Wait + Load
1) Response time in HK, JP, TW and SG are very good. In US San Jose, Shanghai and Bangkok, performance is good (Shanghai has no CDN)
2) Beijing supplement needed — After ICP completion
3) ID and MY need to be supplemented
Page Load Test
Test by CatchPoint
POC vs. Production
The below table shows performance in POC environment (without CDN vs. Alibaba CDN vs. third-party vendor CDN)
These charts and table show performance in production environment.
In production environment, the response time decreased from 26 seconds to 12 seconds (Test URL 1) and from 12 seconds to 10 seconds with Alibaba CDN.
Detailed performance data in production environment
The red dots mean “bytes limit exceeds” error of own test tool.
- Both of two tests’ Median-75 percentile distance is less than 2.5 seconds compared to total distribution (31 seconds). This converts to overall quality average.
- Each test’s 0–75 percentile is analyzed to be around 75% and 60%. Comparing two tests, QT022 provides better UX.
- Both of two tests are analyzed as a percentage of more than 75 percentile, but QT022 is better
Summary of Results
- Generally, the VPN is quite good quality in most areas (including Europe and the Middle East)
- Beijing, Shanghai need to complement — After ICP is completed
- Need to supplement MY — Planed to significantly reinforce the CDN nodes
Test by Gomez
Comparison of Alibaba Cloud CDN with Third-Party Vendor
Summary of Results
- Alibaba CDN shows superior performance compared to other CDNs for all indicators in China
- Alibaba CDN maintains performance similar to other companies in mainland China and other Asian countries (Taiwan, Hong Kong, Japan)
Conclusion
Alibaba Cloud CDN provides excellent network performance and UX quality, and has a local and temporal distribution of response time. In particular, it can be seen that the response time in China is greatly improved with our CDN product. We are constantly increasing the number of nodes in various countries around the world, so our customers can expect better response times globally.
