Hello!
All good stories come to an end. And our story about how we devised a solution for quickly bypassing the Chinese Firewall is no exception. So I'm eager to share with you the final part. the concluding part on this topic.
In the previous part, we talked about the numerous test setups we came up with and the results they yielded. We concluded that it would be good to add CDN! for cohesion in our scheme.
I will tell you how we tested Alibaba Cloud CDN, Tencent Cloud CDN, and Akamai, and which one we ultimately chose. And of course, we will summarize.
Alibaba Cloud CDN
We are hosted on Alibaba Cloud, using IPSEC and CEN from them as well. It makes sense to first try their solutions.
Alibaba Cloud has two types of products that might suit us: CDN and DCDN. The first option is a classic CDN for a specific domain (subdomain). The second option stands for Dynamic Route for CDN (which I call dynamic CDN), it can be activated in Full-site mode (for wildcard domains), it also caches static content and accelerates dynamic content, meaning the dynamic aspects of the page will also load through the provider's fast networks. This is important for us, as our website is mostly dynamic, using numerous subdomains, and it's easier to configure CDN once for a wildcard — *.semrushchina.cn.
We have already seen this product in earlier stages of our Chinese project, but at that time it wasn't functional, and the developers promised that the product would soon be available to all customers. And it has become available.
With DCDN, you can:
- set up SSL termination with your certificate,
- enable dynamic content acceleration,
- flexibly configure caching for static files,
- perform cache purges,
- support web sockets,
- enable compression, and even HTML Beautifier.
In general, everything is like with mature, large CDN providers.
After specifying the Origin (the place where CDN edge servers will go), you then need to create a CNAME for the wildcard that points to all.semrushchina.cn.w.kunluncan.com (this CNAME was obtained in the Alibaba Cloud console), and the CDN will function.
Based on the test results, this CDN has helped us significantly. The statistics are provided below.
Solution
Uptime
Median
75th Percentile
95th Percentile
Cloudflare
86.6
18s
30s
60s
IPsec
99.79
18s
21s
30s
CEN
99.75
16s
21s
27s
CEN/IPsec + GLB
99.79
13s
16s
25s
Ali CDN + CEN/IPsec + GLB
99.75
10s
12.8s
17.3s
These are very good results, especially when compared to the figures we had at the beginning. However, we knew that the browser test of the American version of our website www.semrush.com averages around 8.3s (a rough estimate) from the US. There's still room for improvement. Moreover, there were additional CDN providers that we were interested in testing.
So we smoothly transition to another giant in the Chinese market — Tencent.
Tencent Cloud
Tencent is still developing its cloud — this is evident from the limited number of products. During its use, we wanted to test not only their CDN but also the overall network infrastructure:
- do they have something similar to CEN?
- how does their IPSEC work? Is it fast, what is the uptime?
- do they have Anycast?
We will address these questions separately.
CEN Analog
Tencent has a product Cloud Connect Network (), which allows connecting VPCs from different regions, including regions within and outside China. The product is currently in internal beta, and you need to create a ticket requesting access. From support, we learned that global accounts (not referring to Chinese citizens or legal entities) cannot participate in the beta testing program and generally cannot connect a region inside China with a region outside. 1-0 in favor of Ali Cloud.
IPSEC
The southernmost region for Tencent is Guangzhou. We set up a tunnel and connected it to the Hong Kong region in GCP (at that point the region was already available). We also launched a second tunnel to Ali Cloud from Shenzhen to Hong Kong. It turned out that over Tencent's network, latency to Hong Kong was overall better (10ms) than from Shenzhen to Hong Kong in Ali (120ms — what?). However, this did not accelerate the website's performance aimed at running through Tencent and this tunnel, which is itself an astonishing fact and once again proved the following: latency — for China this is not a metric to really pay attention to when developing a solution for bypassing the Chinese firewall.
Anycast Internet Acceleration
Another product that allows working through anycast IP is . However, it is also not available for global accounts, so I won't elaborate on it, but knowing that such a product exists might be useful.
The CDN test yielded quite interesting results. Tencent's CDN cannot be enabled for full-site use, only for specific domains. We set up domains and directed traffic to them:
It turned out that this CDN has the following feature: Cross Border Traffic Optimization. This feature should reduce costs when passing traffic through the Chinese firewall. As an example, Origin the IP address of Google's GLB (GLB anycast) was specified. Thus, we aimed to simplify the project's architecture.
The results were very good — on par with Ali Cloud CDN, and in some cases even better. This is surprising because if the tests are successful, it would allow us to eliminate a significant part of the infrastructure, tunnels, CEN, virtual machines, etc.
Our joy was short-lived, as a problem emerged: tests in Catchpoint failed for the Internet provider China Mobile. From any location, we received timeouts through Tencent's CDN. Correspondence with technical support led to nothing. For about a day, we attempted to resolve this issue, but to no avail.
At that moment, I was in China, but I couldn't find public Wi-Fi in this provider's network to verify the issue personally. Everything else seemed fast and good.
However, due to the fact that China Mobile is among the top three largest operators, we were forced to revert the traffic to Ali CDN.
Overall, it was quite an interesting solution that deserves longer testing and troubleshooting of this problem.
Akamai
The last CDN provider we tested was Akamai. This is a large provider that has its own network in China. Of course, we couldn't overlook it.
From the very beginning, we agreed with Akamai on a trial period so we could switch the domain and see how it would perform on their network. I will describe the results of all the testing in terms of 'What I liked' and 'What I didn't like', as well as provide the test results.
What I liked:
- The guys from Akamai were very helpful with all our questions and supported us at all stages of testing. They were constantly trying to improve something on their end and provided good technical advice.
- Akamai works about 10-15% slower than our solution through Ali Cloud CDN. It’s impressive that for Akamai's Origin, we specified the GLB IP address, meaning the traffic did not go through our solution (potentially allowing us to eliminate part of the infrastructure). However, the test results showed that this solution was worse than our current one (comparative results below).
- Both Origin GLB and Origin in China were tested. Both options are approximately the same.
- There is Sure Route (automatic routing optimization). You can place a test object on Origin, and Akamai Edge servers will attempt to fetch it (standard GET). The speed and other metrics for these requests are measured, based on which the Akamai network optimizes routes to ensure traffic flows faster for our site, demonstrating that enabling this feature significantly impacts website speed.
- Versioning configuration in the web interface is great. You can do a Compare for versions, view the diff, and check previous versions.
- You can deploy a new version first only on the Akamai Staging network — the same network as production, but this path does not affect real users. For this test, DNS record spoofing needs to be done on the local machine.
- Very fast loading speed through their network for large static files, as well as apparently any other files. A file from the 'cold' cache is retrieved much faster than the same file from the 'cold' cache of Ali CDN. From the 'hot' cache, the speed is more or less the same.
Ali CDN test:
root@shenzhen1:~# curl -o /dev/null -w@curl_time https://en.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5757k 0 5757k 0 0 513k 0 --:--:-- 0:00:11 --:--:-- 526k
time_namelookup: 0.004286
time_connect: 0.030107
time_appconnect: 0.117525
time_pretransfer: 0.117606
time_redirect: 0.000000
time_starttransfer: 0.840348
----------
time_total: 11.208119
----------
size_download: 5895467 Bytes
speed_download: 525999.000B/sAkamai test:
root@shenzhen1:~# curl -o /dev/null -w@curl_time https://www.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5757k 0 5757k 0 0 1824k 0 --:--:-- 0:00:03 --:--:-- 1825k
time_namelookup: 0.509005
time_connect: 0.528261
time_appconnect: 0.577235
time_pretransfer: 0.577324
time_redirect: 0.000000
time_starttransfer: 1.327013
----------
time_total: 3.154850
----------
size_download: 5895467 Bytes
speed_download: 1868699.000B/sWe noticed that the situation from the example above depends on various factors. At the time of writing this point, I conducted the test again. The results for both platforms turned out to be approximately the same. This tells us that the internet in China behaves differently from time to time, even for large operators and cloud providers.
Adding a big plus for Akamai to the previous point: while Ali shows such bursts of high performance and very low latency (this applies to both Ali CDN, Ali CEN, and Ali IPSEC), every time I test Akamai's network, everything works consistently.
Akamai really has extensive coverage in China and works through many providers.
What we didn't like:
- I don't like the web interface and the way it operates — it's very rudimentary. But in principle, you get used to it (probably).
- Test results are worse than our platform.
- There are more errors in the tests than on our platform (uptime is lower).
- They do not have their own DNS servers in China. This leads to many test errors due to DNS resolve timeout.
- They do not provide their IP ranges -> there is no way to write them down correctly. set_real_ip_from on our servers.
Metrics (~3626 runs; all metrics except Uptime in ms; statistics over a single time interval):
CDN Provider
Median
75%
95%
Response
Webpage Response
Uptime
DNS
Connect
Wait
Load
SSL
Ali CDN
9195
10749
17489
1,715
10,745
99.531
57
17
927
479
200
Akamai
9783
11887
19888
2,352
11,550
98.980
424
91
1408
381
50
Distribution by Percentile (in ms):
Percentile
Akamai
Ali CDN
10
7,092
6,942
20
7,775
7,583
30
8,446
8,092
40
9,146
8,596
50
9,783
9,195
60
10,497
9,770
70
11,371
10,383
80
12,670
11,255
90
15,882
13,165
100
91,592
91,596
The conclusion is this: the option with Akamai is viable but does not provide the same stability and speed metrics as our own solution combined with Ali CDN.
Little notes
Some points did not make it into the narrative, but I would also like to write about them.
Beijing + Tokyo and Hong Kong
As I mentioned earlier, we tested the IPSEC tunnel to Hong Kong (HK). But we also tested CEN to HK. It costs a little less, and it was interesting to see how it would perform between cities separated by ~100km. Interestingly, the latency between these cities was 100ms higher than in our original option (to Taiwan). The speed and stability were also better for Taiwan. In the end, we left HK as a backup IPSEC region.
In addition, we tried to implement such an installation:
- terminating clients in Beijing,
- IPSEC and CEN to Tokyo,
- in Ali CDN specified as the origin server in Beijing.
This setup was not as stable, although in terms of speed it was generally not inferior to our solution. Regarding the tunnel, I observed periodic drops even for CEN, which was supposed to be stable. Therefore, we reverted to the old setup and dismantled this staging.
Below is the latency statistics between different regions over different channels. Perhaps someone will find it interesting.
IPsec
Ali cn-beijing GCP asia-northeast1 — 193ms
Ali cn-shenzhen GCP asia-east2 — 91ms
Ali cn-shenzhen GCP us-east4 — 200ms
CEN
Ali cn-beijing Ali ap-northeast-1 — 54ms (!)
Ali cn-shenzhen Ali cn-hongkong — 6ms (!)
Ali cn-shenzhen Ali us-east1 — 216ms
General Information about the Internet in China
As an addition to the internet issues mentioned at the very beginning, in the first part of the article.
- The internet in China works quite fast domestically.
- The conclusion is based on testing public Wi-Fi networks in various locations where these networks are used by a large number of people.
- The download and upload speeds to servers inside China were about 20 Mbps and 5-10 Mbps, respectively.
- The speed to servers outside China is simply negligible, less than 1 Mbps.
- The internet in China is not very stable.
- Sometimes websites may load quickly, sometimes slowly (at the same time of day on different days), provided that the configuration does not change. We observed this with semrushchina.cn. This can be attributed to Ali CDN, which also operates sporadically depending on the time of day, position of the stars, etc.
- Mobile internet is practically everywhere 4G or 4G+. It works in subways, elevators — basically, everywhere.
- The belief that Chinese users only trust domains in the .cn zone is a myth. We verified this directly with users.
- You can see how redirects to www.baidu.com (also in mainland China).
- Many resources are indeed blocked. Simply put: google.com, Facebook, Twitter. But many Google resources work (of course, not on all Wi-Fi and VPN is not used (on the router side too, that’s for sure).
- Many "technical" domains of blocked corporations also work. This means that one should not recklessly eliminate all seemingly blocked Google and other resources. A list of banned domains should be sought.
- They have only three major internet operators: China Unicom, China Telecom, China Mobile. There are smaller ones, but their market share is insignificant.
Bonus: Final solution scheme
Summary
A year has passed since the start of the project. We started with the fact that our site simply refused to work properly from China, and a simple GET curl took 5.5 seconds.
Then, with such indicators at the first solution (Cloudflare):
Solution
Uptime
Median
75th Percentile
95th Percentile
Cloudflare
86.6
18s
30s
60s
Eventually, we reached the following results (statistics for the last month):
Solution
Uptime
Median
75th Percentile
95th Percentile
Ali CDN + CEN/IPsec + GLB
99.86
8.8s
9.5s
13.7s
As you can see, achieving 100% uptime has not yet been possible, but we will think of something, and then we will share the results with you in a new article :)
Respect to those who read all three parts to the end. I hope you found it as interesting as I did while creating this.
P.S. Previous Parts
Source: habr.com
