Hi!
All good stories come to an end. And our story about how we came up with a Chinese Firewall fast pass solution is no exception. Therefore, I hasten to share with you the latest, final part about this theme.
In the previous part, it was told about the many test benches that we invented, and what results they gave. And we settled on what would be nice to add CDN! for viscosity in our scheme.
I will tell you how we tested Alibaba Cloud CDN, Tencent Cloud CDN and Akamai, and where we ended up. And of course, let's sum it up.
Alibaba Cloud CDN
We are hosted on Alibaba Cloud, we use IPSEC and CEN from them. It would be logical to try their solutions first.
Alibaba Cloud has two kinds of product that can suit us: CDN ΠΈ DCDN. The first option is a classic CDN for a specific domain (subdomain). The second option is deciphered as Dynamic Route for CDN (I call it dynamic CDN), it can be enabled in Full-site mode (for wildcard domains), it also caches statics and accelerates dynamic content on itself, that is, page dynamics will also be loaded through the provider's fast networks. This is important for us, because basically our site is dynamic, it uses many subdomains, and it is more convenient to set up a CDN once for the βasteriskβ - *.semrushchina.cn.
We have already seen this product in the earlier stages of our Chinese project, but then it did not work yet, and the developers promised that the product would soon be available to all customers. And he became.
In DCDN you can:
- configure SSL termination with your certificate,
- enable dynamic content acceleration,
- flexibly configure caching of static files,
- purge cache,
- forward web sockets,
- enable compression and even HTML Beautifier.
In general, everything is like in adults and large CDN providers.
After the Origin (the place where the CDN edge servers will go) is specified, then it remains to create a CNAME for the asterisk, referring to all.semrushchina.cn.w.kunluncan.com (this CNAME was received in the Alibaba Cloud console), and the CDN will work.
According to the test results, this CDN helped us a lot. The statistics are shown below.
Solution
Uptime
Median
75 percentile
95 percentile
Cloudflare
86.6
18s
30s
60s
IPsec
99.79
18s
21s
30s
CEN
99.75
16s
21s
27s
CEN/IPsec+GLB
99.79
13s
16s
25s
Ali CDN + CEN/IPsec + GLB
99.75
10s
12.8s
17.3s
These are very good results, especially if we compare them with what the figures were at the beginning. But we knew that the browser test of the US version of our website www.semrush.com runs from the US in 8.3s on average (a very approximate value). There is something to strive for. Moreover, there were also CDN providers that were interesting to test.
So we are smoothly moving on to another giant in the Chinese market - Tencent.
Tencent Cloud
Tencent is only developing its cloud - this can be seen in a small number of products. In the course of using it, we wanted to test not only their CDN, but the whole network infrastructure:
- do they have something similar to CEN?
- how does IPSEC work for them? Is it fast, what is the uptime?
- do they have Anycast?
Let's examine these questions separately.
CEN analog
Tencent has a product Cloud Connect Network (), allowing you to interconnect VPCs from different regions, including regions inside China and outside. The product is currently in internal beta, and you need to create a ticket with a request to connect to it. From support, we learned that global accounts (we are not talking about Chinese citizens or legal entities) cannot participate in the beta testing program and, in general, connect the region inside China with the region outside. 1-0 in favor of Ali Cloud
IPsec
Tencent's southernmost region is Guangzhou. We assembled a tunnel and connected it to the Hong Kong region in the GCP (then this region was already available). The second tunnel in Ali Cloud from Shenzhen to Hong Kong was also lifted at the same time. It turned out that through the Tencent network, latency to Hong Kong is generally better (10ms) than from Shenzhen to Hong Kong to Ali (120ms - what?). But this did not in any way speed up the work of the site aimed at working through Tencent and this tunnel, which in itself was an amazing fact and once again proved the following: latency for China is not an indicator that you really should pay attention to when developing a solution for passing the Chinese firewall.
Anycast Internet Acceleration
Another product that allows you to work through anycast IP - . But it is also not available to global accounts, so I wonβt talk about it, but knowing that such a product exists can be useful.
But the CDN test showed quite interesting results. Tencent's CDN cannot be enabled on a full-site, only on specific domains. We set up domains and sent traffic to them:
It turned out that this CDN has the following function: Cross Border Traffic Optimization. This feature should reduce costs when traffic passes through the Chinese firewall. As Origin the IP address of the Google GLB (GLB anycast) was specified. Thus, we wanted to simplify the architecture of the project.
The results were very good - at the level of Ali Cloud CDN, and in some places even better. This is surprising, because if the tests are successful, you can abandon a significant part of the infrastructure, tunnels, CEN, virtual machines, etc.
We didnβt rejoice for long, as a problem was revealed: the tests in Catchpoint were failing for the Internet provider China Mobile. We received a timeout from any location via Tencent's CDN. Correspondence with technical support did not lead to anything. For about a day we tried to solve this problem, but nothing came of it.
I was in China at the time, but I couldn't find public Wi-Fi on that provider's network to see for myself. Otherwise, everything looked fast and good.
However, due to the fact that China Mobile is one of the three largest operators, we were forced to return traffic to Ali CDN.
But in general it was quite an interesting solution that deserves more testing and troubleshooting of this problem.
Akamai
The last CDN provider we tested is Akamai. This is a huge provider that has its own network in China. Of course, we couldn't get past it.
From the very beginning, we agreed with Akamai on a trial period so that we could switch the domain and see how it would work on their network. I will describe the result of all testing in the form of βWhat I likedβ and βWhat I didnβt likeβ, and I will also give the test results.
What I liked:
- The guys from Akamai were very helpful in all matters and accompanied us at all stages of testing. Constantly trying to improve something on their side. They gave good technical advice.
- Akamai is about 10-15% slower than our solution via Ali Cloud CDN. It is impressive that in Origin for Akamai we specified the GLB IP address, that is, the traffic did not go through our solution (it is possible to potentially abandon part of the infrastructure). But still, the test results showed that this solution is worse than our current version (comparative results below).
- Tested by both Origin GLB and Origin in China. Both options are about the same.
- There is Sure route (automatic routing optimization). You can host a test object on Origin, and the Edge servers of Akamai will try to pick it up (normal GET). For these requests, speed and other metrics are measured, on the basis of which the Akamai network optimizes routes so that traffic goes faster for our site and it was clear that the inclusion of this feature really had a big impact on the speed of the site.
- Versioning the configuration in the web interface is cool. It is possible to make Compare for versions, to look at diff. View previous versions.
- You can roll out a new version first only on the Akamai Staging network - the same network as production, only this way will not affect real users. For this test, you need to spoof DNS records on the local machine.
- Very fast download speed through their network of large statics, and, apparently, any other files. A file from a βcoldβ cache is taken many times faster than the same file from a βcoldβ Ali CDN cache. From the βhotβ cache, the speed is plus or minus the same.
Ali CDN test:
root@shenzhen1:~# curl -o /dev/null -w@curl_time https://en.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5757k 0 5757k 0 0 513k 0 --:--:-- 0:00:11 --:--:-- 526k
time_namelookup: 0.004286
time_connect: 0.030107
time_appconnect: 0.117525
time_pretransfer: 0.117606
time_redirect: 0.000000
time_starttransfer: 0.840348
----------
time_total: 11.208119
----------
size_download: 5895467 Bytes
speed_download: 525999.000B/sAkamai test:
root@shenzhen1:~# curl -o /dev/null -w@curl_time https://www.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5757k 0 5757k 0 0 1824k 0 --:--:-- 0:00:03 --:--:-- 1825k
time_namelookup: 0.509005
time_connect: 0.528261
time_appconnect: 0.577235
time_pretransfer: 0.577324
time_redirect: 0.000000
time_starttransfer: 1.327013
----------
time_total: 3.154850
----------
size_download: 5895467 Bytes
speed_download: 1868699.000B/sWe noticed that the situation from the example above depends on various factors. At the time of this writing, I ran the test again. The results for both platforms were about the same. This tells us that the Internet in China, even for large operators and cloud providers, behaves differently from time to time.
Iβll add Akamaiβs big plus to the previous point: if Ali shows similar flashes of high performance and very low (this applies to Ali CDN, Ali CEN, and Ali IPSEC), then Akamai every time, no matter how I test their network, everything works stably.
Akamai do have a great coverage in China and work through many providers.
What did not like:
- I donβt like the web interface and the scheme of work - they are so zero. But in principle you get used to it (probably).
- The test results are worse than our site.
- There are more errors during tests than on our site (uptime below).
- There are no DNS servers in China. Hence a lot of errors in the tests due to DNS resolve timeout.
- They do not provide their IP ranges -> there is no way to register the correct ones set_real_ip_from on our servers.
Metrics (~3626 runs; all metrics, except Uptime, in ms; statistics for one period of time):
CDN Provider
Median
75%
95%
Response
Webpage Response
Uptime
DNS
Connect
Wait
Load
SSL
AliCDN
9195
10749
17489
1,715
10,745
99.531
57
17
927
479
200
Akamai
9783
11887
19888
2,352
11,550
98.980
424
91
1408
381
50
Distribution by Percentile (in ms):
percentile
Akamai
AliCDN
10
7,092
6,942
20
7,775
7,583
30
8,446
8,092
40
9,146
8,596
50
9,783
9,195
60
10,497
9,770
70
11,371
10,383
80
12,670
11,255
90
15,882
13,165
100
91,592
91,596
The conclusion is this: the Akamai option is viable, but does not provide the same stability and speed indicators as our own solution, coupled with Ali CDN.
Little Notes
Some moments were not included in the story, but I would like to write about them too.
Beijing + Tokyo and Hong Kong
As I said above, we tested the IPSEC tunnel to Hong Kong (HK). But we also tested CEN to HK. It costs a little less, and it was interesting how it would work between cities with a distance of ~100km. It turned out to be interesting that the latency between these cities is 100ms higher than in our original version (to Taiwan). Speed, stability was also better for Taiwan. As a result, we left HK as a backup IPSEC region.
In addition, we tried to raise such an installation:
- termination of clients in Beijing,
- IPSEC and CEN to Tokyo,
- in Ali CDN, the server in Beijing was indicated as the origin.
This scheme was not as stable, although in general it was not inferior to our solution in terms of speed. As for the tunnel, I have seen intermittent drops even for CEN, which should have been stable. Therefore, we returned to the old scheme and dismantled this staging.
Below are statistics on latency between different regions via different channels. Maybe someone will be interested in it.
IPsec
Ali cn-beijing <β> GCP asia-northeast1 - 193ms
Ali cn-shenzhen <β> GCP asia-east2 - 91ms
Ali cn-shenzhen <β> GCP us-east4 - 200ms
CEN
Ali cn-beijing <β> Ali ap-northeast-1 - 54ms (!)
Ali cn-shenzhen <β> Ali cn-hongkong - 6ms (!)
Ali cn-shenzhen <β> Ali us-east1 - 216ms
General information about the Internet in China
As an addition to the Internet problems described at the very beginning, in the first part of the article.
- Internet in China is pretty fast inside.
- The conclusion was made on the basis of testing public Wi-Fi networks in various locations where network data is used by a large number of people.
- The download and upload speeds to servers within China were around 20 Mbps and 5-10 Mbps, respectively.
- The speed to servers outside of China is just miserable, less than 1 Mbps.
- The Internet in China is not very stable.
- Sometimes sites can open quickly, sometimes slowly (at the same time of day on different days), provided that the configuration does not change. We observed this in the example of semrushchina.cn. This can be attributed to Ali CDN, which also works this way and that, depending on the time of day, the position of the stars, etc.
- Mobile Internet is almost everywhere 4G or 4G+. Catches in the subway, elevators - in short, everywhere.
- It is a myth that Chinese users only trust domains in the .cn zone. We learned this directly from users.
- You can see how redirects to www.baidu.com (also in mainland China).
- Many resources are indeed blocked. Primitive: google.com, Facebook, Twitter. But many Google resources work (of course, not all Wi-Fi and VPN are not used at the same time (on the side of the router, too, that's for sure).
- Many "technical" domains of blocked corporations also work. This means that you should not always recklessly cut out all the Google and other seemingly blocked resources. You need to look for some list of banned domains.
- They have only three main Internet operators: China Unicom, China Telecom, China Mobile. There are even smaller ones, but their market share is insignificant
Bonus: final solution scheme
Π‘onclusion
A year has passed since the start of the project. We started with the fact that our site generally refused to work normally from China, but simply GET curl took 5.5 seconds.
Then, from such indicators in the first decision (Cloudflare):
Solution
Uptime
Median
75 percentile
95 percentile
Cloudflare
86.6
18s
30s
60s
We ended up with the following results (statistics for the last month):
Solution
Uptime
Median
75 percentile
95 percentile
Ali CDN + CEN/IPsec + GLB
99.86
8.8s
9.5s
13.7s
As you can see, it has not yet been possible to achieve 100% uptime, but we will come up with something, and then we will tell you about the results in a new article :)
To those who have read all three parts to the end - respect. I hope you enjoyed it as much as I did when I did it.
PS Previous parts
Source: habr.com