Hello everyone!
This is Nikita — a systems engineer from the company SEMrush. Today, I will share how we faced the challenge of ensuring our service semrush.com operated reliably in China, and the issues we encountered during this process (considering our data center is located on the east coast of the USA).
This will be a lengthy story, divided into several articles. I will recount how it all went for us: from a completely non-functioning service in China to performance metrics on par with its American version for American users. I promise it will be interesting and useful. So, here we go.
Problems of the Chinese Internet
Even the most distant person from the specifics of network administration has at least heard once about the Great Chinese Firewall. Ooh, sounds cool, right? But what is it, and how does it really work — that's a rather complex question. There are many articles on the internet dedicated to it, but from a technical standpoint, the workings of this firewall are not described anywhere. Which, however, is not surprising. I confess that after a year of working on this, I still can't say exactly how it works, but I can share my observations and practical conclusions. And we will start with the rumors about this firewall.
There are many rumors about this very firewall. Let's compile the main and most interesting ones into a single list:
- Google, Facebook, Twitter, and other similar services are blocked and do not operate in China.
- Any traffic going TO and FROM China is parsed and restricted using machine learning (in the case of suspicious traffic), which significantly slows down the traffic passing through the border.
- Chinese intelligence will hack any encrypted traffic going through their firewall.
- VPN tunnels and IPSEC tunnels are unstable, frequently dropping and constantly being blocked.
- The simpler the encryption, the easier the passphrase used for authenticating/encrypting traffic, the faster it passes through the Chinese firewall.
Here’s what we managed to find out about these rumors:
- Google, Facebook, Twitter, and other similar services are indeed blocked (your K.O.), but many technical domains of Google, for example, are not banned and are operational (like gstatic.com). Hence, it follows that one should not recklessly eliminate all seemingly blocked Google and other resources.
- Any traffic crossing the border indeed adds significant delay to its time. Look at two results. One website, one page, simple GET. curlThe first measurement from China itself (the beautiful city of Shenzhen). The second measurement from outside in Hong Kong (which has sovereignty, and there is no firewall between it and the world). The distance between the cities in a straight line is approximately 30-40 km.
nikita@china-shenzhen:~# curl -o /dev/null -w@curl_time "https://www.semrush.com/info/ebay.com"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 381k 0 381k 0 0 71824 0 --:--:-- 0:00:05 --:--:-- 82832
time_namelookup: 0.004500
time_connect: 0.169342
time_appconnect: 0.723189
time_pretransfer: 0.723499
time_redirect: 0.000000
time_starttransfer: 1.532912
----------
time_total: 5.443407
----------
size_download: 390968 Bytes
speed_download: 71824.000B/s
nikita@china-hongkong:~# curl -o /dev/null -w@curl_time "https://www.semrush.com/info/ebay.com"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 319k 0 319k 0 0 2555k 0 --:--:-- --:--:-- --:--:-- 2573k
time_namelookup: 0.029366
time_connect: 0.030742
time_appconnect: 0.047310
time_pretransfer: 0.047388
time_redirect: 0.000000
time_starttransfer: 0.120793
----------
time_total: 0.124871
----------
size_download: 326755 Bytes
speed_download: 2616740.000B/sNote the time_connect. Overall, you can see the result: the firewall adds an extra 4 seconds, which is incredibly long.
- VPNs and IPSEC tunnels do often drop. I will elaborate on this later. VPN servers used by users tend to get blocked over time (usually within a day of use).
- There’s a belief among people living in China that the simpler the traffic encryption, the faster it crosses the border, because it is easy to discern that there’s nothing illegal in it. Likewise, 'clean' traffic gets more bandwidth and speed, while 'dirty' traffic, which is indecipherable, gets a slower passage. For example, I will show curl to ifconfig.co over HTTPS and HTTP.
curl -o /dev/null -w@curl_time "https://ifconfig.co/"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13 100 13 0 0 2 0 0:00:06 0:00:05 0:00:01 3
time_namelookup: 0.004305
time_connect: 0.397465
time_appconnect: 5.149305
time_pretransfer: 5.149393
time_redirect: 0.000000
time_starttransfer: 5.568847
----------
time_total: 5.568893
----------
size_download: 13 Bytes
speed_download: 2.000B/s
curl -o /dev/null -w@curl_time "http://ifconfig.co/"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13 100 13 0 0 28 0 --:--:-- --:--:-- --:--:-- 28
time_namelookup: 0.004282
time_connect: 0.212457
time_appconnect: 0.000000
time_pretransfer: 0.212484
time_redirect: 0.000000
time_starttransfer: 0.450565
----------
time_total: 0.450620
----------
size_download: 13 Bytes
speed_download: 28.000B/sThere is a 5-second difference in the total load time for 13 bytes. Moreover, testing this multiple times reveals that GET requests over HTTP consistently complete in a similar timeframe, while responses over HTTPS can sometimes take 3, 5, 10, or even 17 seconds. SSL errors occasionally occur:
Unknown SSL protocol error in connection to ifconfig.co:443.
So, what do we have:
- The issues created by the Chinese firewall as described above.
- Pings to external resources and within tunnels are occasionally lost.
- Latency between two points constantly fluctuates, and often it is simply unpredictable. When connecting different cities/regions, one might expect that due to geographical proximity, latency would be lower, but the opposite is often true.
- Internet and communication channels operate at varying speeds. There is a slight dependence on the time of day and day of the week, but not always.
- DNS requests to the outside world from China sometimes exceed acceptable timeouts.
The picture is shaping up to be simply 'excellent'.
As I mentioned, our data center is in the eastern United States, and all of SEMrush consists of dozens of interconnected products, backends, frontends, databases, and all of this is in the data center and the cloud. Our team of system administrators was tasked with quickly becoming operational in China with minimal effort.
We needed to answer an important question: can we solve all the problems associated with the Chinese internet and firewall at the network/cloud/server level with minimal resources?
We started with obtaining .
ICP license
To host your service within mainland China and conduct tests, you must first obtain an ICP license for the domain.
If user traffic to your site is terminated within Mainland China, and your domain does not have an ICP license, your traffic will be blocked by the provider/hosting service. Interestingly, the ICP license specifies a particular provider, whether it's Cloudflare or Alibaba Cloud. Therefore, if you obtained an ICP license for Cloudflare and hosted your site with them, it will not be possible to 'seamlessly' migrate to Alibaba Cloud afterward. You will need to add another hosting service to this license.
By obtaining an ICP license for the domain, we were able to come up with and implement specific technical ideas and solutions.
Testing solutions
But before we directly start creating staging options, adjusting settings, optimizing site performance and speed, we need to choose a testing tool to see which of our actions improve or undermine the site's performance.
Our testing tool had to meet two main requirements:
- it must be able to run tests from China,
- it must support browser tests.
This is how we found ! They have excellent testing point coverage worldwide. In China, this tool allows tests to be run from a countless number of provinces, each with several different providers + the ability to perform Backbone-tests (something like a virtual machine in a data center) and Lastmile-tests (as close as possible to user conditions, aka workstation). The latter type of test is more expensive.
Having signed an annual contract (which is the minimum), we began exploring the tool. To be honest, we were pleasantly surprised by its functionality. You can run:
- DNS tests,
- Web tests (browser, simple GET/POST, mobile client emulation, etc.),
- Transactional checks (e.g., login),
- API tests,
- Ping, traceroute, NTP, etc.
It’s impossible to list everything. And most importantly, each test can be quite well customized by adding a batch of headers and other parameters. In the end, you get a huge amount of information that fully describes your test. Speaking of what is most interesting to us (browser tests), the results include:
- Connect, Wait, Load, SSL, DNS time,
- TTFB, TTLB, Document complete, Render time, DOM load,
- Response (something akin to Time To First Byte), Webpage Response (something akin to Time To Last Byte),
- Any percentiles, Average, Median time
- and so on.
Accordingly, all these metrics help to see changes clearly and understand whether things have improved. We primarily looked at Response, Webpage Response, Median, 75 and 95 Percentiles.
An important question that lingered from the very beginning: can we trust Catchpoint?? Отражает ли этот инструмент реальную скорость загрузки сайта в Китае из разных городов или же это просто какой-то тест в вакууме, не имеющий ничего общего с real user experience?
This is a significant problem because being in Russia, it's nearly impossible to accurately know how a site from China works. Using a socks-proxy through a virtual machine, the website loading time stretches over a couple of minutes, which is unacceptable for testing. Therefore, the only manual testing option remains using curl and simple GET requests from the console with time measurement. This helps because this test accurately reflects the speed of the network solution, and if there are also browser tests, it’s even better.
Later, we went to China ourselves and confirmed that Catchpoint can be trusted, as it accurately reflects the real performance metrics.
Cloudflare China Network
Since we successfully use Cloudflare for the main domain semrush.com, we decided to immediately try their feature called . This option is enabled only for Enterprise websites upon a separate request and for an additional fee. It is also only available for sites that have the corresponding ICP license, where Cloudflare is specified as the provider. Once enabled, the site gains access to Cloudflare's 'Chinese CDN'—traffic from Chinese regions is directed to the nearest CF PoP (Points of Presence), and then it is delivered to the origin through its networks or those of partners/providers.
The scheme for this testing setup is presented below.
For us, this is a great option. It turns out that the second domain will also be behind CF, which does not increase the number of solutions used in the company and does not complicate the infrastructure significantly.
We launched browser tests, and here are the results:
Red diamonds represent test failures. Failures at the bottom are DNS errors (resolve timeout). Failures at the top are timeouts.
Uptime: 86.6
Median: 18s
75 Percentile: 29.3s
95 Percentile: 60s
The median, after removing the loading reCaptcha (a Google service blocked in China), decreased from 28 to 18 seconds. However, these are still terrible metrics, considering that a similar test for semrush.com (from the USA) gave less than 10 seconds for 95% of users (from the USA) on the same page (static + dynamic).
You can enter each test and take a look Waterfall and other more detailed parameters. We started investigating the causes of errors, and while timeouts are somewhat understandable: the internet in China 'sometimes connects and sometimes disconnects', which leads to unstable and varying connection speeds and resource loading from abroad, DNS errors surprised us greatly. We found that PoP Cloudflare does indeed have points of presence in China, the site address resolves to a single anycast IP, but the DNS servers used are American, causing DNS queries to be routed across the border, which is why they sometimes fail.
After clarifying this issue with CF, it turned out that they do not have their own DNS servers in China, and when they will have them is still unknown.
Therefore, we decided to test only Cloudflare's DNS and changed the operation mode of Cloudflare for our site to 'DNS Only’. This is a mode where Cloudflare does not proxy traffic, and thus does not provide DDoS protection, CDN, and other features, and operates as a regular DNS server.
This setup is schematically presented in the following figure. The figure takes into account the acquired knowledge that Cloudflare's DNS servers are behind a firewall.
In Catchpoint, we launched simple GET tests (non-browser based) which showed a lot of failures. The reason for these was the same DNS errors.
We started debugging these errors with dig and discovered that on the first request the address resolves correctly, but on repeated requests we get each time SERVFAIL and not found. Why is that?
root@iZwz97n2wgbp61qucbfrjsZ:~# host semrushchina.cn
semrushchina.cn has address 220.170.186.192
Host semrushchina.cn not found: 2(SERVFAIL)
root@iZwz97n2wgbp61qucbfrjsZ:~# host semrushchina.cn
semrushchina.cn has address 220.170.186.192
Host semrushchina.cn not found: 2(SERVFAIL)
root@iZwz97n2wgbp61qucbfrjsZ:~# host semrushchina.cn
semrushchina.cn has address 220.170.186.192
Host semrushchina.cn not found: 2(SERVFAIL)
root@iZwz97n2wgbp61qucbfrjsZ:~# host semrushchina.cn
semrushchina.cn has address 220.170.186.192
Host semrushchina.cn not found: 2(SERVFAIL)When querying Cloudflare's NS servers directly, there are no such errors:
root@iZwz97n2wgbp61qucbfrjsZ:~# for i in `seq 1 2`; do host semrushchina.cn ray.ns.cloudflare.com.; done
Using domain server:
Name: ray.ns.cloudflare.com.
Address: 173.245.59.138#53
Aliases:
semrushchina.cn has address 220.170.186.192
semrushchina.cn has address 220.170.186.192
Using domain server:
Name: ray.ns.cloudflare.com.
Address: 173.245.59.138#53
Aliases:
semrushchina.cn has address 220.170.186.192
semrushchina.cn has address 220.170.186.192This means the problem lies with the 'local' DNS server or the provider's server.
Further investigation revealed that SERVFAIL we receive the resolution for AAAA-records.
It turned out that when querying Cloudflare AAAA-the record that does not exist in the domain, Cloudflare responded A-with a response that is an error and does not comply with the RFC. This was not liked by the local resolver (x.x.x.x) and it responded SERVFAIL. The behavior is clearly visible in the log below:
root@iZwz97n2wgbp61qucbfrjsZ:~# dig -t AAAA semrushchina.cn @x.x.x.x
; <> DiG 9.10.3-P4-Ubuntu <> -t AAAA semrushchina.cn @x.x.x.x
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 55467
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;semrushchina.cn. IN AAAA
;; Query time: 334 msec
;; SERVER: x.x.x.x#53(x.x.x.x)
;; WHEN: Tue Aug 14 23:38:50 CST 2018
;; MSG SIZE rcvd: 44
root@iZwz97n2wgbp61qucbfrjsZ:~# dig -t AAAA semrushchina.cn @dana.ns.cloudflare.com.
; <> DiG 9.10.3-P4-Ubuntu <> -t AAAA semrushchina.cn @dana.ns.cloudflare.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63944
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;semrushchina.cn. IN AAAA
;; ANSWER SECTION:
semrushchina.cn. 300 IN A 220.170.186.192
;; Query time: 185 msec
;; SERVER: 173.245.58.105#53(173.245.58.105)
;; WHEN: Tue Aug 14 23:43:03 CST 2018
;; MSG SIZE rcvd: 60
We submitted a bug report to Cloudflare, and they fixed it after some time. It turned out to be interesting: currently, there is still no IPv6 support in China, so Cloudflare could not provide its IPv6 address in response to the request AAAA-records. In the end, it was resolved so that for China, Cloudflare began to respond NODATA to such requests.
Thus, DNS errors in Catchpoint tests significantly decreased, but not entirely. Timeouts were also still present:
And we began to search for another solution.
In the next part, I will tell you how we tested the Chinese cloud Alibaba Cloud, how we were able to quickly create PoC (Proof of Concept) solutions with a bit of Nginx 'magic', and how we developed Multi-Cloud solutions, one of which ultimately greatly helped to speed up the service from China.
Stay tuned!
Next parts
Source: habr.com
