Hello!
This is Nikita again — a systems engineer from the company SEMrush. With this article, I continue the story about how we conceived a workaround for the Chinese Firewall for our service semrush.com.
In I discussed:
- what problems arise after the decision is made that 'we need to ensure our service works in China'
- what issues exist with the Chinese internet
- why an ICP license is necessary
- how and why we decided to test our test environments using Catchpoint
- what results our first solution, based on Cloudflare China Network, yielded
- how we found a bug in Cloudflare's DNS
This part is the most interesting, in my opinion, as it focuses on specific technical implementations of staging. So let's start, or rather continue, with Alibaba Cloud.
Alibaba Cloud
Alibaba Cloud — a fairly large cloud provider that has all the services allowing it to genuinely call itself a cloud provider. It's good that foreign users can register, and that a significant part of the site is translated into English (a luxury in China). In this cloud, you can operate in many regions of the world, continental China, as well as Oceania (Hong Kong, Taiwan, etc.).
IPSEC
We started with geography. Since our test site was hosted in Google Cloud, we needed to 'connect' Alibaba Cloud with GCP, so we opened the list of locations where Google is present. At that time, they still did not have their own data center in Hong Kong.
The closest region turned out to be asia-east1 (Taiwan). For Ali, the closest region of continental China to Taiwan turned out to be cn-shenzhen (Shenzhen).
Using terraform We described and set up the entire infrastructure in GCP and Ali. The 100 Mbps tunnel between the clouds was established almost instantly. On the Shenzhen and Taiwan sides, we set up proxy virtual machines. In Shenzhen, user traffic is terminated and proxied through the tunnel to Taiwan, and from there it goes directly to the external IP of our service in us-east (Eastern US). The ping between the virtual machines via the tunnel is 24ms, which is not so bad.
Simultaneously, we set up a testing zone in Alibaba Cloud DNS. After delegating the zone to Ali's NS, the resolution time decreased from 470 ms to 50 ms. Previously, the zone was also on Cloudflare.
Alongside the tunnel to asia-east1 we established another tunnel from Shenzhen directly to us-east4They created more proxy virtual machines and began measuring both solutions by routing test traffic using Cookies or DNS. The schematic of the test setup is described in the following illustration:
The latency for the tunnels was as follows:
Ali cn-shenzhen GCP asia-east1 — 24ms
Ali cn-shenzhen GCP us-east4 — 200ms
Browser tests conducted by Catchpoint reported a significant improvement in performance metrics.
Compare the test results for the two solutions:
Solution
Uptime
Median
75th Percentile
95th Percentile
Cloudflare
86.6
18s
30s
60s
IPsec
99.79
18s
21s
30s
These are the results of the solution using the IPSEC tunnel through asia-east1. Results through us-east4 were worse, and there were more errors, so I won't provide those results.
From the results of this test of the two tunnels, one of which terminates in the nearest region to China and the other at the final destination, it became clear that it is essential to 'emerge' from under the Chinese firewall as quickly as possible, and then use fast networks (CDN providers, cloud providers, etc.). One shouldn't try to go through the firewall directly to reach the destination. That is not the fastest route.
Overall, the results are decent; however, semrush.com has a median of 8.8s, and the 75th percentile is 9.4s (in the same test).
And before moving on, I would like to make a small lyrical digression.
A philosophical digression
After a user accesses the website www.semrushchina.cn, which resolves through 'fast' Chinese DNS servers, the HTTP request goes through our fast solution. The response returns along the same path, but in all JS scripts, HTML pages, and other elements of the web page, the domain is specified as semrush.com for additional resources needed to be loaded during page rendering. That is, the client resolves the 'main' A record www.semrushchina.cn and enters the fast tunnel, quickly receiving the response — the HTML page, which specifies that:
- download a specific JS file from sso.semrush.com,
- fetch CSS files from cdn.semrush.com,
- and also get images from dab.semrush.com.
- and so on.
The browser then starts going into the 'external' internet for these resources, passing each time through the time-consuming response-consuming firewall.
But in the previous test, results were presented when there are no resources on the page semrush.com, only semrushchina.cn, while *.semrushchina.cn resolves to the address of a virtual machine in Shenzhen, to then enter the tunnel.
Only by maximizing traffic through our solution for bypassing the Chinese firewall can we achieve acceptable speeds and site availability metrics, as well as honest results from the solutions tests.
We accomplished this without a single code modification on the product teams' side.
Subfilter
The solution was born almost immediately after the problem arose. We needed PoC (Proof of Concept) that our firewall bypass solutions actually work well. For this, it's essential to funnel all website traffic through this solution. And we implemented in nginx.
Subfilter — this is a fairly simple module in nginx that allows replacing one line in a response body with another. So we replaced all occurrences semrush.com to semrushchina.cn in all responses.
And... it didn't work because we were receiving compressed content from the backends, so the subfilter couldn't find the required line. We had to add another local server in nginx, which decompressed the response and sent it to the next local server that handled the string replacement, compression, and delivery to the next proxy server in the chain.
As a result, where the client would have received .semrush.com, they received .semrushchina.cn and obediently passed through our solution.
However, it's not enough to just change the domain in one direction, as the backends still expect semrush.com in subsequent requests from the client. Accordingly, on the same server where the replacement is made in one direction, we use a simple regular expression to extract the subdomain from the request and then work proxy_pass with the variable $host, set to $subdomain.semrush.com. It may seem confusing, but it works. And it works well. For specific domains that require different logic, individual server blocks are created with separate configurations. Below are abbreviated nginx configurations for clarity and to demonstrate this scheme.
The next config handles all requests from China to .semrushchina.cn:
listen 80;
server_name ~^(?[w-]+).semrushchina.cn$;
sub_filter '.semrush.com' '.semrushchina.cn';
sub_filter_last_modified on;
sub_filter_once off;
sub_filter_types *;
gzip on;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
location / {
proxy_pass http://127.0.0.1:8083;
proxy_set_header Accept-Encoding "";
proxy_set_header Host $subdomain.semrush.com;
proxy_set_header X-Accept-Encoding $http_accept_encoding;
}
}This config proxies to localhost port 83, where the next config is waiting:
listen 127.0.0.1:8083;
server_name *.semrush.com;
location / {
resolver 8.8.8.8 ipv6=off;
gunzip on;
proxy_pass https://$host;
proxy_set_header Accept-Encoding gzip;
}
}To reiterate, these are trimmed configurations.
Something like this. It may look complicated, but it’s easy in practice 🙂
End of the lyrical digression
For a while, we were happy because the myth about falling IPSEC tunnels was not confirmed. But then the tunnels started dropping. Several times a day for a few minutes. A little, but it was unacceptable for us. Since both tunnels terminated on Ali's side on one router, we assumed that this might be a regional issue and we needed to bring up a backup region.
We did. The tunnels began to drop at different times, but we had a perfect failover working at the upstream level in nginx. But then the tunnels started to drop roughly simultaneously 🙂 And once again, we had 502 and 504 errors. Uptime began to deteriorate, so we started working on the option with Alibaba CEN (Cloud Enterprise Network).
CEN
CEN — is the connectivity of two VPCs from different regions within Alibaba Cloud, meaning you can connect private networks of any regions within the cloud to each other. And most importantly, this channel has quite strict SLArequirements. It is very stable in terms of both speed and uptime. But it’s never as simple as it seems:
- it’s VERY difficult to obtain if you are not a Chinese citizen or legal entity,
- you have to pay for every megabit of channel bandwidth.
Having obtained the ability to connect Mainland China and Overseas, we created CEN between two Ali regions: cn-shenzhen and us-east-1 (the closest point to us-east4). In Ali, us-east-1 we raised another virtual machine to have another hop.
It turned out that:
The results of the browser tests are below:
Solution
Uptime
Median
75th Percentile
95th Percentile
Cloudflare
86.6
18s
30s
60s
IPsec
99.79
18s
21s
30s
CEN
99.75
16s
21s
27s
The metrics are slightly better than IPSEC. But with IPSEC, it’s potentially possible to transfer at a speed of 100 Mbps, while with CEN only at a speed of 5 Mbps and at a higher cost.
A hybrid solution seems to be in order, right? To combine the speed of IPSEC and the stability of CEN.
That's how we acted, routing traffic through both IPSEC and CEN in case of an IPSEC tunnel failure. Uptime significantly improved, but the website load speed still left much to be desired. Then I drew all the schemes we had already used and tested, and decided to try adding a bit of GCP to this scheme, namely GLB.
GLB
GLB — this is (or Google Cloud Load Balancer). It has an important advantage for us: in the context of CDN, it has anycast IP, which allows routing traffic to the closest data center to the client, resulting in quicker access to Google's fast network and less travel through the 'regular' internet.
Without further ado, we deployed HTTP/HTTPS LB in GCP and set our virtual machines with subfilter as the backend.
There were several schemes:
- Use Cloudflare China Network, but this time we specified the global IP GLB.
- To terminate clients in cn-shenzhen, and from there proxy the traffic straight to GLB.
- Going directly from China to GLB.
- To terminate clients in cn-shenzhen, then proxying to asia-east1 via IPSEC (to us-east4 via CEN), from there going to GLB (don’t worry, there will be a picture and explanation below)
We tested all these options and a few more hybrid ones:
- Cloudflare + GLB
This scheme did not meet our uptime and DNS error requirements. But the test was conducted before the bug fix on the CF side; it may have improved since then (however, this does not rule out HTTP timeouts).
- Ali + GLB
This scheme also did not meet our uptime standards, as GLB frequently dropped from the upstream due to connection timeouts or inability to connect in a reasonable time, since for servers inside China, the GLB address remains external, thus falling behind China's firewall. No magic happened.
- GLB only
An option similar to the previous one, only it did not use servers inside China: the traffic went straight to GLB (DNS records were changed). Accordingly, the results were unsatisfactory, as regular Chinese clients using regular internet providers faced much worse firewall traversal than those using Ali Cloud.
- Shenzhen -> (CEN/IPSEC) -> Proxy -> GLB
Here we decided to use the best of all solutions:
- stability and guaranteed SLA from CEN
- high speed from IPSEC
- Google's 'fast' network and its anycast.
The scheme looks roughly like this: user traffic is terminated on a virtual machine in ch-shenzhenConfigured are upstreams for nginx, some of which reference private IP servers located on the other end of the IPSEC tunnel, while others refer to private server addresses on the other side of the CEN. IPSEC was set up for the region asia-east1 in GCP (it was the closest region to China at the time the solution was created. Now GCP also has a presence in Hong Kong). CEN is for the region us-east1 in Ali Cloud.
Subsequently, traffic from both ends was directed to anycast IP GLB, meaning to the nearest Google point of presence, which then routed it through its networks to the region us-east4 in GCP, where virtual machines with substitutes (using subfilter in nginx) were hosted.
This hybrid solution, as we expected, allowed leveraging the advantages of each technology. Overall, traffic flows through fast IPSEC, but if any issues arise, we can quickly and temporarily remove these servers from upstreams and redirect traffic only through CEN until the tunnel stabilizes.
By implementing the fourth solution from the list above, we achieved what we aimed for, which was required by the business at that time.
Results of browser tests for the new solution compared to previous ones:
Solution
Uptime
Median
75th Percentile
95th Percentile
Cloudflare
86.6
18s
30s
60s
IPsec
99.79
18s
21s
30s
CEN
99.75
16s
21s
27s
CEN/IPsec + GLB
99.79
13s
16s
25s
CDN
In our implemented solution, everything is fine, but there is no CDN that could accelerate traffic at the regional and even city levels. In theory, this should speed up the website operation for end users by utilizing the fast communication channels of the CDN provider. And we have been thinking about this all along. Now, it’s time for the next iteration of the project: searching for and testing CDN providers in China.
And I will tell you about it in the next, final part 🙂
Source: habr.com
