Mapindiro atakaita kuburikidza neGreat Firewall yeChina (Chikamu 2)

Nhai!

Nikita anewe zvakare, injinjiniya kubva kukambani SEMrush. Uye nechinyorwa ichi ndinoenderera mberi nenyaya yekuti takauya sei nemhinduro yekugadzirisa Chinese Firewall yebasa redu semrush.com.

В yapfuura chikamu Ndati:

• matambudziko api anomuka mushure mekunge sarudzo yaitwa "Tinofanirwa kuita kuti basa redu rishande muChina"
• Ndeapi matambudziko ane iyo China Internet ine?
• nei uchida rezinesi reICP?
• sei uye nei takasarudza kuyedza matestbeds edu neCatchpoint
• chii chaive mhedzisiro yemhinduro yedu yekutanga yakavakirwa Cloudflare China Network
• Mawaniro atakaita bug muCloudflare DNS

Ichi chikamu ndicho chinonyanya kufadza, mumaonero angu, nekuti chinotarisa pane chaiwo tekinoroji mashandisirwo ekuita. Uye isu tichatanga, kana kuti toenderera mberi, ne Alibaba Cloud.

Alibaba Cloud

Alibaba Cloud ibasa rakakura rakakura rekupa, iro rine ese masevhisi anoitendera kuti iite yakatendeseka pachayo mupi wegore. Zvakanaka kuti vane mukana wekunyoresa kune vashandisi vekune dzimwe nyika, uye kuti yakawanda yesaiti inoshandurirwa muchiChirungu (kuChina ichi chinhu chepamusoro). Mune iri gore, unogona kushanda nematunhu mazhinji epasi, mainland China, pamwe neOceanic Asia (Hong Kong, Taiwan, nezvimwewo).

IPSEC

Takatanga ne geography. Sezvo saiti yedu yekuyedza yaive paGoogle Cloud, taifanira "kubatanidza" Alibaba Cloud neGCP, saka takavhura rondedzero yenzvimbo umo Google iripo. Panguva iyoyo vanga vasati vawana yavo data center muHong Kong.
Nzvimbo yepedyo yakazova Asia-kumabvazuva1 (Taiwan). Ali yakazova dunhu repedyo renyika yeChina kuenda kuTaiwan cn-shenzhen (Shenzhen).

Nekubatsirwa kwe terraform yakatsanangura uye yakasimudza zvese zvivakwa muGCP neAli. Mugero we100 Mbit/s pakati pemakore wakakwira zvakapotsa ipapo. Kudivi reShenzhen neTaiwan, proxying chaiwo michina yakasimudzwa. MuShenzhen, traffic yevashandisi inomiswa, proxied kuburikidza mugero kuenda kuTaiwan, uye kubva ipapo inoenda yakananga kune yekunze IP yebasa redu mu. us-kumabvazuva (USA East Coast). Ping pakati pemashini chaiwo kuburikidza netunnel 24ms, izvo zvisina kunyanya kuipa.

Panguva imwe cheteyo, takaisa nzvimbo yekuedza mukati Alibaba Cloud DNS. Mushure mekupa nzvimbo kuNS Ali, nguva yekugadzirisa yakadzikira kubva pa470 ms kuenda 50 ms. Izvi zvisati zvaitika, zone yaivewo paCloudlfare.

Parallel to the tunnel to Asia-kumabvazuva1 akasimudza imwe mugero kubva Shenzhen zvakananga kuti us-kumabvazuva4. Ikoko vakagadzira mamwe maproxy chaiwo michina uye vakatanga kuyedza mhinduro dzese, routing test traffic vachishandisa Cookies kana DNS. Bhenji rekuyedza rinotsanangurwa schematically mumufananidzo unotevera:

Latency yetunnels yakave seizvi:
Ali cn-shenzhen <—> GCP asia-east1 — 24ms
Ali cn-shenzhen <—> GCP us-east4 — 200ms

Catchpoint browser bvunzo dzakashuma kuvandudzwa kwakanaka.

Enzanisa mhinduro dzebvunzo dzemhinduro mbiri:

chisarudzo
Uptime
Median
75 Percentile
95 Percentile

Cloudflare
86.6
18s
30s
60s

IPsec
99.79
18s
21s
30s

Iyi idata kubva kune mhinduro inoshandisa IPSEC tunnel kuburikidza Asia-kumabvazuva1. Kuburikidza nesu-kumabvazuva4 mhedzisiro yaive yakaipisisa, uye pakanga paine zvikanganiso zvakawanda, saka ini handizopa mhedzisiro.

Zvichienderana nemhedzisiro yebvunzo iyi yenzira mbiri, imwe yacho inogumiswa munzvimbo iri padyo neChina, uye imwe yacho kunzvimbo yekupedzisira, zvakava pachena kuti zvakakosha "kubuda" kubva pasi peChinese firewall nekukurumidza se zvinogoneka, uye wobva washandisa ma network anokurumidza (CDN providers, Cloud providers, etc.). Hapana chikonzero chekuyedza kupinda nepafirewall uye usvike kwauri kuenda mune imwe yakawa swoop. Iyi haisiyo nzira inokurumidza.

Kazhinji, mhedzisiro haina kushata, zvisinei, semrush.com ine median ye8.8s, uye 75 Percentile 9.4s (pamwe bvunzo).
Uye ndisati ndaenderera mberi, ndinoda kuita pfupiso yerwiyo.

Zvinyorwa zvekunyengera

Mushure mokunge mushandisi apinda panzvimbo www.semrushchina.cn, iyo inogadzirisa kuburikidza ne "nekukurumidza" Chinese DNS maseva, chikumbiro cheHTTP chinoenda kuburikidza nemhinduro yedu yekukurumidza. Mhinduro inodzoserwa nenzira imwechete, asi dura rinotsanangurwa mune ese maJS zvinyorwa, HTML mapeji uye zvimwe zvinhu zvewebhu peji. semrush.com kune zvimwe zviwanikwa zvinofanirwa kutakurwa kana peji raitwa. Ndiko kuti, mutengi anogadzirisa iyo "main" A-rekodhi www.semrushchina.cn uye inopinda mumugero wekukurumidza, inokurumidza kugamuchira mhinduro - peji reHTML rinoti:

• dhawunirodha akadaro uye akadaro js kubva sso.semrush.com,
• Tora iwo CSS mafaera kubva cdn.semrush.com,
• uye zvakare tora mimwe mifananidzo kubva dab.semrush.com
• uye zvichingodaro.

Bhurawuza rinotanga kuenda kuInternet "yekunze" yezviwanikwa izvi, nguva yega yega ichipfuura nepafirewall inodya nguva yekupindura.

Asi bvunzo yapfuura inoratidza mhedzisiro kana pasina zviwanikwa pane peji semrush.comchete semrushchina.cn, uye *.semrushchina.cn inogadzirisa kukero yemuchina chaiwo muShenzhen kuitira kuti ugozopinda mugero.

Nenzira iyi chete, nekusundidzira zvese zvinogoneka traffic kusvika pakakwirira kuburikidza nemhinduro yako yekukurumidza kupfuura iyo Chinese firewall, unogona kuwana zvinogamuchirika kumhanya uye webhusaiti kuwanikwa zviratidzo, pamwe nekutendeka mhedzisiro yemhinduro bvunzo.
Isu takaita izvi tisina kana kodhi imwe gadziriso padivi rechikwata chigadzirwa.

Subfilter

Mhinduro yacho yakazvarwa nguva pfupi mushure mekunge dambudziko iri rabuda. Taida PoC (Uchapupu hwePfungwa) kuti yedu firewall yekupinda mhinduro inoshanda chaizvo. Kuti uite izvi, iwe unofanirwa kuputira yese saiti traffic mune iyi mhinduro zvakanyanya sezvinobvira. Uye takanyorera subfilter mu nginx.

Subfilter imodhi iri nyore mu nginx iyo inokutendera kuti uchinje mutsara mumwe mumutumbi wekupindura kune mumwe mutsara. Saka takachinja zvese zvinoitika semrush.com pamusoro semrushchina.cn mumhinduro dzose.

Uye ... hazvina kushanda nekuti takagamuchira zvakamanikidzwa zvemukati kubva kumashure, saka subfilter haina kuwana mutsara unodiwa. Ini ndaifanira kuwedzera imwe sevha yemunharaunda kune nginx, iyo yakadzora mhinduro uye ndokuipfuudza kune inotevera yemunharaunda sevha, iyo yakanga yatove yakabatikana kutsiva tambo, kuimanikidza, uye kuitumira kune inotevera proxy server mumaketani.

Somugumisiro, mutengi aizowana kupi .semrush.com, akagamuchira .imrushchina.cn uye nokuteerera akafamba nomuchisarudzo chedu.

Nekudaro, hazvina kukwana kungochinja iyo domain neimwe nzira, nekuti iyo yekumashure ichiri kutarisira semrush.com mune zvinotevera zvikumbiro kubva kumutengi. Saizvozvo, pane imwechete sevha panoitwa imwe-nzira yekutsiva, tichishandisa yakapusa kutaura tinowana subdomain kubva kuchikumbiro, uye tobva taita. proxy_pass with variable $host, inoratidzwa mukati $subdomain.semrush.com. Zvingaita sezvinovhiringa, asi zvinoshanda. Uye inoshanda nemazvo. Kune ega ega madomasi anoda akasiyana mantiki, ingo gadzira yako sevha mabhuroko uye gadzira yakaparadzana gadziriso. Pazasi akapfupikiswa nginx configs yekujekesa uye kuratidzwa kwechirongwa ichi.

Iyo inotevera config inoshanda zvese zvikumbiro kubva kuChina kuenda .semrushchina.cn:

    listen 80;

    server_name ~^(?<subdomain>[w-]+).semrushchina.cn$;

    sub_filter '.semrush.com' '.semrushchina.cn';
    sub_filter_last_modified on;
    sub_filter_once off;
    sub_filter_types *;

    gzip on;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;

    location / {
        proxy_pass http://127.0.0.1:8083;
        proxy_set_header Accept-Encoding "";
        proxy_set_header Host $subdomain.semrush.com;
        proxy_set_header X-Accept-Encoding $http_accept_encoding;
    }
}

Iyi config proxies ku localhost kuchiteshi 83, uye inotevera config yakamirira ipapo:

    listen 127.0.0.1:8083;

    server_name *.semrush.com;

    location / {
        resolver 8.8.8.8 ipv6=off;
        gunzip on;
        proxy_pass https://$host;
        proxy_set_header Accept-Encoding gzip;
    }
}

Ndinodzokorora, aya akachekwa configs.

Saizvozvo. Zvinogona kutaridzika zvakaoma, asi zviri mumashoko. Muchokwadi, zvese zviri nyore kupfuura steamed turnips :)

Kupera kwekuderera

Kwenguva yakati takafara nekuti ngano yekudonha IPSEC tunnel haina kusimbiswa. Asi ipapo migero yakatanga kudonha. Kanoverengeka pazuva kwemaminitsi mashoma. Zvishoma, asi izvo hazvina kuenderana nesu. Sezvo ese matani akamiswa parutivi rweAli pane imwechete router, takasarudza kuti pamwe iri raive dambudziko redunhu uye taifanira kusimudza dunhu rekuchengetedza.

Vakainhonga. Matanho akatanga kutadza panguva dzakasiyana, asi failover yakashanda zvakanaka kwatiri padanho rekumusoro mu nginx. Asi zvino migero yakatanga kudonha panguva imwe chete :) Uye 502 uye 504 yakatanga kudonha zvakare. Uptime yakatanga kuderera, saka takatanga kushanda pasarudzo ne. Alibaba CEN (Cloud Enterprise Network).

cen

cen - Uku ndiko kubatana kweVPC mbiri kubva kumatunhu akasiyana mukati meAlibaba Cloud, ndiko kuti, unogona kubatanidza akavanzika network echero matunhu mukati megore nemumwe. Uye zvinonyanya kukosha: iyi chiteshi ine yakanyatsoomesesa SLA. Iyo yakagadzikana kwazvo zvese mukumhanya uye uptime. Asi hazvisi nyore zvakadaro:

• ZVAKAoma kuwana kana usiri vagari vekuChina kana sangano repamutemo,
• Iwe unofanirwa kubhadhara kune yega yega megabit yechannel huwandu.

Kuva nemukana wekubatanidza Mainland China и mhiri kwemakungwa, takagadzira CEN pakati pematunhu maviri eAli: cn-shenzhen и us-kumabvazuva-1 (nzvimbo yepedyo kwatiri-kumabvazuva4). Muna Ali us-kumabvazuva-1 yakasimudza mumwe muchina chaiwo kuti pave nemumwe tauka.

Zvakaitika seizvi:

Zviyedzo zvebrowser zviri pazasi:

chisarudzo
Uptime
Median
75 Percentile
95 Percentile

Cloudflare
86.6
18s
30s
60s

IPsec
99.79
18s
21s
30s

cen
99.75
16s
21s
27s

Kuita kuri nani zvishoma pane IPSEC. Asi kuburikidza neIPSEC unogona kudhawunirodha nekumhanya kwe100 Mbit/s, uye kuburikidza neCEN chete nekumhanya kwe5 Mbit/s uye nezvimwe.

Inonzwika kunge hybrid, handiti? Sanganisa IPSEC kumhanya uye CEN kugadzikana.

Izvi ndizvo zvatakaita, tichibvumira traffic kuburikidza nezvose IPSEC uye CEN muchiitiko chekutadza kweIPSEC tunnel. Uptime yave yakakwira zvakanyanya, asi iyo saiti yekurodha yekumhanyisa ichiri kusiya zvakanyanya kudiwa. Ipapo ndakadhirowa ese masekete atakange tatoshandisa uye kuyedza, uye ndikafunga kuedza kuwedzera zvishoma GCP kudunhu iri, iro. chivharo.

chivharo

chivharo - ichi chi Global Load Balancer (kana Google Cloud Load Balancer). Iine zvakakosha zvakakosha kwatiri: mumamiriro ezvinhu eCDN ane anycast IP, iyo inokubvumira kuti uende kumigwagwa kune data data iri pedyo nemutengi, kuitira kuti motokari inokurumidza kupinda muGoogle inokurumidza network uye zvishoma inopinda ne "regular" Internet.

Pasina kufunga kaviri, takasimudza HTTP/HTTPS LB Isu takaisa edu chaiwo machina ane subfilter muGCP uye sebackend.

Kwaive nehurongwa hwakawanda:

• Kushandisa Cloudflare China Network, asi panguva ino Origin inofanira kutsanangura pasi rose IP GLB.
• Kugumisa vatengi pa cn-shenzhen, uye kubva ipapo proxy traffic yakananga ku chivharo.
• Enda zvakananga kubva kuChina kuenda chivharo.
• Kugumisa vatengi pa cn-shenzhen, kubva ipapo proxy kuenda Asia-kumabvazuva1 kuburikidza neIPSEC (in us-kumabvazuva4 kuburikidza neCEN), kubva ipapo enda kuGLB (wakadzikama, pachava nemufananidzo uye tsananguro pazasi)

Takaedza zvese izvi sarudzo uye akati wandei mamwe mahybrids:

• Cloudflare + GLB

Ichi chirongwa hachina kuenderana nesu nekuda kwekukwira uye DNS kukanganisa. Asi bvunzo yakaitwa bug isati yagadziriswa padivi reCF, pamwe zviri nani izvozvi (zvisinei, izvi hazvisanganisire HTTP nguva yekubuda).

• Ali + GLB

Ichi chirongwa zvakare hachina kuenderana nesu maererano neuptime, sezvo GLB yaiwanzodonha kubva kumusoro nekuda kwekusakwanisa kwekubatanidza panguva inogamuchirwa kana nguva yekubuda, nekuti kune server mukati meChina, iyo GLB kero inoramba iri kunze, uye nekudaro kuseri kweiyo. Chinese firewall. Mashiripiti haana kuitika.

• GLB chete

Sarudzo yakafanana neyakapfuura, chete haina kushandisa maseva muChina pachayo: traffic yakananga kuGLB (iyo DNS marekodhi akachinjwa). Saizvozvo, mhedzisiro yacho yanga isingagutsi, sezvo vatengi veChinese vakajairwa vachishandisa masevhisi evanopa Internet vakajairwa vane mamiriro akaipisisa nekupfuura firewall kupfuura Ali Cloud.

• Shenzhen -> (CEN/IPSEC) -> Proxy -> GLB

Pano takasarudza kushandisa akanakisa emhinduro dzese:

• kugadzikana uye yakavimbiswa SLA kubva kuCEN
• kumhanya kukuru kubva kuIPSEC
• Netiweki yeGoogle "yekukurumidza" uye chero kukanda kwayo.

Iyo hurongwa inotaridzika seizvi: mushandisi traffic inomiswa pamushini chaiwo mukati ch-shenzhen. Nginx kumusoro kwenzizi kwakagadziridzwa ipapo, mamwe acho anonongedza kune yakavanzika IP maseva ari kune imwe mugumo weIPSEC mugero, uye mamwe ekumusoro anonongedzera kune yakavanzika kero yemaseva kune rimwe divi reCEN. IPSEC yakagadzirirwa kudunhu Asia-kumabvazuva1 muGCP (ndiyo yaive dunhu raive pedyo neChina panguva iyo mhinduro yagadzirwa. GCP ikozvino yavawo nevekuHong Kong). CEN - kunzvimbo us-kumabvazuva1 muAli Cloud.

Ipapo traffic kubva kumativi ese maviri yakanangana anycast IP GLB, ndiko kuti, kusvika padyo nenzvimbo yekuvapo kweGoogle, uye yakapfuura nemanetiweki ayo kudunhu us-kumabvazuva4 muGCP, maive nemichina yekutsiva chaiyo (ine subfilter mu nginx).

Iyi hybrid mhinduro, sezvataitarisira, yakatora mukana wemabhenefiti ega ega tekinoroji. Kazhinji, traffic inoenda nekukurumidza IPSEC, asi kana matambudziko atanga, isu nekukurumidza uye kwemaminetsi mashoma tinokanda maseva aya kubva kumusoro uye kutumira traffic chete kuburikidza neCEN kusvika mugero wadzikama.

Nekushandisa iyo 4th mhinduro kubva pane rondedzero iri pamusoro, isu takawana zvataida uye izvo bhizinesi raida kwatiri panguva iyoyo nenguva.

Browser bvunzo mhinduro yemhinduro nyowani zvichienzaniswa neyapfuura:

chisarudzo
Uptime
Median
75 Percentile
95 Percentile

Cloudflare
86.6
18s
30s
60s

IPsec
99.79
18s
21s
30s

cen
99.75
16s
21s
27s

CEN/IPsec + GLB
99.79
13s
16s
25s

CDN

Zvese zvakanaka mumhinduro yatakaita, asi hapana CDN inogona kumhanyisa traffic padunhu uye kunyangwe dhorobha. Mune dzidziso, izvi zvinofanirwa kukurumidzira saiti yevashandisi vekupedzisira nekushandisa nzira dzekutaurirana dzinokurumidza dzemupi weCDN. Uye taifunga nezvazvo nguva dzose. Uye ikozvino, nguva yasvika yeinotevera iteration yeprojekiti: kutsvaga nekuyedza CDN vanopa muChina.

Uye ini ndichakuudza nezve izvi mune inotevera, yekupedzisira chikamu :)

Source: www.habr.com