Indaba emayelana nokushoda kwamaphakethe e-DNS avela kusekelo lobuchwepheshe be-Google Cloud

Kusuka ku-Google Blog Editor: Wake wazibuza ukuthi onjiniyela be-Google Cloud Technical Solutions (TSE) bazisingatha kanjani izicelo zakho zosekelo? Onjiniyela Bezobuchwepheshe Be-TSE banesibopho sokuhlonza nokulungisa imithombo yezinkinga ezibikwe ngabasebenzisi. Ezinye zalezi zinkinga zilula, kodwa ngezinye izikhathi uthola ithikithi elidinga ukunakwa konjiniyela abambalwa ngesikhathi esisodwa. Kulesi sihloko, omunye wabasebenzi be-TSE uzositshela ngenkinga eyodwa ekhohlisayo evela kumkhuba wakhe wakamuva - icala lamaphakethe e-DNS alahlekile. Kule ndaba, sizobona ukuthi onjiniyela bakwazi kanjani ukuxazulula isimo, nokuthi yiziphi izinto ezintsha abazifundile ngenkathi belungisa iphutha. Sithemba ukuthi le ndaba ayigcini nje ngokukufundisa ngesiphazamisi esijulile, kodwa futhi ikunikeza ukuqonda ngezinqubo ezingena ekugcwaliseni ithikithi lokusekela nge-Google Cloud.

Ukuxazulula izinkinga kuyisayensi kanye nobuciko. Konke kuqala ngokwakha i-hypothesis mayelana nesizathu sokuziphatha okungajwayelekile kwesistimu, emva kwalokho ihlolwe amandla. Kodwa-ke, ngaphambi kokuba sakhe inkolelo-mbono, kufanele siyichaze ngokucacile futhi siyakhe inkinga ngokunembile. Uma umbuzo uzwakala ungacacile kakhulu, kuzodingeka uhlaziye yonke into ngokucophelela; Lona "ubuciko" bokuxazulula izinkinga.

Ngaphansi kwe-Google Cloud, izinqubo ezinjalo ziba nzima nakakhulu, njengoba i-Google Cloud izama konke okusemandleni ayo ukuqinisekisa ubumfihlo babasebenzisi bayo. Ngenxa yalokhu, onjiniyela be-TSE abakwazi ukufinyelela ukuhlela amasistimu akho, noma ikhono lokubuka ukulungiselelwa ngokubanzi njengoba kwenza abasebenzisi. Ngakho-ke, ukuhlola noma yimiphi imibono yethu, thina (onjiniyela) asikwazi ukuguqula uhlelo ngokushesha.

Abanye abasebenzisi bakholelwa ukuthi sizolungisa yonke into efana noomakhenikha kusevisi yemoto, futhi sivele basithumele umazisi womshini obonakalayo, kanti empeleni inqubo yenzeka ngefomethi yengxoxo: ukuqoqa ulwazi, ukwakha nokuqinisekisa (noma ukuphika) imibono ecatshangelwayo, futhi, ekugcineni, izinkinga zesinqumo zisekelwe ekuxhumaneni neklayenti.

Inkinga okukhulunywa ngayo

Namuhla sinendaba enesiphetho esihle. Esinye sezizathu zokuxazululwa ngempumelelo kwecala elihlongozwayo incazelo eningiliziwe nenembile yenkinga. Ngezansi ungabona ikhophi yethikithi lokuqala (elihlelwe ukufihla ulwazi oluyimfihlo):

Lo mlayezo uqukethe ulwazi oluningi oluwusizo kithi:

• I-VM ethile icacisiwe
• Inkinga ngokwayo ibonisiwe - i-DNS ayisebenzi
• Kuboniswa lapho inkinga izibonakalisa khona - i-VM nesiqukathi
• Izinyathelo ezithathwe umsebenzisi ukuze ahlonze inkinga ziyakhonjiswa.

Isicelo sibhaliswe ngokuthi “P1: I-Critical Impact - Isevisi Engasebenziseki ekukhiqizeni”, okusho ukuqapha njalo isimo 24/7 ngokohlelo “Landela Ilanga” (ungafunda kabanzi mayelana okubalulekile kwezicelo zabasebenzisi), ngokudluliswa kwayo isuka eqenjini elilodwa labasekeli bezobuchwepheshe iye kwelinye ngokushintshwa kwendawo ngayinye yesikhathi. Empeleni ngesikhathi inkinga ifika eqenjini lethu eZurich, lase lizungeza imbulunga yonke. Ngalesi sikhathi, umsebenzisi wayesethathe izinyathelo zokunciphisa, kodwa wayesaba ukuphinda isimo ekukhiqizeni, njengoba umsuka wawungakatholakali.

Ngesikhathi ithikithi lifika eZurich, sase sinolwazi olulandelayo esandleni:

• Okuqukethwe /etc/hosts
• Okuqukethwe /etc/resolv.conf
• isiphetho iptables-save
• Ihlanganiswe yiqembu ngrep pcap ifayela

Ngale datha, besilungele ukuqala "uphenyo" nesigaba sokuxazulula inkinga.

Izinyathelo zethu zokuqala

Okokuqala, sihlole izingodo nesimo seseva yemethadatha futhi saqinisekisa ukuthi isebenza ngendlela efanele. Iseva yemethadatha iphendula ekhelini le-IP 169.254.169.254 futhi, phakathi kwezinye izinto, inesibopho sokulawula amagama wesizinda. Siphinde sahlola kabili ukuthi i-firewall isebenza kahle nge-VM futhi ayiwavimbi amaphakethe.

Kwakuwuhlobo oluthile lwenkinga exakile: isheke le-nmap laphikisa umbono wethu oyinhloko mayelana nokulahleka kwamaphakethe e-UDP, ngakho-ke ngokwengqondo saqhamuka nezinye izindlela ezimbalwa nezindlela zokuwahlola:

• Ingabe amaphakethe alahlwa ngokukhetha? => Hlola imithetho ye-iptables
• Ayincane kakhulu? MTU? => Hlola okukhiphayo ip a show
• Ingabe inkinga ithinta kuphela amaphakethe e-UDP noma i-TCP futhi? => Shayela kude dig +tcp
• Ingabe amaphakethe akhiqizwe yi-dig abuyisiwe? => Shayela kude tcpdump
• Ingabe i-libdns isebenza kahle? => Shayela kude strace ukuhlola ukudluliswa kwamaphakethe kuzo zombili izinkomba

Lapha sinquma ukubiza umsebenzisi ukuze axazulule izinkinga bukhoma.

Ngesikhathi socingo siyakwazi ukuhlola izinto ezimbalwa:

• Ngemuva kokuhlolwa okuningana asifaki imithetho ye-iptables ohlwini lwezizathu
• Sihlola izixhumanisi zenethiwekhi namathebula omzila, bese sihlola kabili ukuthi i-MTU ilungile
• Siyakuthola lokho dig +tcp google.com (TCP) isebenza njengoba kufanele, kodwa dig google.com (UDP) ayisebenzi
• Eseshayekile tcpdump isasebenza dig, sithola ukuthi amaphakethe e-UDP ayabuyiselwa
• Siyahamba strace dig google.com futhi sibona indlela yokumba izingcingo ngendlela efanele sendmsg() и recvms(), nokho esesibili siphazanyiswa ukuphela kwesikhathi

Ngeshwa, ukuphela kweshifu kuyafika futhi siphoqeleka ukuthi sikhulise inkinga endaweni yesikhathi esizayo. Isicelo, nokho, sivuse intshisekelo ethimbeni lethu, futhi uzakwethu uphakamisa ukuthi kwakhiwe iphakethe lokuqala le-DNS kusetshenziswa imojula yePython eyi-scrapy.

from scapy.all import *

answer = sr1(IP(dst="169.254.169.254")/UDP(dport=53)/DNS(rd=1,qd=DNSQR(qname="google.com")),verbose=0)
print ("169.254.169.254", answer[DNS].summary())

Lesi siqeshana sakha iphakethe le-DNS bese sithumela isicelo kuseva yemethadatha.

Umsebenzisi uqhuba ikhodi, impendulo ye-DNS ibuyiselwa, futhi isicelo siyayithola, iqinisekisa ukuthi ayikho inkinga ezingeni lenethiwekhi.

Ngemva kolunye "uhambo lokuzungeza umhlaba," isicelo sibuyela eqenjini lethu, futhi ngidlulisela kimi ngokuphelele, ngicabanga ukuthi kuzoba lula kakhulu kumsebenzisi uma isicelo siyeka ukuzungeza indawo ngezindawo.

Okwamanje, umsebenzisi uvuma ngomusa ukunikeza isifinyezo sesithombe sohlelo. Lezi yizindaba ezinhle kakhulu: ikhono lokuhlola uhlelo ngokwami ​​lenza ukuxazulula izinkinga ngokushesha okukhulu, ngoba akusadingeki ngicele umsebenzisi ukuthi asebenzise imiyalo, ngithumele imiphumela futhi ngiyihlaziye, ngingenza konke ngokwami!

Ozakwethu sebeqala ukungifela umona kancane. Ngesikhathi sesidlo sasemini sixoxa ngokuguqulwa, kodwa akekho onombono wokuthi kwenzekani. Ngenhlanhla, umsebenzisi ngokwakhe usevele ethathe izinyathelo zokunciphisa imiphumela futhi akajahile, ngakho sinesikhathi sokuhlukanisa inkinga. Futhi njengoba sinesithombe, singakwazi ukwenza noma yiziphi izivivinyo esizithandayo. Kuhle!

Ukuthatha isinyathelo emuva

Omunye wemibuzo ethandwa kakhulu yenhlolokhono yezikhundla zonjiniyela wezinhlelo uthi: “Kwenzekani uma u-ping www.google.com? Umbuzo muhle, ngoba umuntu omele ukhetho udinga ukuchaza yonke into kusuka kugobolondo kuya esikhaleni somsebenzisi, kuya ku-kernel yesistimu bese kuya kunethiwekhi. Ngiyamamatheka: kwesinye isikhathi imibuzo yenhlolokhono iba usizo empilweni yangempela...

Nginquma ukusebenzisa lo mbuzo wakwa-HR enkingeni yamanje. Uma sikhuluma nje, uma uzama ukuthola igama le-DNS, okulandelayo kuyenzeka:

1. Uhlelo lokusebenza lubiza umtapo wolwazi wesistimu njenge-libdns
2. libdns ihlola ukucushwa kwesistimu lapho iseva ye-DNS okufanele ixhumane nayo (kumdwebo lokhu ithi 169.254.169.254, iseva yemethadatha)
3. I-libdns isebenzisa amakholi wesistimu ukuze idale isokhethi ye-UDP (SOKET_DGRAM) futhi ithumele amaphakethe e-UDP anombuzo we-DNS kuzo zombili izinkomba.
4. Ngesixhumi esibonakalayo se-sysctl ungamisa isitaki se-UDP ezingeni le-kernel
5. I-kernel isebenzisana nehadiwe ukuze idlulise amaphakethe ngenethiwekhi ngesixhumi esibonakalayo senethiwekhi
6. I-hypervisor ibamba bese idlulisela iphakethe kuseva yemethadatha lapho uthintana nayo
7. Iseva yemethadatha, ngomlingo wayo, inquma igama le-DNS bese ibuyisela impendulo isebenzisa indlela efanayo

Ake ngikukhumbuze ukuthi yimiphi imibono esesivele siyicabangile:

I-hypothesis: Imitapo yolwazi ephukile

• Isivivinyo 1: sebenzisa i-strace ohlelweni, hlola ukuthi i-dig ibiza izingcingo ezilungile zesistimu
• Umphumela: Kubizwa izingcingo ezilungile
• Isivivinyo 2: ukusebenzisa i-srapy ukuze uhlole ukuthi singakwazi yini ukunquma amagama ngokudlula amalabhulali esistimu
• Umphumela: singakwazi
• Isivivinyo 3: sebenzisa i-rpm –V kuphakheji ye-libdns namafayela omtapo wezincwadi we-md5sum
• Umphumela: ikhodi yelabhulali ifana ngokuphelele nekhodi ohlelweni lokusebenza
• Isivivinyo 4: faka isithombe sesistimu yempande yomsebenzisi ku-VM ngaphandle kwalokhu kuziphatha, sebenzisa i-chroot, bheka ukuthi i-DNS iyasebenza yini
• Umphumela: I-DNS isebenza kahle

Isiphetho esisekelwe ekuhlolweni: inkinga ayikho emitatsheni yolwazi

I-hypothesis: Kunephutha kuzilungiselelo ze-DNS

• Isivivinyo 1: hlola i-tcpdump futhi ubone ukuthi amaphakethe e-DNS athunyelwe futhi abuyiselwe ngendlela efanele ngemuva kokusebenza dig
• Umphumela: amaphakethe asakazwa ngendlela efanele
• Isivivinyo 2: hlola kabili iseva /etc/nsswitch.conf и /etc/resolv.conf
• Umphumela: konke kulungile

Isiphetho esisekelwe ekuhlolweni: inkinga ayikho ngokucushwa kwe-DNS

I-hypothesis: umongo wonakele

• Isivivinyo: faka i-kernel entsha, hlola isiginesha, qala kabusha
• Umphumela: ukuziphatha okufanayo

Isiphetho esisekelwe ekuhlolweni: i-kernel ayilimele

I-hypothesis: ukuziphatha okungalungile kwenethiwekhi yomsebenzisi (noma isixhumi esibonakalayo senethiwekhi ye-hypervisor)

• Isivivinyo 1: Hlola izilungiselelo zakho zohlelo lokuvikela
• Umphumela: i-firewall idlula amaphakethe e-DNS kukho kokubili umsingathi kanye ne-GCP
• Isivivinyo sesi-2: nqamula ithrafikhi futhi uqaphe ukulunga kokudluliselwa nokubuyiselwa kwezicelo ze-DNS
• Umphumela: i-tcpdump iqinisekisa ukuthi umsingathi uthole amaphakethe okubuyisela

Isiphetho esisekelwe ekuhlolweni: inkinga ayikho kunethiwekhi

I-hypothesis: iseva yemethadatha ayisebenzi

• Isivivinyo 1: hlola amalogi eseva yemethadatha ukuze uthole okudidayo
• Umphumela: akukho okudidayo kumalogi
• Isivivinyo 2: Dlula iseva yemethadatha nge dig @8.8.8.8
• Umphumela: Ukulungiswa kwephukile ngisho nangaphandle kokusebenzisa iseva yemethadatha

Isiphetho esisekelwe ekuhlolweni: inkinga ayikho ngeseva yemethadatha

Isiphetho sendaba: sihlole wonke amasistimu angaphansi ngaphandle izilungiselelo zesikhathi sokusebenza!

Ngena kuzilungiselelo ze-Kernel Runtime

Ukuze ulungiselele indawo yokusebenzisa i-kernel, ungasebenzisa izinketho zomugqa womyalo (grub) noma isixhumi esibonakalayo se-sysctl. Ngabheka phakathi /etc/sysctl.conf futhi cabanga nje, ngithole izilungiselelo ezimbalwa zangokwezifiso. Ngizizwa sengathi ngibambe okuthile, ngalahla zonke izilungiselelo ezingezona zenethiwekhi noma ezingezona ze-tcp, ngasala nezilungiselelo zezintaba. net.core. Ngabe sengiya lapho izimvume zokusingatha zazikhona ku-VM futhi ngaqala ukusebenzisa izilungiselelo ngamunye ngamunye, omunye emva komunye, nge-VM ephukile, ngaze ngathola umlandu:

net.core.rmem_default = 2147483647

Nansi, ukucushwa kwe-DNS-breaking! Ngasithola isikhali sokubulala. Kodwa kungani lokhu kwenzeka? Ngangisadinga isisusa.

Usayizi webhafa wephakethe le-DNS oyisisekelo ulungiswa nge net.core.rmem_default. Inani elijwayelekile lisendaweni ethile cishe ku-200KiB, kodwa uma iseva yakho ithola amaphakethe e-DNS amaningi, ungase ufune ukwandisa usayizi webhafa. Uma i-buffer igcwele uma iphakethe elisha lifika, isibonelo ngenxa yokuthi uhlelo lokusebenza alucubunguli ngokushesha ngokwanele, uzoqala ukulahlekelwa amaphakethe. Iklayenti lethu likhulise ngendlela efanele usayizi webhafa ngoba lalesaba ukulahleka kwedatha, njengoba lalisebenzisa uhlelo lokuqoqa amamethrikhi ngamaphakethe e-DNS. Inani alisethile laliyinani eliphakeme elingenzeka: 231-1 (uma isethelwe ku-231, i-kernel izobuyisela "INGQINGO ENGAVALIWE").

Kungazelelwe ngabona ukuthi kungani i-nmap ne-scapy zisebenza kahle: bezisebenzisa izisekelo ezingavuthiwe! Amasokhethi aluhlaza ahlukile kumasokhethi avamile: adlula ama-iptables, futhi awakhihliwe!

Kodwa kungani "i-buffer enkulu kakhulu" idala izinkinga? Ngokusobala ayisebenzi njengoba bekuhlosiwe.

Ngalesi sikhathi ngingakwazi ukukhiqiza kabusha inkinga kuma-kernel amaningi kanye nokusabalalisa okuningi. Inkinga isivele ivele ku-3.x kernel futhi manje isivele naku-5.x kernel.

Eqinisweni, ekuqaleni

sysctl -w net.core.rmem_default=$((2**31-1))

I-DNS iyeke ukusebenza.

Ngaqala ukufuna amanani okusebenza ngokusebenzisa i-algorithm elula yokucinga kanambambili futhi ngathola ukuthi uhlelo lusebenza no-2147481343, kodwa le nombolo yayiyisethi yezinombolo ezingenangqondo kimi. Ngiphakamise iklayenti ukuthi lizame le nombolo, futhi laphendula ngokuthi uhlelo lusebenza ne-google.com, kodwa lisanikeze iphutha kwezinye izizinda, ngakho ngaqhubeka nophenyo lwami.

Ngifakile iwashi elidonsela phansi, ithuluzi obekufanele ngabe lisetshenziswe ngaphambili: libonisa kahle ukuthi iphakethe ligcina kuphi ku-kernel. Icala kwaba umsebenzi udp_queue_rcv_skb. Ngilande imithombo ye-kernel futhi ngengeza embalwa imisebenzi printk ukulandelela ukuthi iphakethe liphelelaphi. Ngokushesha ngathola isimo esifanele if, futhi wamane wayigqolozela isikhathi esithile, ngoba kungaleso sikhathi lapho yonke into ekugcineni yahlangana yaba yisithombe esiphelele: 231-1, inombolo engasho lutho, isizinda esingasebenzi... Kwakuyingxenye yekhodi ku. __udp_enqueue_schedule_skb:

if (rmem > (size + sk->sk_rcvbuf))
		goto uncharge_drop;

Sicela uqaphele:

• rmem iwuhlobo lwe-int
• size iwuhlobo lwe-u16 (engasayiniwe ishumi nesithupha-bit int) futhi igcina usayizi wephakethe
• sk->sk_rcybuf iwuhlobo lwe-int futhi igcina usayizi webhafa, ngokwencazelo, olingana nenani elikuyo net.core.rmem_default

Nini sk_rcvbuf isondela ku-231, ukufingqa usayizi wephakethe kungase kubangele ukuchichima okuphelele. Futhi njengoba kuyi-int, inani layo liba libi, ngakho-ke isimo siba yiqiniso uma kufanele sibe ngamanga (ungafunda kabanzi ngalokhu kokuthi isixhumanisi).

Iphutha lingalungiswa ngendlela engasho lutho: ngokusakaza unsigned int. Ngisebenzise ukulungisa futhi ngaqala kabusha uhlelo futhi i-DNS yasebenza futhi.

Ukunambitha ukunqoba

Ngadlulisela lokho engikutholile kuklayenti futhi ngathumela LKML isiqephu se-kernel. Ngijabulile: zonke izingxenye zendida ziyahlangana, ngingachaza kahle ukuthi kungani siqaphele esikubonile, futhi okubaluleke kakhulu, sikwazile ukuthola isisombululo senkinga ngenxa yokusebenza kwethu njengeqembu!

Kuyafaneleka ukuqaphela ukuthi icala liphenduke layivelakancane, futhi ngenhlanhla asivamile ukuthola izicelo eziyinkimbinkimbi ezinjalo ezivela kubasebenzisi.

Source: www.habr.com