Ibali malunga neepakethi ze-DNS ezilahlekileyo ezivela kwinkxaso yobugcisa yeLifu likaGoogle

Ukusuka kuMhleli weBlog kaGoogle: Ngaba wakha wazibuza ukuba iinjineli zikaGoogle Cloud Technical Solutions (TSE) zisingatha njani izicelo zakho zenkxaso? IiNjineli zeNkxaso kwezobuGcisa ze-TSE zinoxanduva lokuchonga nokulungisa imithombo yeengxaki exelwe ngabasebenzisi. Ezinye zezi ngxaki zilula kakhulu, kodwa ngamanye amaxesha udibana netikiti elifuna ingqalelo yeenjineli ezininzi ngaxeshanye. Kweli nqaku, omnye wabasebenzi be-TSE uya kusixelela ngengxaki enye ekhohlisayo kwindlela yakhe yakutshanje - imeko yeepakethi zeDNS ezingekhoyo. Kweli bali, siza kubona indlela iinjineli ekwazile ngayo ukusombulula le meko, kwaye zeziphi izinto ezintsha abazifundileyo ngelixa belungisa impazamo. Siyathemba ukuba eli bali alizukufundisa kuphela nge-bug ehleli nzulu, kodwa likunika ukuqonda kwiinkqubo eziya ekufakeni itikiti lenkxaso ngeLifu likaGoogle.

Ukusombulula iingxaki zombini isayensi kunye nobugcisa. Konke kuqala ngokwakha i-hypothesis malunga nesizathu sokuziphatha okungaqhelekanga kwenkqubo, emva koko kuvavanywa amandla. Nangona kunjalo, ngaphambi kokuba senze i-hypothesis, kufuneka siyichaze ngokucacileyo kwaye siyiyile ingxaki ngokuchanekileyo. Ukuba umbuzo uvakala ungacacanga, ngoko kuya kufuneka uhlalutye yonke into ngononophelo; Lo “bubugcisa” bokusombulula ingxaki.

Ngaphantsi kweLifu likaGoogle, ezo nkqubo ziba nzima ngakumbi, njengoko iLifu likaGoogle lizama kangangoko ukuqinisekisa ubumfihlo babasebenzisi bayo. Ngenxa yoku, iinjineli ze-TSE azikwazi ukufikelela ekuhleleni iisistim zakho, okanye ukukwazi ukujonga ulungelelwaniso ngokubanzi njengoko abasebenzisi besenza. Ngoko ke, ukuvavanya nayiphi na ingcamango yethu, thina (iinjineli) asikwazi ukuguqula ngokukhawuleza inkqubo.

Abanye abasebenzisi bakholelwa ukuba siya kulungisa yonke into efana noomatshini kwinkonzo yemoto, kwaye ngokulula usithumelele i-id yomatshini obonakalayo, kanti enyanisweni inkqubo yenzeka ngefomathi yencoko: ukuqokelela ulwazi, ukwenza kunye nokuqinisekisa (okanye ukuchasa) iingcamango, kwaye, ekugqibeleni, iingxaki zesigqibo zisekelwe kunxibelelwano nomxhasi.

Ingxaki ebuzwayo

Namhlanje sinebali elinesiphelo esihle. Esinye sezizathu zesisombululo esiyimpumelelo setyala elicetywayo yinkcazo ecacileyo kunye nechanekileyo yengxaki. Ngezantsi ungabona ikopi yetikiti lokuqala (elihlelwe ukufihla ulwazi oluyimfihlo):

Lo myalezo uqulethe ulwazi oluninzi oluluncedo kuthi:

• I-VM ethile ixeliwe
• Ingxaki ngokwayo ibonisiwe - i-DNS ayisebenzi
• Kuboniswa apho ingxaki izibonakalisa khona - VM kunye nesikhongozeli
• Amanyathelo athathwe ngumsebenzisi ukuchonga ingxaki abonisiwe.

Isicelo sabhaliswa njenge "P1: Impembelelo Ebalulekileyo - Inkonzo Engasetyenziswayo kwimveliso", oku kuthetha ukubeka iliso rhoqo kwimeko ye-24/7 ngokwenkqubo ethi "Landela iLanga" (unokufunda ngakumbi malunga iiprayorithi zezicelo zabasebenzisi), kunye nokudluliselwa kwayo ukusuka kwelinye iqela lenkxaso yobugcisa ukuya kwelinye ngokutshintsha kwendawo nganye. Nyani, ngelaxesha ingxaki ifikelela kwiqela lethu eZurich, lalisele lijikeleze iglowubhu. Ngeli xesha, umsebenzisi uthathe amanyathelo okunciphisa, kodwa wayesoyika ukuphindaphinda imeko kwimveliso, ekubeni unobangela wengcambu wawungekafunyanwa.

Ngexesha itikiti lifika eZurich, sasisele sinalo lwazi lulandelayo esandleni:

• Umxholo /etc/hosts
• Umxholo /etc/resolv.conf
• isiphelo iptables-save
• Idityaniswe liqela ngrep pcap ifayile

Ngale datha, besikulungele ukuqalisa "uphando" kunye nesigaba sokusombulula ingxaki.

Amanyathelo ethu okuqala

Okokuqala, sihlolisise iilogi kunye nesimo seseva yemethadatha kwaye siqinisekisa ukuba isebenza ngokuchanekileyo. Iseva yemethadatha iphendula kwidilesi ye-IP 169.254.169.254 kwaye, phakathi kwezinye izinto, inoxanduva lokulawula amagama esizinda. Siphinde sajonga kabini ukuba i-firewall isebenza ngokuchanekileyo kunye ne-VM kwaye ayivali iipakethi.

Yayiyingxaki ethile engaqhelekanga: ukukhangela kwe-nmap kwayikhaba ingqikelelo yethu ephambili malunga nokulahleka kweepakethi ze-UDP, ke ngengqondo seza nezinye iindlela ezininzi kunye neendlela zokuzijonga:

• Ngaba iipakethi zilahlwa ngokufanelekileyo? => Khangela imigaqo ye-iptables
• Ngaba ayincinci kakhulu? UMNTU? => Jonga imveliso ip a show
• Ngaba ingxaki ichaphazela kuphela iipakethi ze-UDP okanye i-TCP ngokunjalo? => Qhuba umke dig +tcp
• Ngaba iipakethi zokwemba ezenziweyo zibuyisiwe? => Qhuba umke tcpdump
• Ngaba i-libdns isebenza ngokuchanekileyo? => Qhuba umke strace ukujonga ukuhanjiswa kweepakethi kumacala omabini

Apha sithatha isigqibo sokubiza umsebenzisi ukuba alungise iingxaki ngokuphila.

Ngexesha lomnxeba siyakwazi ukujonga izinto ezininzi:

• Emva kweetshekhi ezininzi asibandakanyi imithetho ye-iptables kuluhlu lwezizathu
• Sijonga ujongano lwenethiwekhi kunye neetafile zomzila, kwaye jonga kabini ukuba i-MTU ichanekile
• Sifumanisa ukuba dig +tcp google.com (TCP) isebenza njengoko kufanelekile, kodwa dig google.com (UDP) ayisebenzi
• Ukubaleka kude tcpdump isasebenza dig, sifumanisa ukuba iipakethi ze-UDP zibuyiselwa
• Siyaqhuba strace dig google.com kwaye sibona indlela ukumba iminxeba ngokuchanekileyo sendmsg() и recvms(), nangona kunjalo eyesibini iphazanyiswa lixesha lokuvala

Ngelishwa, isiphelo sokutshintsha sifika kwaye siyanyanzeleka ukuba sinyuse ingxaki kwindawo yexesha elizayo. Isicelo, nangona kunjalo, sivuse umdla kwiqela lethu, kwaye umntu osebenza naye ucebisa ukwenza ipakethe yokuqala ye-DNS usebenzisa imodyuli yePython escrapy.

from scapy.all import *

answer = sr1(IP(dst="169.254.169.254")/UDP(dport=53)/DNS(rd=1,qd=DNSQR(qname="google.com")),verbose=0)
print ("169.254.169.254", answer[DNS].summary())

Esi siqwenga sidala ipakethe yeDNS kwaye sithumela isicelo kwiseva yemetadata.

Umsebenzisi uqhuba ikhowudi, impendulo ye-DNS ibuyiswa, kwaye isicelo siyifumana, siqinisekisa ukuba akukho ngxaki kwinqanaba lenethiwekhi.

Emva kolunye "uhambo olujikelezayo lwehlabathi," isicelo sibuyela kwiqela lethu, kwaye ndiyidlulisela ngokupheleleyo kum, ndicinga ukuba kuya kuba lula ngakumbi kumsebenzisi ukuba isicelo siyeka ukujikeleza ukusuka kwindawo ukuya kwenye.

Okwangoku, umsebenzisi uyavuma ngobubele ukubonelela ngomfanekiso wenkqubo. Ezi ziindaba ezimnandi kakhulu: ukukwazi ukuvavanya inkqubo ngokwam kwenza ingxaki ngokukhawuleza, kuba akusafuneki ukuba ndibuze umsebenzisi ukuba aqhube imiyalelo, ndithumele iziphumo kwaye ndihlalutye, ndingenza yonke into ngokwam!

Oogxa bam baqala ukundimonela kancinci. Kwisidlo sasemini sixoxa ngoguqulo, kodwa akukho mntu unoluvo lokuba kuqhubeka ntoni. Ngethamsanqa, umsebenzisi ngokwakhe sele ethathe amanyathelo okunciphisa imiphumo kwaye akakhawulezi, ngoko sinexesha lokusabalalisa ingxaki. Kwaye ekubeni sinomfanekiso, sinokuqhuba naluphi na uvavanyo olunomdla kuthi. Kakhulu!

Ukuthatha inyathelo umva

Omnye weyona mibuzo idumileyo yodliwano-ndlebe kwizikhundla zobunjineli beenkqubo yile: “Kwenzeka ntoni xa u-ping www.google.com? Umbuzo mkhulu, kuba umviwa kufuneka achaze yonke into ukusuka kwiqokobhe ukuya kwindawo yomsebenzisi, ukuya kwi-kernel yenkqubo kwaye emva koko kuthungelwano. Ndiyancuma: ngamanye amaxesha imibuzo yodliwano-ndlebe iba luncedo ebomini bokwenyani...

Ndithatha isigqibo sokusebenzisa lo mbuzo we-HR kwingxaki yangoku. Ukuthetha nje, xa uzama ukumisela igama le-DNS, oku kulandelayo kuyenzeka:

1. Isicelo sibiza ilayibrari yesixokelelwano efana ne-libdns
2. libdns ijonga ubumbeko lwesixokelelwano apho iseva yeDNS kufuneka iqhagamshelane nayo (kwidayagram le yi-169.254.169.254, iseva yemetadata)
3. i-libdns isebenzisa iifowuni zenkqubo ukwenza i-socket ye-UDP (SOKET_DGRAM) kwaye ithumele iipakethi ze-UDP ngombuzo we-DNS macala omabini.
4. Ngokusebenzisa i-sysctl interface ungaqwalasela i-UDP stack kwinqanaba le-kernel
5. I-kernel inxibelelana ne-hardware ukuthumela iipakethi kwinethiwekhi nge-interface yenethiwekhi
6. I-hypervisor ibamba kwaye ithumele ipakethi kwiseva yemetadata xa udibana nayo
7. Iseva yemethadatha, ngomlingo wayo, imisela igama le-DNS kwaye ibuyisela impendulo isebenzisa indlela efanayo

Makhe ndikukhumbuze ukuba zeziphi iingcamango esele siziqwalasele:

Ingqikelelo: Amathala eencwadi aphukileyo

• Uvavanyo 1: qhuba umtya kwisistim, khangela ukuba ukumba ubiza iifowuni ezichanekileyo zesistim
• Isiphumo: Iifowuni zesistim ezichanekileyo ziyabizwa
• Uvavanyo 2: usebenzisa israpy ukujonga ukuba singakwazi na ukumisela amagama ngokugqitha amathala eencwadi
• Isiphumo: sinako
• Uvavanyo 3: sebenzisa i-rpm -V kwiphakheji ye-libdns kunye neefayile zethala leencwadi ze-md5sum
• Isiphumo: ikhowudi yelayibrari ifana ngokupheleleyo nekhowudi kwinkqubo yokusebenza
• Uvavanyo 4: nyusa umfanekiso wenkqubo yengcambu yomsebenzisi kwiVM ngaphandle kolu kuziphatha, sebenzisa i-chroot, bona ukuba iDNS iyasebenza
• Isiphumo: I-DNS isebenza ngokuchanekileyo

Isiphelo esisekwe kuvavanyo: ingxaki ayikho kumathala eencwadi

I-hypothesis: Kukho impazamo kwiisetingi ze-DNS

• Uvavanyo loku-1: khangela i-tcpdump kwaye ubone ukuba iipakethi ze-DNS zithunyelwe kwaye zibuyiselwe ngokuchanekileyo emva kokubaleka ukumba
• Isiphumo: iipakethi zihanjiswa ngokuchanekileyo
• Uvavanyo 2: khangela kabini kwiseva /etc/nsswitch.conf и /etc/resolv.conf
• Isiphumo: yonke into ichanekile

Isiphelo esisekwe kuvavanyo: ingxaki ayikho kuqwalaselo lweDNS

I-hypothesis: eyonakeleyo ingundoqo

• Uvavanyo: faka i-kernel entsha, khangela utyikityo, qala kwakhona
• Isiphumo: ukuziphatha okufanayo

Isiphelo esisekwe kuvavanyo: ikernel ayonakaliswa

I-hypothesis: ukuziphatha okungalunganga kwenethiwekhi yomsebenzisi (okanye i-hypervisor network interface)

• Uvavanyo 1: Jonga iisetingi zefirewall zakho
• Isiphumo: I-firewall idlula iipakethi ze-DNS kuzo zombini inginginya kunye ne-GCP
• Uvavanyo lwe-2: thintela i-traffic kwaye ubeke iliso ukuchaneka kokuhanjiswa kunye nokubuyiswa kwezicelo ze-DNS
• Isiphumo: I-tcpdump iqinisekisa ukuba umamkeli ufumene iipakethi zokubuya

Isiphelo esisekwe kuvavanyo: ingxaki ayikho kwi network

I-hypothesis: iseva yemetadata ayisebenzi

• Uvavanyo loku-1: khangela iilogi zeseva yemethadatha ye-anomalies
• Isiphumo: akukho zimpazamo kwiilog
• Uvavanyo 2: Yidlula iseva yemetadata nge dig @8.8.8.8
• Isiphumo: Isisombululo saphukile nangaphandle kokusebenzisa iseva yemetadata

Isiphelo esisekwe kuvavanyo: ingxaki ayikho kumncedisi we metadata

Undoqo: sivavanye zonke iinkqubo ezisezantsi ngaphandle iisetingi zexesha lokusebenza!

Ukuntywila kwi-Kernel Runtime Settings

Ukuqwalasela imeko-bume yophumezo lwe kernel, ungasebenzisa iinketho zelayini yomyalelo (grub) okanye ujongano lwe sysctl. Ndajonga /etc/sysctl.conf kwaye cinga nje, ndifumene useto oluninzi lwesiko. Ndiziva ngathi ndibambe into ethile, ndalahla zonke iisetingi ezingezizo inethiwekhi okanye ezingezizo i-tcp, ndishiyeke nezicwangciso zentaba. net.core. Emva koko ndaya apho iimvume zomkhosi zazikhona kwi-VM kwaye ndaqalisa ukusebenzisa useto nganye nganye, enye emva kwenye, ngeVM eyaphukileyo, de ndafumana umoni:

net.core.rmem_default = 2147483647

Nantsi, i-DNS-breaking configuration! Ndasifumana isixhobo sokubulala. Kodwa kutheni kusenzeka oku? Bendisafuna intshukumisa.

Ubungakanani bepakethe ye-DNS yebuffer esisiseko iqwalaselwe nge net.core.rmem_default. Ixabiso eliqhelekileyo likwindawo ethile malunga ne-200KiB, kodwa ukuba umncedisi wakho ufumana iipakethi ezininzi ze-DNS, unokufuna ukonyusa ubungakanani bebuffer. Ukuba i-buffer igcwele xa ipakethi entsha ifika, umzekelo ngenxa yokuba usetyenziso aluyiqhubeki ngokukhawuleza ngokwaneleyo, ngoko uya kuqalisa ukuphulukana neepakethi. Umxhasi wethu wandise ngokuchanekileyo ubungakanani be-buffer kuba wayesoyika ilahleko yedatha, kuba wayesebenzisa isicelo sokuqokelela iimetrics ngeepakethi zeDNS. Ixabiso elimiselweyo lalinobuninzi obunokwenzeka: 231-1 (ukuba imiselwe kwi-231, i-kernel iya kubuya "INVALID ARGUMENT").

Ngequbuliso ndiye ndaqonda ukuba kutheni i-nmap kunye ne-scapy zisebenza ngokuchanekileyo: bezisebenzisa iisokethi ezingakrwada! Iziseko ezikrwada zahlukile kwiisokethi eziqhelekileyo: zidlula iiptables, kwaye azikhutshelwanga!

Kodwa kutheni "i-buffer enkulu kakhulu" ibangela iingxaki? Ngokucacileyo ayisebenzi njengoko bekucetyiwe.

Ngeli xesha ndiyakwazi ukuvelisa kwakhona ingxaki kwiinkozo ezininzi kunye nokuhanjiswa okuninzi. Ingxaki sele ivele kwi-3.x kernel kwaye ngoku iphinde yavela kwi-5.x kernel.

Ngokuqhelekileyo, ekuqaleni

sysctl -w net.core.rmem_default=$((2**31-1))

I-DNS iyekile ukusebenza.

Ndaqala ukukhangela amaxabiso okusebenza ngokusebenzisa i-algorithm elula yokukhangela ibhinari kwaye ndafumanisa ukuba inkqubo isebenze nge-2147481343, kodwa eli nani yayiyisethi yamanani angenantsingiselo kum. Ndicebise ukuba umxhasi azame le nombolo, kwaye waphendula ukuba inkqubo isebenze kunye ne-google.com, kodwa inike impazamo kwezinye iindawo, ngoko ndaqhubeka nophando lwam.

Ndifakile dropwatch, isixhobo ebekufanele ukuba sisetyenziswe ngaphambili: ibonisa ngqo apho kwi-kernel ipakethi iphelela khona. Unobangela yayingumsebenzi udp_queue_rcv_skb. Ndikhuphele imithombo ye-kernel kwaye ndongeza ezimbalwa imisebenzi printk ukulandelela apho kanye kanye ipakethi iphelela khona. Ngokukhawuleza ndafumana imeko efanelekileyo if, kwaye wayijonga nje ixesha elithile, kuba ngelo xesha yonke into ekugqibeleni yadibana ibe ngumfanekiso opheleleyo: 231-1, inani elingenantsingiselo, isizinda esingasebenziyo ... Kwakuyikhowudi yekhowudi kwi. __udp_enqueue_schedule_skb:

if (rmem > (size + sk->sk_rcvbuf))
		goto uncharge_drop;

Nceda uqaphele:

• rmem luhlobo lwe int
• size yeyohlobo u16 (engabhalwanga elinesithandathu-bit int) kwaye igcina ubungakanani bepakethi
• sk->sk_rcybuf iludidi lwe int kwaye igcina ubungakanani be buffer leyo, ngengcaciso, ilingana nexabiso kwi net.core.rmem_default

Xa sk_rcvbuf isondela ku-231, ukushwankathela ubungakanani bepakethi kunokubangela ukuphuphumala okupheleleyo. Kwaye kuba iyi-int, ixabiso layo liba libi, ke imeko iba yinyani xa kufanele ukuba bubuxoki (unokufunda ngakumbi ngale nto apha unxibelelwano).

Impazamo inokulungiswa ngendlela engabalulekanga: ngokuphosa unsigned int. Ndasebenzisa ukulungiswa kwaye ndaphinda ndaqalisa inkqubo kwaye i-DNS yasebenza kwakhona.

Ukungcamla uloyiso

Ndathumela iziphumo zam kumxhasi kwaye ndathumela I-LKML isiqwenga se-kernel. Ndiyavuya: isiqwenga ngasinye sephazili siyadibana, ndingachaza kanye ukuba kutheni siqwalasele oko sikubonileyo, kwaye okona kubaluleke kakhulu, sikwazile ukufumana isisombululo kwingxaki ngokusebenzisana!

Kufanelekile ukuqaphela ukuba ityala liye labonakala linqabile, kwaye ngethamsanqa asifane sifumane izicelo ezinzima ezinjalo kubasebenzisi.

umthombo: www.habr.com