Nyaya yekushaikwa kweDNS mapaketi kubva kuGoogle Cloud technical support

Kubva kuGoogle Blog Editor: Wakambozvibvunza here kuti mainjiniya eGoogle Cloud Technical Solutions (TSE) anobata sei zvikumbiro zvako zvekutsigira? TSE Technical Support Engineers vane basa rekuona nekugadzirisa zvitubu zvinoshumwa nemushandisi zvematambudziko. Mamwe ematambudziko aya ari nyore, asi dzimwe nguva unosangana netikiti rinoda kutariswa kwemainjiniya akati wandei kamwechete. Muchinyorwa chino, mumwe wevashandi veTSE achatiudza nezve dambudziko rimwe rinonyengerera kubva mukuita kwake kwazvino - nyaya yekushayikwa kweDNS mapaketi. Munyaya ino, tichaona kuti mainjiniya akakwanisa sei kugadzirisa mamiriro acho, uye ndezvipi zvinhu zvitsva zvavakadzidza vachigadzirisa chikanganiso. Tinovimba kuti nyaya iyi haingokudzidzisei nezve bug yakadzika-yakadzika, asi inokupawo nzwisiso yemaitiro anoenda mukunyora tikiti rekutsigira neGoogle Cloud.

Troubleshooting zvose sainzi uye unyanzvi. Izvo zvose zvinotanga nekuvaka fungidziro pamusoro pechikonzero chekusazvibata kwemaitiro ehurongwa, mushure mezvo inoedzwa simba. Zvisinei, tisati tagadzira fungidziro, tinofanira kunyatsotsanangura uye kunyatsogadzira dambudziko racho. Kana mubvunzo uchiita seusina kujeka, saka iwe uchafanirwa kuongorora zvese nemazvo; Iyi ndiyo "unyanzvi" hwekugadzirisa matambudziko.

Pasi peGoogle Cloud, maitiro akadai anowedzera kuoma, sezvo Google Cloud inoedza nepainogona napo kuvimbisa kuvanzika kwevashandisi vayo. Nekuda kweizvi, mainjiniya eTSE haakwanise kugadzirisa masisitimu ako, kana kugona kuona zvigadziriso zvakafara sezvinoita vashandisi. Naizvozvo, kuti tiedze chero yekufungidzira kwedu, isu (mainjiniya) hatigone kukurumidza kugadzirisa sisitimu.

Vamwe vashandisi vanotenda kuti isu tichagadzirisa zvese semakanika musevhisi yemotokari, uye ingotitumira id yemuchina chaiwo, asi muchokwadi maitiro acho anoitika muchimiro chekutaura: kuunganidza ruzivo, kuumba uye kusimbisa (kana kuramba) fungidziro, uye, pakupedzisira, matambudziko echisarudzo anobva pakukurukurirana nemutengi.

Dambudziko riripo

Nhasi tine nyaya ine magumo akanaka. Chimwe chezvikonzero zvekugadzirisa kwakabudirira kwenyaya yakarongwa irondedzero yakadzama uye yakarurama yedambudziko. Pazasi unogona kuona kopi yetikiti rekutanga (rakapepetwa kuvanza zvakavanzika ruzivo):

Iyi meseji ine ruzivo rwakawanda runobatsira kwatiri:

• Specific VM yakataurwa
• Dambudziko pacharo rinoratidzwa - DNS haishande
• Inoratidzwa apo dambudziko rinozviratidza - VM uye mudziyo
• Matanho atorwa nemushandisi kuona dambudziko anoratidzwa.

Chikumbiro ichi chakanyoreswa se "P1: Critical Impact - Sevhisi Isingashandiswe mukugadzirwa", zvinoreva kugara uchitarisa mamiriro ezvinhu 24/7 maererano ne "Tevera Zuva" chirongwa (unogona kuverenga zvakawanda nezve. zvakakosha zvezvikumbiro zvevashandisi), nekutamiswa kwayo kubva kune imwe tekinoroji yekutsigira timu kuenda kune imwe neimwe nguva yekuchinja nzvimbo. Muchokwadi, panguva iyo dambudziko rakasvika kuchikwata chedu muZurich, rakanga ratotenderera pasirese. Panguva ino, mushandisi akange atora matanho ekuderedza, asi aitya kudzokorora kwemamiriro ezvinhu mukugadzirwa, sezvo chikonzero chaive chisati chawanikwa.

Pakasvika tikiti kuZurich, takanga tatova neruzivo rwunotevera paruoko:

• Zvemukati /etc/hosts
• Zvemukati /etc/resolv.conf
• mhedziso iptables-save
• Yakaunganidzwa neboka ngrep pcap file

Neiyi data, isu takanga takagadzirira kutanga "yekuferefeta" uye kugadzirisa dambudziko.

Matanho edu ekutanga

Chekutanga pane zvese, takatarisa matanda uye chimiro chemetadata server uye takaita shuwa kuti yaishanda nemazvo. Iyo metadata server inopindura kune IP kero 169.254.169.254 uye, pakati pezvimwe zvinhu, ine basa rekutonga mazita emadomasi. Isu takaongororawo kaviri kuti firewall inoshanda nemazvo neVM uye haivharidzi mapaketi.

Yakanga iri imwe mhando yedambudziko risinganzwisisike: iyo nmap cheki yakaramba fungidziro yedu huru nezvekurasika kwemapaketi eUDP, saka isu mupfungwa takauya nedzimwe sarudzo dzakati wandei uye nzira dzekudzitarisa:

• Mapakiti anodonhedzwa zvakasarudzwa here? => Tarisa iptables mitemo
• Haisi idiki here? MUNHU? => Tarisa zvabuda ip a show
• Dambudziko rinokanganisa chete UDP mapaketi kana TCP zvakare? => Dzinga kure dig +tcp
• Mapaketi akacherwa anodzoswa here? => Dzinga kure tcpdump
• Libdns inoshanda nemazvo here? => Dzinga kure strace kutarisa kutapurirana kwemapaketi mumativi ese

Pano isu tinosarudza kufonera mushandisi kugadzirisa matambudziko live.

Panguva yekufona tinokwanisa kutarisa zvinhu zvakati:

• Mushure mekutarisa kwakati wandei isu tinobvisa iptables mitemo kubva pane rondedzero yezvikonzero
• Isu tinotarisa network interfaces uye routing matafura, uye kaviri-tarisa kuti MTU ndeyechokwadi
• Tinozviona izvozvo dig +tcp google.com (TCP) inoshanda sezvainofanira, asi dig google.com (UDP) haishande
• Vachidzinga tcpdump ichiri kushanda dig, tinoona kuti mapepa eUDP ari kudzorerwa
• Tinodzinga strace dig google.com uye tinoona kuti kuchera sei kufona nenzira kwayo sendmsg() и recvms(), zvisinei yechipiri inovhiringwa nekufamba kwenguva

Nehurombo, kupera kwekuchinja kunosvika uye isu tinomanikidzwa kuwedzera dambudziko kune inotevera nguva zone. Chikumbiro ichi, zvisinei, chakamutsa kufarira muchikwata chedu, uye waanoshanda naye anokurudzira kugadzira iyo yekutanga DNS package uchishandisa scrapy Python module.

from scapy.all import *

answer = sr1(IP(dst="169.254.169.254")/UDP(dport=53)/DNS(rd=1,qd=DNSQR(qname="google.com")),verbose=0)
print ("169.254.169.254", answer[DNS].summary())

Ichi chidimbu chinogadzira DNS packet uye chinotumira chikumbiro kune metadata server.

Mushandisi anomhanyisa kodhi, iyo DNS mhinduro inodzoserwa, uye application inoigamuchira, ichisimbisa kuti hapana dambudziko padanho retiweki.

Mushure meimwe "rwendo rwepasirese," chikumbiro chinodzokera kuchikwata chedu, uye ndinochiendesa kwandiri zvachose, ndichifunga kuti zvichava nyore kumushandisi kana chikumbiro chikamira kutenderera kubva panzvimbo nenzvimbo.

Zvichakadaro, mushandisi anobvuma nemutsa kupa mufananidzo weiyo system system. Idzi inhau dzakanaka kwazvo: kugona kuyedza iyo sisitimu pachangu kunoita kuti dambudziko rikurumidze, nekuti ini handichafanira kubvunza mushandisi kuti aite mirairo, nditumire mhinduro uye ndizviongorore, ndinogona kuita zvese ini!

Vandinoshanda navo vave kundiitira shanje zvishoma. Pakudya kwemasikati tinokurukura nezvekutendeuka, asi hapana anoziva zviri kuitika. Neraki, mushandisi pachake akatotora matanho ekudzikisa mhedzisiro uye haana kukurumidza, saka isu tine nguva yekuparadzanisa dambudziko. Uye sezvo tine mufananidzo, tinogona kumhanya chero bvunzo dzinotifadza. Hukuru!

Kutora nhanho kumashure

Imwe yemibvunzo yakakurumbira yekubvunzurudza yezvinzvimbo zveinjiniya ndeye: "Chii chinoitika kana iwe uchirira www.google.com? Mubvunzo wakanaka, sezvo mukwikwidzi achida kutsanangura zvese kubva kugoko kuenda kunzvimbo yemushandisi, kune system kernel uyezve kune network. Ndinonyemwerera: dzimwe nguva mibvunzo yekubvunzurudza inoshanduka kuve inobatsira muhupenyu chaihwo ...

Ini ndinosarudza kushandisa uyu mubvunzo weHR kune dambudziko razvino. Zvichireva kutaura, paunoyedza kuona zita reDNS, zvinotevera zvinoitika:

1. Chishandiso chinodaidza raibhurari yehurongwa senge libdns
2. libdns inotarisa masisitimu ehurongwa kune iyo DNS server yainofanirwa kubata (mudhayagiramu iyi 169.254.169.254, metadata server)
3. libdns inoshandisa masystem ekufona kugadzira UDP socket (SOKET_DGRAM) uye kutumira UDP mapaketi ane DNS mubvunzo mumativi ese.
4. Kuburikidza neiyo sysctl interface unogona kugadzirisa iyo UDP stack padanho re kernel
5. Iyo kernel inodyidzana nehardware kufambisa mapaketi pamusoro petiweki kuburikidza netiweki interface
6. Iyo hypervisor inobata uye inotumira pakiti kune metadata server paunosangana nayo
7. Iyo metadata server, nemashiripiti ayo, inosarudza zita reDNS uye inodzosera mhinduro uchishandisa nzira imwechete

Rega ndikuyeuchidze kuti ndedzipi fungidziro dzatatoongorora:

Hypothesis: Maraibhurari akaputswa

• Muedzo 1: mhanyisa strace musystem, tarisa kuti dig inodaidza iyo chaiyo system mafoni
• Mhedzisiro: Chaiyo system inofona inodanwa
• Muedzo 2: kushandisa srapy kutarisa kana isu tichigona kuona mazita nekupfuura system library
• Mhedzisiro: tinogona
• Muedzo 3: mhanya rpm -V pane libdns package uye md5sum raibhurari mafaera
• Mhedzisiro: iyo raibhurari kodhi yakafanana zvachose nekodhi mune inoshanda sisitimu yekushandisa
• Muedzo wechina: isa mufananidzo wemudzi wemushandisi paVM isina maitiro aya, mhanyisa chroot, ona kana DNS ichishanda
• Mhedzisiro: DNS inoshanda nemazvo

Mhedziso yakavakirwa pabvunzo: dambudziko harisi mumaraibhurari

Hypothesis: Pane kukanganisa muDNS marongero

• Muedzo 1: tarisa tcpdump uye ona kana DNS mapaketi akatumirwa uye akadzoserwa nemazvo mushure mekumhanya dig
• Mhedzisiro: mapaketi anofambiswa nemazvo
• Muedzo 2: tarisa kaviri pane server /etc/nsswitch.conf и /etc/resolv.conf
• Mhedzisiro: zvese zvakanaka

Mhedziso yakavakirwa pabvunzo: dambudziko harisi neiyo DNS kumisikidzwa

Hypothesis: musimboti wakakuvadzwa

• Muedzo: isa kernel nyowani, tarisa siginecha, tangazve
• Mhedzisiro: maitiro akafanana

Mhedziso yakavakirwa pabvunzo: kernel haina kukuvara

Hypothesis: maitiro asina kunaka emushandisi network (kana hypervisor network interface)

• Muedzo 1: Tarisa zvigadziriso zvako zvefirewall
• Mhedzisiro: iyo firewall inopfuura DNS mapaketi pane ese ari maviri anotambira uye GCP
• Muedzo 2: bvisa traffic uye tarisa iko kurongeka kwekufambisa uye kudzoka kweDNS zvikumbiro
• Mhedzisiro: tcpdump inosimbisa kuti muenzi agamuchira mapepa ekudzoka

Mhedziso yakavakirwa pabvunzo: dambudziko harisi munetwork

Hypothesis: metadata server haisi kushanda

• Muedzo wekutanga: tarisa metadata server logs yeanomalies
• Mhedzisiro: hapana anomalies mumatanda
• Muedzo wechipiri: Bvisa metadata server kuburikidza dig @8.8.8.8
• Mhedzisiro: Kugadziriswa kwakatyoka kunyangwe pasina kushandisa metadata server

Mhedziso yakavakirwa pabvunzo: dambudziko harisi nemetadata server

Chinonyanya kukosha: takaedza ese subsystems kunze Runtime settings!

Kunyura muKernel Runtime Settings

Kugadzirisa kernel execution nharaunda, unogona kushandisa command line sarudzo (grub) kana iyo sysctl interface. Ndakatarisa mukati /etc/sysctl.conf uye ingofunga, ndakawana akati wandei etsika marongero. Ndichinzwa sekunge ndakanga ndabata pane chimwe chinhu, ndakarasa ese asiri-network kana asiri-tcp marongero, ndichisara nemakomo. net.core. Ipapo ndakaenda kwaive nemvumo yevaenzi yaive muVM ndokutanga kushandisa zvigadziriso imwe neimwe, imwe neimwe, neVM yakatyoka, kudzamara ndawana muparadzi:

net.core.rmem_default = 2147483647

Hezvino izvo, DNS-kutyora gadziriro! Ndakawana chombo chemhondi. Asi nei izvi zviri kuitika? Ndakanga ndichiri kuda chinangwa.

Iyo yakakosha DNS packet buffer size inogadziriswa kuburikidza net.core.rmem_default. Yakajairika kukosha kune imwe nzvimbo yakatenderedza 200KiB, asi kana sevha yako ikagamuchira akawanda eDNS mapaketi, ungangoda kuwedzera buffer saizi. Kana iyo buffer yakazara kana pakiti nyowani yasvika, semuenzaniso nekuti iyo application haisi kuigadzirisa nekukurumidza zvakakwana, ipapo iwe unotanga kurasikirwa mapaketi. Mutengi wedu akawedzera saizi yebhafa nemazvo nekuti aitya kurasikirwa nedata, sezvo aishandisa application yekuunganidza metrics kuburikidza neDNS mapaketi. Mutengo waakaisa ndiwo wakanyanya kukwanisika: 231-1 (kana yakaiswa ku231, kernel ichadzoka "INVALID ARGUMENT").

Pakarepo ndakaziva kuti nei nmap uye scapy yakashanda nemazvo: ivo vaishandisa zvigadziko zvakabikwa! Zvigadziko zvakasvibirira zvakasiyana nezvigadziko zvenguva dzose: zvinodarika iptables, uye hazvina buffer!

Asi nei "buffer yakakura kwazvo" ichikonzera matambudziko? Zviri pachena kuti hazvishande sezvakarongwa.

Panguva ino ndaigona kuburitsa dambudziko pane dzakawanda kernels uye kugovera kwakawanda. Dambudziko rakatoonekwa pa 3.x kernel uye ikozvino rakaonekwawo pa 5.x kernel.

Chokwadi, pakutanga

sysctl -w net.core.rmem_default=$((2**31-1))

DNS yakamira kushanda.

Ndakatanga kutsvaga maitiro ekushanda kuburikidza neakareruka bhinari yekutsvaga algorithm uye ndakaona kuti sisitimu yakashanda ne2147481343, asi iyi nhamba yaive isina zvazvinoreva seti yenhamba kwandiri. Ndakakurudzira mutengi kuti aedze nhamba iyi, uye akapindura kuti sisitimu yakashanda negoogle.com, asi yakapa chikanganiso nemamwe madomasi, saka ndakaenderera mberi nekuferefeta kwangu.

Ndakaisa dropwatch, chishandiso chaifanira kunge chakamboshandiswa kare: chinonyatsoratidza kuti mu kernel pakiti inoguma kupi. Mhosva ndiyo yaive basa udp_queue_rcv_skb. Ndakatora zvinyorwa zve kernel uye ndakawedzera zvishoma mabasa printk kutsvaga paiperera packet chaipo. Ndakakurumidza kuwana mamiriro chaiwo if, uye akangotarisa pairi kwenguva yakati, nokuti panguva iyo zvinhu zvose zvakazosangana pamwe chete mumufananidzo wose: 231-1, nhamba isina maturo, nzvimbo isingashandisi ... Yakanga iri chidimbu chekodhi mukati. __udp_enqueue_schedule_skb:

if (rmem > (size + sk->sk_rcvbuf))
		goto uncharge_drop;

Tapota cherechedza:

• rmem ndeye type int
• size ndeyemhando u16 (isina kusaina gumi nematanhatu-bit int) uye inochengeta saizi yepakiti
• sk->sk_rcybuf ndeyemhando int uye inochengeta saizi yebhafa iyo, netsanangudzo, yakaenzana nekukosha mukati net.core.rmem_default

When sk_rcvbuf inosvika 231, kupfupisa saizi yepakiti inogona kukonzera integer kufashukira. Uye sezvo iri int, kukosha kwayo kunova kwakashata, saka mamiriro acho anova echokwadi kana achifanirwa kunge ari enhema (unogona kuverenga zvakawanda nezve izvi pa batanidzo).

Iko kukanganisa kunogona kugadziriswa nenzira diki: nekukanda unsigned int. Ndakashandisa kugadzirisa uye ndakatangazve system uye DNS yakashanda zvakare.

Kuravira kukunda

Ndakaendesa zvandainge ndawana kuclient ndokutumira LKML kernel patch. Ndinofara: chidimbu chega chega chepuzzle chinoenderana, ndinogona kutsanangura chaizvo kuti sei takaona zvatakaona, uye zvakanyanya kukosha, takakwanisa kuwana mhinduro kudambudziko nekuda kwekubatana kwedu!

Izvo zvakakosha kuti tizive kuti nyaya yacho yakave isingawanzo, uye nerombo rakanaka isu hatiwanzogashira zvikumbiro zvakaoma kudaro kubva kuvashandisi.

Source: www.habr.com