Nkhani yosowa mapaketi a DNS ochokera ku Google Cloud technical support

Kuchokera ku Google Blog Editor: Kodi mudayamba mwadzifunsapo momwe mainjiniya a Google Cloud Technical Solutions (TSE) amachitira zopempha zanu? Mainjiniya othandizira aukadaulo a TSE ali ndi udindo wozindikira ndi kukonza zomwe zidanenedwa ndi ogwiritsa ntchito. Ena mwamavutowa ndi osavuta, koma nthawi zina mumapeza tikiti yomwe imafunikira chidwi ndi mainjiniya angapo nthawi imodzi. M'nkhaniyi, m'modzi mwa ogwira ntchito ku TSE atiuza za vuto limodzi lovuta kwambiri kuchokera muzochita zake zaposachedwa - mlandu wosowa mapaketi a DNS. M'nkhaniyi, tiwona momwe mainjiniya adakwanitsa kuthetsa vutoli, ndi zinthu zatsopano zomwe adaphunzira pokonza cholakwikacho. Tikukhulupirira kuti nkhaniyi sikuti ingokuphunzitsani za cholakwika chokhazikika, komanso imakupatsani chidziwitso panjira zomwe zimakupangitsani kutumiza tikiti yothandizira ndi Google Cloud.

Kuthetsa mavuto ndi sayansi komanso luso. Zonse zimayamba ndikumanga lingaliro la chifukwa cha khalidwe losavomerezeka la dongosolo, pambuyo pake limayesedwa kuti likhale ndi mphamvu. Komabe, tisanapange lingaliro, tiyenera kufotokoza momveka bwino ndikukonza vutolo. Ngati funso likuwoneka losamveka bwino, ndiye kuti muyenera kusanthula zonse mosamala; Ichi ndi "luso" la kuthetsa mavuto.

Pansi pa Google Cloud, njira zoterezi zimakhala zovuta kwambiri, monga Google Cloud imayesetsa kutsimikizira zinsinsi za ogwiritsa ntchito. Chifukwa cha ichi, mainjiniya a TSE alibe mwayi wosintha makina anu, komanso amatha kuwona masinthidwe mozama momwe ogwiritsa ntchito amachitira. Chifukwa chake, kuyesa malingaliro athu aliwonse, ife (akatswiri) sitingasinthe dongosololi mwachangu.

Ogwiritsa ntchito ena amakhulupirira kuti tidzakonza zonse monga zimango pagalimoto yamagalimoto, ndikungotumiza ma id a makina owoneka bwino, pomwe kwenikweni njirayi imachitika mwanjira yolankhulirana: kusonkhanitsa zambiri, kupanga ndi kutsimikizira (kapena kutsutsa) malingaliro, ndipo, pamapeto pake, vuto lachigamulo limakhazikitsidwa pakulankhulana ndi kasitomala.

Vuto mu funso

Lero tili ndi nkhani yokhala ndi mathero abwino. Chimodzi mwa zifukwa zomwe zathetsera bwino nkhaniyi ndikulongosola mwatsatanetsatane komanso molondola za vutoli. Pansipa mutha kuwona tikiti yoyamba (yosinthidwa kubisa zinsinsi):

Uthengawu uli ndi zambiri zothandiza kwa ife:

• Specific VM yatchulidwa
• Vuto lokha likuwonetsedwa - DNS siigwira ntchito
• Zimawonetsedwa pomwe vuto likuwonekera - VM ndi chidebe
• Njira zomwe wogwiritsa adatenga kuti adziwe vuto zimawonetsedwa.

Pempholo lidalembetsedwa ngati "P1: Critical Impact - Service Unusable in production", kutanthauza kuyang'anira nthawi zonse 24/7 malinga ndi dongosolo la "Follow the Sun" (mutha kuwerenga zambiri za zoyamba za zopempha za ogwiritsa ntchito), ndikusamutsa kuchokera ku gulu lina laukadaulo kupita ku lina ndikusintha kwanthawi zonse. M'malo mwake, pomwe vuto lidafika ku timu yathu ku Zurich, linali litazungulira kale padziko lonse lapansi. Panthawiyi, wogwiritsa ntchitoyo anali atatengapo njira zochepetsera, koma ankawopa kubwereza zomwe zikuchitika pakupanga, popeza chifukwa chake chinali chisanadziwike.

Pomwe tikiti idafika ku Zurich, tinali ndi chidziwitso chotsatirachi:

• Zokhutira /etc/hosts
• Zokhutira /etc/resolv.conf
• Pomaliza iptables-save
• Zosonkhanitsidwa ndi timu ngrep pcap file

Ndi deta iyi, tinali okonzeka kuyamba gawo la "kufufuza" ndi kuthetsa mavuto.

Mayendedwe athu oyamba

Choyamba, tidayang'ana zipika ndi mawonekedwe a seva ya metadata ndikuwonetsetsa kuti ikugwira ntchito moyenera. Seva ya metadata imayankha ku adilesi ya IP 169.254.169.254 ndipo, mwa zina, ili ndi udindo woyang'anira mayina a mayina. Tidawonanso kawiri kuti chowotcha moto chimagwira ntchito bwino ndi VM ndipo sichimatsekereza mapaketi.

Linali vuto linalake lachilendo: cheke cha nmap chinatsutsa lingaliro lathu lalikulu la kutayika kwa mapaketi a UDP, kotero m'malingaliro tidapeza zosankha zingapo ndi njira zowunikira:

• Kodi mapaketi amatayidwa mwa kusankha? => Onani malamulo a iptables
• Kodi sichochepa kwambiri? MTU? => Onani zotsatira ip a show
• Kodi vutoli limakhudzanso mapaketi a UDP okha kapena TCP? => Chotsani kutali dig +tcp
• Kodi mapaketi opangidwa kukumba amabwezedwa? => Chotsani kutali tcpdump
• Kodi libdns ikugwira ntchito bwino? => Chotsani kutali strace kuti muwone kufalikira kwa mapaketi mbali zonse ziwiri

Apa tasankha kuyimbira wosuta kuti athetse mavuto amoyo.

Pakuyimba foni timatha kuyang'ana zinthu zingapo:

• Pambuyo pofufuza kangapo timachotsa malamulo a iptables pamndandanda wa zifukwa
• Timayang'ana mawonekedwe a netiweki ndi matebulo owongolera, ndikuwunikanso kawiri kuti MTU ndiyolondola
• Ife tikupeza izo dig +tcp google.com (TCP) imagwira ntchito momwe iyenera, koma dig google.com (UDP) sikugwira ntchito
• Atathamangitsidwa tcpdump ikugwirabe ntchito dig, tikupeza kuti mapaketi a UDP akubwezeredwa
• Timayendetsa kutali strace dig google.com ndipo tikuwona momwe kukumba mafoni molondola sendmsg() и recvms(), komabe yachiwiri imasokonezedwa ndi kutha kwa nthawi

Tsoka ilo, kutha kwa kusinthaku kumafika ndipo timakakamizika kukulitsa vutoli kudera lanthawi ina. Pempholi, komabe, lidadzutsa chidwi ndi gulu lathu, ndipo mnzake akuti tipange phukusi loyambirira la DNS pogwiritsa ntchito scrapy Python module.

from scapy.all import *

answer = sr1(IP(dst="169.254.169.254")/UDP(dport=53)/DNS(rd=1,qd=DNSQR(qname="google.com")),verbose=0)
print ("169.254.169.254", answer[DNS].summary())

Chidutswachi chimapanga paketi ya DNS ndikutumiza pempho ku seva ya metadata.

Wogwiritsa amayendetsa kachidindo, yankho la DNS limabwezeretsedwa, ndipo pulogalamuyo ilandila, kutsimikizira kuti palibe vuto pamanetiweki.

Pambuyo pa "ulendo wina wapadziko lonse lapansi," pempholo likubwerera ku gulu lathu, ndipo ndimadzisamutsa kwathunthu, ndikuganiza kuti zidzakhala zosavuta kwa wogwiritsa ntchito ngati pempholo likusiya kuzungulira malo ndi malo.

Pakalipano, wogwiritsa ntchito amavomereza mwachifundo kuti apereke chithunzithunzi cha chithunzi cha dongosolo. Iyi ndi nkhani yabwino kwambiri: kutha kuyesa dongosolo ndekha kumapangitsa kuthetsa mavuto mofulumira kwambiri, chifukwa sindiyeneranso kufunsa wogwiritsa ntchito kuti ayendetse malamulo, nditumizireni zotsatira ndikuzisanthula, ndikhoza kuchita zonse ndekha!

Anzanga ayamba kundichitira kaduka pang'ono. Pa nkhomaliro timakambirana za kutembenuka, koma palibe amene akudziwa zomwe zikuchitika. Mwamwayi, wogwiritsa ntchitoyo adachitapo kale njira zochepetsera zotsatira zake ndipo sakufulumira, kotero tili ndi nthawi yothetsa vutoli. Ndipo popeza tili ndi chithunzi, titha kuyesa mayeso aliwonse omwe amatisangalatsa. Zabwino!

Kubwerera mmbuyo

Limodzi mwamafunso otchuka omwe amafunsidwa pamaudindo opanga makina ndi: "Kodi chimachitika ndi chiyani mukayimba www.google.com? Funso ndilabwino, popeza ofuna kusankhidwa amayenera kufotokozera chilichonse kuyambira pachipolopolo mpaka malo ogwiritsa ntchito, mpaka pamakina adongosolo kenako pamaneti. Ndimamwetulira: nthawi zina mafunso oyankhulana amakhala othandiza pamoyo weniweni ...

Ndasankha kugwiritsa ntchito funso la HR pavuto lomwe lilipo. Mwachidule, mukayesa kudziwa dzina la DNS, zotsatirazi zimachitika:

1. Pulogalamuyi imayitanitsa laibulale yamakina monga libdns
2. libdns imayang'ana kasinthidwe kadongosolo komwe seva ya DNS iyenera kulumikizana nayo (pachithunzichi ndi 169.254.169.254, seva ya metadata)
3. libdns amagwiritsa ntchito mafoni kuti apange socket ya UDP (SOKET_DGRAM) ndikutumiza mapaketi a UDP okhala ndi funso la DNS mbali zonse ziwiri.
4. Kupyolera mu mawonekedwe a sysctl mutha kukonza stack ya UDP pamlingo wa kernel
5. Kernel imalumikizana ndi hardware kuti itumize mapaketi pa intaneti kudzera pa intaneti
6. Hypervisor imagwira ndikutumiza paketiyo ku seva ya metadata ikalumikizana nayo
7. Seva ya metadata, mwamatsenga ake, imasankha dzina la DNS ndikubwezera yankho pogwiritsa ntchito njira yomweyo

Ndiroleni ndikukumbutseni zomwe takambirana kale:

Zongoyerekeza: Malaibulale osweka

• Mayeso 1: thamangitsani dongosolo, onetsetsani kuti dig imayitanira mafoni olondola
• Zotsatira: Ma foni olondola amayitanidwa
• Mayeso 2: kugwiritsa ntchito srapy kuti muwone ngati titha kudziwa mayina omwe akudutsa malaibulale adongosolo
• Zotsatira: titha
• Mayeso 3: thamangani rpm -V pa libdns phukusi ndi md5sum library library
• Zotsatira: nambala ya laibulale ndiyofanana kwathunthu ndi kachidindo kogwiritsa ntchito
• Yesani 4: yonjezerani chithunzi cha mizu ya wosuta pa VM popanda khalidweli, yendetsani chroot, muwone ngati DNS ikugwira ntchito
• Zotsatira: DNS imagwira ntchito bwino

Pomaliza potengera mayeso: vuto mulibe m'malaibulale

Zongoyerekeza: Pali zolakwika pamakonzedwe a DNS

• Mayeso 1: yang'anani tcpdump ndikuwona ngati mapaketi a DNS atumizidwa ndikubwezedwa molondola mutatha kukumba
• Zotsatira: mapaketi amafalitsidwa molondola
• Mayeso 2: fufuzani kawiri pa seva /etc/nsswitch.conf и /etc/resolv.conf
• Zotsatira: zonse ndi zolondola

Pomaliza potengera mayeso: vuto siliri ndi kasinthidwe ka DNS

Hypothesis: pachimake kuwonongeka

• Mayeso: khazikitsani kernel yatsopano, fufuzani siginecha, yambitsaninso
• Zotsatira: machitidwe ofanana

Pomaliza potengera mayeso: kernel sichiwonongeka

Hypothesis: machitidwe olakwika a network network (kapena hypervisor network interface)

• Mayeso 1: Yang'anani makonda anu a firewall
• Chotsatira: firewall imadutsa mapaketi a DNS pa onse omwe akukhala nawo komanso GCP
• Mayeso 2: ikani magalimoto ndikuyang'anira kulondola kwa kutumizira ndikubweza zopempha za DNS
• Zotsatira: tcpdump imatsimikizira kuti wolandirayo walandira mapaketi obwerera

Pomaliza potengera mayeso: vuto siliri pa network

Malingaliro: seva ya metadata sikugwira ntchito

• Mayeso 1: yang'anani zipika za seva ya metadata pazosokoneza
• Zotsatira: palibe anomalies mu zipika
• Mayeso 2: Dulani seva ya metadata kudzera dig @8.8.8.8
• Zotsatira: Chigamulo chimasweka ngakhale osagwiritsa ntchito seva ya metadata

Pomaliza potengera mayeso: vuto siliri ndi seva ya metadata

Pansi mzere: tinayesa ma subsystems onse kupatula zosintha nthawi yothamanga!

Kulowa mu Kernel Runtime Zokonda

Kuti mukonze malo opangira kernel, mutha kugwiritsa ntchito zosankha za mzere wamalamulo (grub) kapena mawonekedwe a sysctl. Ndinayang'ana mkati /etc/sysctl.conf ndipo tangoganizani, ndapeza makonda angapo. Ndikumva ngati ndagwira china chake, ndidataya zokonda zonse zomwe sizinali za netiweki kapena zosagwirizana ndi tcp, ndikutsalira ndi mapiri. net.core. Kenako ndidapita komwe zilolezo zolandila zidali mu VM ndikuyamba kugwiritsa ntchito zoikika chimodzi ndi chimodzi, chimodzi ndi china, ndi VM yosweka, mpaka ndidapeza wolakwa:

net.core.rmem_default = 2147483647

Izi ndizo, kasinthidwe kosokoneza DNS! Ndinapeza chida chakupha. Koma n’chifukwa chiyani zimenezi zikuchitika? Ndinafunikirabe cholinga.

Kukula koyambira kwa paketi ya DNS kumakonzedwa kudzera net.core.rmem_default. Mtengo wamba ndi penapake mozungulira 200KiB, koma ngati seva yanu ilandila mapaketi ambiri a DNS, mungafune kuwonjezera kukula kwa buffer. Ngati buffer ili yodzaza paketi yatsopano ikafika, mwachitsanzo chifukwa pulogalamuyo siyikuikonza mwachangu, ndiye kuti mudzayamba kutaya mapaketi. Makasitomala athu adawonjezera kukula kwa buffer chifukwa amawopa kutayika kwa data, popeza anali kugwiritsa ntchito pulogalamu yotolera ma metrics kudzera pamapaketi a DNS. Mtengo umene adakhazikitsa unali wotheka kwambiri: 231-1 (ngati itayikidwa ku 231, kernel idzabwerera "KUKHALA ZOSAVUTA").

Mwadzidzidzi ndidazindikira chifukwa chake nmap ndi scapy zidagwira ntchito moyenera: anali kugwiritsa ntchito zitsulo zosaphika! Miyendo yaiwisi ndi yosiyana ndi sockets wamba: amadutsa ma iptables, ndipo samasungidwa!

Koma chifukwa chiyani "buffer yayikulu kwambiri" imayambitsa mavuto? Mwachionekere sizigwira ntchito monga momwe anafunira.

Panthawiyi nditha kubweretsanso vuto pamakhola angapo komanso magawo angapo. Vuto linawonekera kale pa 3.x kernel ndipo tsopano linawonekeranso pa 5.x kernel.

Zoonadi, poyambira

sysctl -w net.core.rmem_default=$((2**31-1))

DNS inasiya kugwira ntchito.

Ndidayamba kuyang'ana magwiridwe antchito pogwiritsa ntchito njira yosavuta yosakira bayinare ndipo ndidapeza kuti makinawa amagwira ntchito ndi 2147481343, koma nambala iyi inali manambala opanda tanthauzo kwa ine. Ndinapempha kasitomala kuti ayesere nambala iyi, ndipo adayankha kuti dongosololi limagwira ntchito ndi google.com, komabe linapereka zolakwika ndi madera ena, kotero ndinapitiriza kufufuza kwanga.

Ndayika dropwatch, chida chomwe chimayenera kugwiritsidwa ntchito kale: chimasonyeza kumene mu kernel paketi imathera. Wolakwa anali ntchitoyo udp_queue_rcv_skb. Ndidatsitsa magwero a kernel ndikuwonjezera ochepa ntchito printk kuti muwone komwe paketiyo imathera. Ndinapeza mwamsanga chikhalidwe choyenera if, ndikungoyang'ana kwa nthawi ndithu, chifukwa ndi nthawi yomwe zonse zinafika pamodzi mu chithunzi chonse: 231-1, nambala yopanda tanthauzo, malo osagwira ntchito ... Inali chidutswa cha code mu __udp_enqueue_schedule_skb:

if (rmem > (size + sk->sk_rcvbuf))
		goto uncharge_drop;

Chonde dziwani:

• rmem ndi mtundu int
• size ndi yamtundu wa u16 (yosasainidwa sikisitini-bit int) ndipo imasunga kukula kwake
• sk->sk_rcybuf ndi yamtundu wa int ndikusunga kukula kwa buffer komwe, mwatanthauzo, ndi kofanana ndi mtengo net.core.rmem_default

pamene sk_rcvbuf ikuyandikira 231, kuwerengera kukula kwa paketi kungayambitse kuchuluka kusefukira. Ndipo popeza ndi int, mtengo wake umakhala wolakwika, ndiye kuti mkhalidwewo umakhala wowona ngati uyenera kukhala wabodza (mutha kuwerenga zambiri za izi pa kugwirizana).

Cholakwikacho chikhoza kuwongoleredwa mwanjira yocheperako: mwa kuponyera unsigned int. Ndinagwiritsa ntchito kukonza ndikuyambitsanso dongosolo ndipo DNS inagwiranso ntchito.

Kulawa kwa chigonjetso

Ndinatumiza zomwe ndapeza kwa kasitomala ndikutumiza LKML kernel chigamba. Ndine wokondwa: chidutswa chilichonse chazithunzi chimagwirizana, nditha kufotokoza ndendende chifukwa chomwe tidawonera zomwe tidawona, ndipo koposa zonse, tidatha kupeza yankho la vutoli chifukwa chantchito yathu yamagulu!

Ndikoyenera kuzindikira kuti mlanduwu udakhala wosowa, ndipo mwamwayi sitilandila zopempha zovuta zotere kuchokera kwa ogwiritsa ntchito.

Source: www.habr.com