Ib zaj dab neeg hais txog cov pob ntawv DNS uas ploj lawm los ntawm Google Cloud technical support

Los ntawm Google Blog Editor: Koj puas tau xav paub yuav ua li cas Google Cloud Technical Solutions (TSE) engineers tswj koj qhov kev thov txhawb nqa? TSE technical support engineers yog lub luag haujlwm rau kev txheeb xyuas thiab kho cov neeg siv qhia txog qhov teeb meem. Qee qhov teeb meem no yooj yim heev, tab sis qee zaum koj tuaj hla daim pib uas yuav tsum tau saib xyuas ntawm ntau tus engineers ib zaug. Hauv tsab xov xwm no, ib tus neeg ua haujlwm TSE yuav qhia peb txog ib qho teeb meem nyuaj heev los ntawm nws qhov kev coj ua tsis ntev los no - rooj plaub ntawm cov pob ntawv DNS uas ploj lawm. Hauv zaj dab neeg no, peb yuav pom tias cov engineers tswj tau li cas los daws qhov teeb meem no, thiab dab tsi tshiab uas lawv tau kawm thaum kho qhov yuam kev. Peb cia siab tias zaj dab neeg no tsis yog tsuas yog qhia koj txog kab mob sib sib zog nqus, tab sis kuj muab kev nkag siab rau koj txog cov txheej txheem uas nkag mus rau hauv daim ntawv them nyiaj yug nrog Google Cloud.

Kev daws teeb meem yog ob qho tib si science thiab kos duab. Nws tag nrho pib nrog kev tsim lub tswv yim hais txog qhov laj thawj rau kev coj tus cwj pwm tsis zoo ntawm lub cev, tom qab ntawd nws raug sim rau lub zog. Txawm li cas los xij, ua ntej peb tsim ib qho kev xav, peb yuav tsum qhia meej meej thiab meej tsim qhov teeb meem. Yog hais tias cov lus nug suab tsis meej, ces koj yuav tau soj ntsuam txhua yam kom zoo zoo; Qhov no yog "kos duab" ntawm kev daws teeb meem.

Raws li Google Huab, cov txheej txheem zoo li no dhau los ua qhov nyuaj dua, vim Google Cloud sim nws qhov zoo tshaj plaws los lav qhov kev ceev ntiag tug ntawm nws cov neeg siv. Vim li no, TSE engineers tsis muaj kev nkag mus rau kev hloov kho koj lub tshuab, thiab tsis muaj peev xwm saib cov kev teeb tsa dav dav li cov neeg siv ua. Yog li ntawd, txhawm rau sim ib qho ntawm peb cov kev xav, peb (cov kws tsim txuj ci) tsis tuaj yeem hloov kho sai sai.

Qee cov neeg siv ntseeg tias peb yuav kho txhua yam zoo li kev siv tshuab hauv kev pabcuam tsheb, thiab tsuas yog xa peb tus lej ntawm lub tshuab virtual, qhov tseeb, cov txheej txheem yuav tshwm sim hauv kev sib tham: sau cov ntaub ntawv, tsim thiab lees paub (lossis tsis lees paub) kev xav, thiab, thaum kawg, cov teeb meem kev txiav txim siab yog nyob ntawm kev sib txuas lus nrog tus neeg siv khoom.

Teeb meem nyob rau hauv nqe lus nug

Niaj hnub no peb muaj ib zaj dab neeg nrog qhov xaus zoo. Ib qho ntawm cov laj thawj rau kev daws teeb meem zoo ntawm rooj plaub uas tau hais tseg yog cov lus piav qhia ntxaws thiab meej ntawm qhov teeb meem. Hauv qab no koj tuaj yeem pom ib daim qauv ntawm thawj daim pib (kho kho kom zais cov ntaub ntawv tsis pub lwm tus paub):

Cov lus no muaj ntau cov ntaub ntawv tseem ceeb rau peb:

• Specific VM tau teev tseg
• Qhov teeb meem nws tus kheej tau qhia - DNS tsis ua haujlwm
• Nws yog qhia qhov twg qhov teeb meem tshwm sim nws tus kheej - VM thiab thawv
• Cov kauj ruam uas tus neeg siv tau coj los txheeb xyuas qhov teeb meem tau qhia.

Qhov kev thov tau sau npe ua "P1: Kev cuam tshuam tseem ceeb - Kev pabcuam tsis siv hauv kev tsim khoom", uas txhais tau tias kev saib xyuas tas li ntawm qhov xwm txheej 24/7 raws li "Ua raws Lub Hnub" (koj tuaj yeem nyeem ntxiv txog qhov tseem ceeb ntawm cov neeg siv kev thov), nrog nws cov kev hloov ntawm ib pab neeg txhawb nqa mus rau lwm tus nrog txhua lub sijhawm ua haujlwm. Qhov tseeb, los ntawm lub sij hawm qhov teeb meem mus txog peb pab neeg nyob rau hauv Zurich, nws twb twb ncig lub ntiaj teb no. Los ntawm lub sij hawm no, tus neeg siv tau siv kev txo qis, tab sis ntshai ntawm qhov rov ua dua ntawm qhov xwm txheej hauv kev tsim khoom, txij li lub hauv paus ua rau tseem tsis tau pom.

Thaum lub sijhawm daim pib mus txog Zurich, peb twb muaj cov ntaub ntawv hauv qab no ntawm tes:

• Cov ntsiab lus /etc/hosts
• Cov ntsiab lus /etc/resolv.conf
• xaus iptables-save
• Ua ke los ntawm pab neeg ngrep pcap ua

Nrog rau cov ntaub ntawv no, peb tau npaj los pib "kev tshawb nrhiav" thiab kev daws teeb meem.

Peb thawj kauj ruam

Ua ntej tshaj plaws, peb tau tshawb xyuas cov cav thiab cov xwm txheej ntawm metadata server thiab xyuas kom meej tias nws ua haujlwm raug. Tus neeg rau zaub mov metadata teb rau IP chaw nyob 169.254.169.254 thiab, ntawm lwm yam, yog lub luag haujlwm rau kev tswj cov npe sau npe. Peb kuj tau kuaj xyuas ob zaug tias lub firewall ua haujlwm raug nrog VM thiab tsis thaiv cov pob ntawv.

Nws yog qee yam teeb meem coj txawv txawv: nmap kos tsis lees paub peb lub ntsiab lus xav txog qhov poob ntawm UDP pob ntawv, yog li peb lub hlwb tuaj nrog ntau txoj kev xaiv thiab kev kuaj xyuas lawv:

• Cov pob ntawv puas raug xaiv? => Tshawb xyuas iptables cov cai
• Tsis yog nws me dhau lawm? TUS NEEG? => Tshawb xyuas cov zis ip a show
• Puas yog qhov teeb meem cuam tshuam tsuas yog UDP pob ntawv lossis TCP ib yam nkaus? => Tsav tsheb mus dig +tcp
• Puas yog cov pob khoom tsim tawm rov qab? => Tsav tsheb mus tcpdump
• Puas yog libdns ua haujlwm raug? => Tsav tsheb mus strace txhawm rau txheeb xyuas kev sib kis ntawm cov pob ntawv hauv ob qho tib si

Ntawm no peb txiav txim siab hu rau tus neeg siv los daws teeb meem nyob.

Thaum hu peb tuaj yeem tshawb xyuas ntau yam:

• Tom qab ob peb daim tshev peb cais cov cai iptables los ntawm cov npe ntawm cov laj thawj
• Peb tshawb xyuas network interfaces thiab routing tables, thiab xyuas ob zaug tias MTU yog lawm
• Peb nrhiav tau qhov ntawd dig +tcp google.com (TCP) ua haujlwm raws li nws yuav tsum, tab sis dig google.com (UDP) tsis ua haujlwm
• Tau tsav tsheb mus tcpdump nws tseem ua haujlwm dig, peb pom tias cov pob ntawv UDP raug xa rov qab
• Peb tsav tsheb mus strace dig google.com thiab peb pom li cas khawb kom raug hu sendmsg() и recvms(), txawm li cas los xij qhov thib ob yog cuam tshuam los ntawm lub sijhawm

Hmoov tsis zoo, qhov kawg ntawm kev hloov pauv tuaj txog thiab peb raug yuam kom nce qhov teeb meem mus rau lub sijhawm tom ntej. Qhov kev thov, txawm li cas los xij, ua rau muaj kev txaus siab rau peb pab neeg, thiab cov npoj yaig qhia tias tsim cov pob DNS thawj zaug siv cov khoom siv Python scrapy.

from scapy.all import *

answer = sr1(IP(dst="169.254.169.254")/UDP(dport=53)/DNS(rd=1,qd=DNSQR(qname="google.com")),verbose=0)
print ("169.254.169.254", answer[DNS].summary())

Qhov tawg no tsim ib pob ntawv DNS thiab xa cov lus thov mus rau metadata server.

Tus neeg siv khiav cov cai, cov lus teb DNS raug xa rov qab, thiab daim ntawv thov tau txais nws, lees paub tias tsis muaj teeb meem ntawm qib network.

Tom qab lwm qhov "ib puag ncig-lub ntiaj teb mus txawv tebchaws," qhov kev thov rov qab los rau peb pab neeg, thiab kuv tau hloov tag nrho rau kuv tus kheej, xav tias nws yuav yooj yim dua rau cov neeg siv yog tias qhov kev thov nres ncig ntawm qhov chaw mus rau qhov chaw.

Nyob rau hauv lub meantime, tus neeg siv siab zoo pom zoo muab ib tug snapshot ntawm lub system duab. Qhov no yog cov xov xwm zoo heev: lub peev xwm los ntsuas qhov system kuv tus kheej ua teeb meem sai dua, vim tias kuv tsis tas yuav nug tus neeg siv los khiav cov lus txib, xa kuv cov txiaj ntsig thiab txheeb xyuas lawv, Kuv tuaj yeem ua txhua yam ntawm kuv tus kheej!

Kuv cov npoj yaig tab tom pib khib kuv me ntsis. Thaum noj su peb tham txog kev hloov dua siab tshiab, tab sis tsis muaj leej twg muaj lub tswv yim dab tsi tshwm sim. Hmoov zoo, tus neeg siv nws tus kheej twb tau ntsuas los txo qhov tshwm sim thiab tsis nrawm, yog li peb muaj sijhawm los txiav txim siab qhov teeb meem. Thiab txij li thaum peb muaj duab, peb tuaj yeem khiav txhua qhov kev sim uas peb nyiam. Zoo heev!

Coj ib kauj ruam rov qab

Ib qho ntawm cov lus nug nrov tshaj plaws hauv kev xam phaj rau cov haujlwm engineers yog: "Yuav ua li cas thaum koj ping www.google.com? Cov lus nug zoo heev, txij li tus neeg sib tw yuav tsum tau piav qhia txhua yam los ntawm lub plhaub rau cov neeg siv qhov chaw, mus rau cov kab ke system thiab tom qab ntawd mus rau lub network. Kuv luag ntxhi: qee zaum kev xam phaj cov lus nug tig tawm kom muaj txiaj ntsig hauv lub neej tiag tiag ...

Kuv txiav txim siab siv lo lus nug HR no rau qhov teeb meem tam sim no. Hais lus ntxhib, thaum koj sim txiav txim siab lub npe DNS, cov hauv qab no tshwm sim:

1. Daim ntawv thov hu rau lub tsev qiv ntawv zoo li libdns
2. libdns tshawb xyuas qhov system teeb tsa uas DNS server nws yuav tsum tiv tauj (hauv daim duab no yog 169.254.169.254, metadata server)
3. libdns siv lub kaw lus hu los tsim UDP qhov (SOKET_DGRAM) thiab xa UDP pob ntawv nrog cov lus nug DNS hauv ob qho tib si
4. Los ntawm sysctl interface koj tuaj yeem teeb tsa UDP pawg ntawm qib ntsiav
5. Lub kernel cuam tshuam nrog cov khoom siv kom xa cov pob ntawv hla lub network ntawm lub network interface
6. Lub hypervisor ntes thiab xa cov pob ntawv mus rau metadata server thaum sib cuag nrog nws
7. Tus neeg rau zaub mov metadata, los ntawm nws cov khawv koob, txiav txim siab lub npe DNS thiab xa cov lus teb siv tib txoj kev

Cia kuv ceeb toom rau koj tias qhov kev xav uas peb twb tau txiav txim siab:

Hypothesis: Cov tsev qiv ntawv tawg

• Kuaj 1: khiav txoj kab hauv qhov system, xyuas tias khawb hu rau qhov system hu kom raug
• Qhov tshwm sim: raug hu kom raug hu
• Test 2: siv srapy los xyuas seb peb puas tuaj yeem txiav txim siab cov npe hla cov tsev qiv ntawv
• Qhov tshwm sim: peb ua tau
• Kuaj 3: khiav rpm –V ntawm libdns pob thiab md5sum cov ntaub ntawv qiv
• Qhov tshwm sim: lub tsev qiv ntawv code yog tag nrho zoo tib yam rau cov cai hauv kev ua haujlwm ua haujlwm
• Kuaj 4: mount tus neeg siv lub hauv paus system duab ntawm VM yam tsis muaj tus cwj pwm no, khiav chroot, saib seb DNS ua haujlwm
• Cov txiaj ntsig: DNS ua haujlwm raug

Cov lus xaus raws li kev xeem: qhov teeb meem tsis nyob hauv cov tsev qiv ntawv

Hypothesis: Muaj qhov yuam kev hauv DNS nqis

• Kuaj 1: xyuas tcpdump thiab saib seb cov pob ntawv DNS raug xa thiab xa rov qab kom raug tom qab khiav khawb
• Qhov tshwm sim: pob ntawv raug xa mus kom raug
• Kuaj 2: ob zaug kos rau ntawm lub server /etc/nsswitch.conf и /etc/resolv.conf
• Qhov tshwm sim: txhua yam yog lawm

Cov lus xaus raws li kev xeem: qhov teeb meem tsis yog nrog DNS configuration

Hypothesis: core puas

• Kuaj: nruab kernel tshiab, kos npe kos npe, rov pib dua
• Qhov tshwm sim: tus cwj pwm zoo sib xws

Cov lus xaus raws li kev xeem: lub kernel tsis puas

Hypothesis: tus cwj pwm tsis raug ntawm cov neeg siv network (lossis hypervisor network interface)

• Kuaj 1: Tshawb xyuas koj qhov teeb tsa firewall
• Qhov tshwm sim: firewall hla DNS pob ntawv ntawm ob tus tswv tsev thiab GCP
• Xeem 2: cuam tshuam kev khiav tsheb thiab saib xyuas qhov tseeb ntawm kev sib kis thiab xa rov qab ntawm DNS thov
• Qhov tshwm sim: tcpdump lees paub tias tus tswv tsev tau txais cov pob ntawv rov qab

Cov lus xaus raws li kev xeem: qhov teeb meem tsis nyob hauv lub network

Hypothesis: metadata server tsis ua haujlwm

• Xeem 1: xyuas cov metadata neeg rau zaub mov cav rau anomalies
• Qhov tshwm sim: tsis muaj qhov txawv txav hauv cov cav
• Test 2: Bypass lub metadata server ntawm dig @8.8.8.8
• Qhov tshwm sim: Kev daws teeb meem tawg txawm tias tsis siv lub server metadata

Cov lus xaus raws li kev xeem: qhov teeb meem tsis yog nrog cov metadata server

Cov kab hauv qab: peb sim tag nrho cov subsystems tshwj tsis yog runtime nqis!

Nkag mus rau hauv Kernel Runtime Settings

Txhawm rau teeb tsa lub kernel ua tiav ib puag ncig, koj tuaj yeem siv cov kab lus hais kom ua (grub) lossis sysctl interface. Kuv ntsia rau hauv /etc/sysctl.conf thiab cia li xav tias, kuv nrhiav tau ob peb qhov kev cai. Xav zoo li kuv tau rub mus rau ib yam dab tsi, kuv muab pov tseg tag nrho cov chaw uas tsis yog-network lossis tsis-tcp, tshuav nrog cov chaw roob net.core. Tom qab ntawd kuv tau mus rau qhov chaw tso cai tus tswv tsev nyob hauv VM thiab pib siv cov kev teeb tsa ib qho los ntawm ib qho, ib qho tom qab, nrog VM tawg, txog thaum kuv pom tus neeg ua txhaum:

net.core.rmem_default = 2147483647

Ntawm no nws yog, DNS-tawg teeb tsa! Kuv nrhiav tau riam phom tua neeg. Tab sis yog vim li cas qhov no tshwm sim? Kuv tseem xav tau kev txhawb siab.

Qhov yooj yim DNS pob ntawv tsis loj yog teeb tsa ntawm net.core.rmem_default. Tus nqi ib txwm nyob ib puag ncig 200KiB, tab sis yog tias koj lub server tau txais ntau cov pob ntawv DNS, koj tuaj yeem xav nce qhov tsis zoo. Yog tias qhov tsis muaj puv thaum lub pob ntawv tshiab tuaj txog, piv txwv li vim tias daim ntawv thov tsis ua haujlwm sai txaus, ces koj yuav pib poob pob ntawv. Peb cov neeg siv khoom tau nce qhov tsis zoo vim nws ntshai cov ntaub ntawv poob, txij li nws tau siv daim ntawv thov sau cov metrics los ntawm cov pob ntawv DNS. Tus nqi nws teev yog qhov siab tshaj plaws tau: 231-1 (yog tias teem rau 231, lub ntsiav yuav rov qab "Kev Sib Tham Tsis Zoo").

Tam sim ntawd kuv pom tau tias yog vim li cas nmap thiab scapy ua haujlwm kom raug: lawv tau siv cov khoom siv raw! Raw sockets txawv ntawm cov qhov (socket) li niaj zaus: lawv hla iptables, thiab lawv tsis buffered!

Tab sis vim li cas "tsis loj dhau" ua rau muaj teeb meem? Nws kom meej meej tsis ua hauj lwm raws li npaj.

Ntawm no kuv tuaj yeem tsim qhov teeb meem ntawm ntau lub kernels thiab ntau qhov kev faib tawm. Qhov teeb meem twb tshwm sim ntawm 3.x ntsiav thiab tam sim no nws kuj tshwm sim ntawm 5.x kernel.

Tseeb, thaum pib

sysctl -w net.core.rmem_default=$((2**31-1))

DNS nres ua haujlwm.

Kuv pib nrhiav kev ua haujlwm tseem ceeb los ntawm kev tshawb nrhiav binary yooj yooj yim thiab pom tias lub kaw lus ua haujlwm nrog 2147481343, tab sis tus lej no yog cov lej tsis muaj txiaj ntsig rau kuv. Kuv tau hais kom tus neeg siv khoom sim tus lej no, thiab nws teb tias lub kaw lus ua haujlwm nrog google.com, tab sis tseem muab qhov yuam kev nrog lwm tus thawj, yog li kuv txuas ntxiv kuv qhov kev tshawb nrhiav.

Kuv tau nruab dropwatch, ib qho cuab yeej uas yuav tsum tau siv ua ntej: nws qhia tau hais tias qhov twg hauv cov ntsiav ib pob ntawv xaus. Tus neeg ua txhaum yog lub luag haujlwm udp_queue_rcv_skb. Kuv rub tawm cov kernel qhov chaw thiab ntxiv ob peb zog printk taug qab qhov twg raws nraim lub pob ntawv xaus. Kuv pom qhov xwm txheej sai sai if, thiab tsuas yog ntsia nws rau qee lub sijhawm, vim tias nws yog thaum kawg txhua yam tuaj ua ke rau hauv daim duab: 231-1, tus lej tsis muaj qab hau, tsis ua haujlwm ... Nws yog ib qho ntawm cov lej hauv __udp_enqueue_schedule_skb:

if (rmem > (size + sk->sk_rcvbuf))
		goto uncharge_drop;

Thov nco ntsoov:

• rmem yog type int
• size yog hom u16 (tsis tau kos npe kaum rau-ntsis int) thiab khaws cov pob ntawv loj
• sk->sk_rcybuf yog hom int thiab khaws cia qhov loj me uas, los ntawm kev txhais, yog sib npaug rau tus nqi hauv net.core.rmem_default

Thaum sk_rcvbuf mus txog 231, suav nrog cov pob ntawv loj tuaj yeem ua rau integer overflow. Thiab txij li nws yog ib qho int, nws tus nqi yuav tsis zoo, yog li qhov xwm txheej yuav muaj tseeb thaum nws yuav tsum tsis tseeb (koj tuaj yeem nyeem ntxiv txog qhov no ntawm txuas).

Qhov yuam kev tuaj yeem raug kho nyob rau hauv txoj kev tsis tseem ceeb: los ntawm casting unsigned int. Kuv tau siv qhov kho thiab rov pib lub system thiab DNS ua haujlwm dua.

saj ntawm yeej

Kuv xa kuv qhov kev tshawb pom rau tus neeg siv khoom thiab xa mus LKML kernel thaj. Kuv txaus siab: txhua daim duab dhos ua ke haum ua ke, Kuv tuaj yeem piav qhia meej vim li cas peb tau soj ntsuam qhov peb tau pom, thiab qhov tseem ceeb tshaj, peb tuaj yeem nrhiav kev daws teeb meem ua tsaug rau peb pab pawg!

Nws yog ib qho tsim nyog lees paub tias qhov teeb meem tau tshwm sim tsis tshua muaj, thiab hmoov zoo peb tsis tshua tau txais cov kev thov nyuaj los ntawm cov neeg siv.

Tau qhov twg los: www.hab.com