🥇 Ceev faj ntawm qhov tsis zoo uas ua rau muaj kev sib tw ua haujlwm. Part 1: FragmentSmack/SegmentSmack

Nyob zoo sawv daws! Kuv lub npe yog Dmitry Samsonov, Kuv ua haujlwm ua tus thawj coj hauv Odnoklassniki. Peb muaj ntau tshaj 7 txhiab lub cev servers, 11 txhiab lub thawv hauv peb cov huab thiab 200 daim ntawv thov, uas nyob rau hauv ntau yam configurations tsim 700 pawg sib txawv. Feem ntau ntawm cov servers khiav CentOS 7.
Thaum Lub Yim Hli 14, 2018, cov ntaub ntawv hais txog FragmentSmack qhov tsis zoo tau tshaj tawm
(CVE-2018-5391) thiab SegmentSmack (CVE-2018-5390). Cov no yog qhov tsis zoo nrog lub network tawm tsam vector thiab cov qhab nia siab (7.5), uas ua rau tsis kam lees txais kev pabcuam (DoS) vim muaj peev txheej txaus (CPU). Kev kho cov ntsiav rau FragmentSmack tsis tau npaj rau lub sijhawm ntawd; Ntxiv mus, nws tawm ntau tom qab tshaj tawm cov ntaub ntawv hais txog qhov tsis zoo. Txhawm rau tshem tawm SegmentSmack, nws tau hais kom hloov kho cov ntsiav. Lub pob hloov tshiab nws tus kheej tau tso tawm rau tib hnub, txhua yam uas tseem tshuav yog los nruab nws.
Tsis yog, peb tsis tawm tsam hloov kho cov ntsiav txhua! Txawm li cas los xij, muaj cov nuances ...

Yuav ua li cas peb hloov kho lub kernel ntawm ntau lawm

Feem ntau, tsis muaj dab tsi nyuab:

Download pob ntawv;
Nruab lawv ntawm ntau lub servers (xws li servers hosting peb huab);
Xyuas kom tsis muaj dab tsi tawg;
Nco ntsoov tias tag nrho cov txheej txheem kernel raug siv yam tsis muaj qhov yuam kev;
Tos ob peb hnub;
Tshawb xyuas kev ua haujlwm ntawm server;
Hloov kev xa tawm ntawm cov servers tshiab rau cov kernel tshiab;
Hloov kho tag nrho cov servers los ntawm cov chaw khaws ntaub ntawv (ib lub chaw khaws ntaub ntawv ntawm ib lub sijhawm kom txo qis rau cov neeg siv thaum muaj teeb meem);
Reboot tag nrho cov servers.

Rov ua dua rau txhua ceg ntawm cov kernels peb muaj. Tam sim no nws yog:

Tshuag CentOS 7 3.10 - rau feem ntau cov servers;
Vanilla 4.19 - rau peb ib huab cua, vim peb xav tau BFQ, BBR, thiab lwm yam.;
Elrepo kernel-ml 5.2 - rau heev loaded distributors, vim 4.19 siv los coj tus cwj pwm tsis ruaj khov, tab sis cov yam ntxwv zoo ib yam xav tau.

Raws li koj tau twv, rebooting txhiab tus servers yuav siv sij hawm ntev tshaj plaws. Txij li tsis yog txhua qhov tsis muaj qhov tsis zoo yog qhov tseem ceeb rau txhua tus servers, peb tsuas yog reboot cov uas tuaj yeem siv ncaj qha los ntawm Is Taws Nem. Hauv huab, txhawm rau kom tsis txhob txwv kev hloov pauv, peb tsis khi cov thawv ntim khoom siv sab nraud rau cov neeg siv khoom siv nrog cov ntsiav tshiab, tab sis rov pib dua txhua lub tswv yim yam tsis muaj kev zam. Hmoov zoo, cov txheej txheem muaj yooj yim dua nrog cov servers li niaj zaus. Piv txwv li, cov thawv tsis muaj lub xeev tuaj yeem txav mus rau lwm tus neeg rau zaub mov thaum lub sijhawm rov pib dua.

Txawm li cas los xij, tseem muaj ntau txoj haujlwm, thiab nws tuaj yeem siv sijhawm ob peb lub lis piam, thiab yog tias muaj teeb meem nrog cov tshiab version, mus txog ob peb lub hlis. Cov neeg tawm tsam nkag siab qhov no zoo heev, yog li lawv xav tau txoj kev npaj B.

FragmentSmack/SegmentSmack. Kev daws teeb meem

Hmoov zoo, rau qee qhov tsis zoo xws li txoj kev npaj B muaj, thiab nws hu ua Workaround. Feem ntau, qhov no yog qhov kev hloov pauv hauv cov ntsiav / daim ntawv thov chaw uas tuaj yeem txo qhov ua tau zoo lossis tshem tawm tag nrho cov kev siv ntawm qhov tsis zoo.

Nyob rau hauv rooj plaub ntawm FragmentSmack/SegmentSmack tau thov Qhov kev daws teeb meem no:

«Koj tuaj yeem hloov pauv qhov tseem ceeb ntawm 4MB thiab 3MB hauv net.ipv4.ipfrag_high_thresh thiab net.ipv4.ipfrag_low_thresh (thiab lawv cov counterparts rau ipv6 net.ipv6.ipfrag_high_thresh thiab net.ipv6.ipfrag_low_thresh) rau 256 thiab 192 qis. Kev ntsuam xyuas qhia me me rau qhov poob qis hauv CPU siv thaum muaj kev tawm tsam nyob ntawm kev kho vajtse, chaw, thiab tej yam kev mob. Txawm li cas los xij, tej zaum yuav muaj qee qhov kev ua tau zoo vim yog ipfrag_high_thresh = 262144 bytes, vim tsuas yog ob qho 64K tawg tuaj yeem haum rau hauv cov kab rov ua dua ib zaug. Piv txwv li, muaj kev pheej hmoo tias cov ntawv thov ua haujlwm nrog cov pob ntawv UDP loj yuav tawg".

Cov parameter lawv tus kheej nyob rau hauv cov ntaub ntawv kernel piav raws li nram no:

ipfrag_high_thresh - LONG INTEGER Maximum memory used to reassemble IP fragments.

ipfrag_low_thresh - LONG INTEGER Maximum memory used to reassemble IP fragments before the kernel begins to remove incomplete fragment queues to free up resources. The kernel still accepts new fragments for defragmentation.

Peb tsis muaj UDPs loj ntawm cov kev pabcuam tsim khoom. Tsis muaj kev sib cais ntawm LAN; muaj kev sib cais ntawm WAN, tab sis tsis tseem ceeb. Tsis muaj cov cim qhia - koj tuaj yeem yob tawm Workaround!

FragmentSmack/SegmentSmack. Thawj ntshav

Thawj qhov teeb meem peb tau ntsib yog tias huab ntim qee zaum siv cov chaw tshiab tsuas yog qee qhov (tsuas yog ipfrag_low_thresh), thiab qee zaum tsis siv lawv txhua - lawv tsuas yog poob thaum pib. Nws tsis tuaj yeem rov tsim qhov teeb meem ruaj khov (txhua qhov chaw tau siv manually yam tsis muaj teeb meem). Kev nkag siab yog vim li cas lub thawv sib tsoo thaum pib kuj tsis yooj yim: tsis pom qhov yuam kev. Ib yam yog qhov tseeb: dov rov qab cov chaw daws teeb meem nrog cov thawv sib tsoo.

Vim li cas nws tsis txaus siv Sysctl ntawm tus tswv tsev? Lub thawv nyob hauv nws tus kheej lub network Namespace, yog li tsawg kawg ib feem ntawm lub network Sysctl parameters nyob rau hauv lub thawv yuav txawv ntawm tus tswv tsev.

Yuav ua li cas raws nraim Sysctl nqis siv rau hauv lub thawv? Txij li thaum peb cov thawv ntim khoom tsis tsim nyog, koj yuav tsis tuaj yeem hloov qhov chaw Sysctl los ntawm kev nkag mus rau hauv lub thawv nws tus kheej - koj tsuas yog tsis muaj cai txaus. Txhawm rau khiav cov thawv, peb huab thaum lub sijhawm ntawd siv Docker (tam sim no podman). Cov kev txwv ntawm lub thawv tshiab tau dhau mus rau Docker ntawm API, suav nrog qhov tsim nyog Sysctl chaw.
Thaum tshawb nrhiav los ntawm cov qauv, nws tau pom tias Docker API tsis rov qab txhua qhov yuam kev (tsawg kawg hauv version 1.10). Thaum peb sim pib lub thawv ntawm "docker khiav", peb thaum kawg pom yam tsawg kawg nkaus:

write /proc/sys/net/ipv4/ipfrag_high_thresh: invalid argument docker: Error response from daemon: Cannot start container <...>: [9] System error: could not synchronise with container process.

Tus nqi parameter tsis siv tau. Tab sis vim li cas? Thiab yog vim li cas nws tsis siv tau qee zaum? Nws muab tawm tias Docker tsis tau lees tias qhov kev txiav txim uas Sysctl tsis tau thov (qhov kev sim tshiab kawg yog 1.13.1), yog li qee zaum ipfrag_high_thresh tau sim teem rau 256K thaum ipfrag_low_thresh tseem yog 3M, uas yog, qhov txwv sab saud yog qis dua. tshaj qhov txwv qis, uas ua rau qhov yuam kev.

Lub sijhawm ntawd, peb twb tau siv peb tus kheej cov txheej txheem rau kev teeb tsa lub thawv tom qab pib (khov lub thawv tom qab pawg freezer thiab executing commands nyob rau hauv lub namespace ntawm lub thawv ntawm ip nuj), thiab peb kuj tau ntxiv kev sau ntawv Sysctl tsis pub dhau qhov no. Qhov teeb meem raug daws.

FragmentSmack/SegmentSmack. Thawj Ntshav 2

Ua ntej peb muaj sijhawm los nkag siab txog kev siv Workaround hauv huab, thawj qhov tsis txaus siab los ntawm cov neeg siv pib tuaj txog. Lub sijhawm ntawd, ob peb lub lis piam dhau los txij li thaum pib siv Workaround ntawm thawj servers. Qhov kev tshawb nrhiav thawj zaug tau pom tias cov kev tsis txaus siab tau txais tawm tsam ib tus neeg pabcuam, thiab tsis yog txhua tus servers ntawm cov kev pabcuam no. Qhov teeb meem tau dhau los ua qhov tsis meej heev.

Ua ntej ntawm tag nrho cov, peb, ntawm chav kawm, sim yob rov qab Sysctl chaw, tab sis qhov no tsis muaj kev cuam tshuam. Ntau yam kev ua haujlwm nrog cov neeg rau zaub mov thiab cov ntawv thov kev teeb tsa tsis tau pab ib yam. Reboot pab. Rebooting Linux yog qhov tsis zoo li nws yog ib txwm muaj rau Windows hauv cov hnub qub. Txawm li cas los xij, nws tau pab, thiab peb tau chalk nws mus txog "kernel glitch" thaum siv cov chaw tshiab hauv Sysctl. Yuav ua li cas frivolous nws ...

Peb lub lis piam tom qab qhov teeb meem rov tshwm sim. Kev teeb tsa ntawm cov servers no yooj yim heev: Nginx hauv hom npe / ntsuas ntsuas. Tsis muaj tsheb ntau. Cov lus qhia tshiab: tus naj npawb ntawm 504 yuam kev ntawm cov neeg siv khoom tau nce txhua hnub (Lub Sijhawm Gateway). Daim duab qhia tus naj npawb ntawm 504 yuam kev hauv ib hnub rau qhov kev pabcuam no:

Txhua qhov yuam kev yog hais txog tib lub backend - txog ib qho uas nyob hauv huab. Lub cim xeeb noj graph rau pob tawg ntawm qhov backend zoo li no:

Qhov no yog ib qho ntawm qhov pom tseeb tshaj plaws ntawm qhov teeb meem hauv kev khiav hauj lwm qhov system graphs. Hauv huab, tib lub sijhawm, lwm qhov teeb meem network nrog QoS (Traffic Control) tau kho. Ntawm daim duab ntawm kev nco noj rau pob ntawv tawg, nws zoo ib yam nkaus:

Qhov kev xav tau yooj yim: yog tias lawv zoo ib yam ntawm cov duab, ces lawv muaj qhov laj thawj zoo ib yam. Ntxiv mus, txhua yam teeb meem nrog hom kev nco no tsis tshua muaj neeg.

Lub ntsiab lus ntawm qhov teeb meem ruaj khov yog tias peb siv fq pob ntawv teem caij nrog qhov teeb meem nyob hauv QoS. Los ntawm lub neej ntawd, rau ib qho kev sib txuas, nws tso cai rau koj ntxiv 100 pob ntawv rau hauv kab, thiab qee qhov kev sib txuas, nyob rau hauv cov xwm txheej ntawm kev tsis txaus channel, pib txhaws cov kab rau lub peev xwm. Nyob rau hauv cov ntaub ntawv no, pob ntawv poob lawm. Hauv tc txheeb cais (tc -s qdisc) nws tuaj yeem pom zoo li no:

qdisc fq 2c6c: parent 1:2c6c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms Sent 454701676345 bytes 491683359 pkt (dropped 464545, overlimits 0 requeues 0) backlog 0b 0p requeues 0 1024 flows (1021 inactive, 0 throttled) 0 gc, 0 highprio, 0 throttled, 464545 flows_plimit

"464545 flows_plimit" yog cov pob ntawv poob vim dhau qhov kev txwv tsis pub dhau ntawm ib qho kev sib txuas, thiab "poob 464545" yog qhov suav ntawm tag nrho cov pob ntawv poob ntawm lub sijhawm teem caij no. Tom qab nce cov kab ntev mus rau 1 txhiab thiab rov pib lub ntim, qhov teeb meem tsis tshwm sim. Koj tuaj yeem zaum rov qab thiab haus ib lub smoothie.

FragmentSmack/SegmentSmack. Ntshav kawg

Ua ntej, ob peb lub hlis tom qab tshaj tawm ntawm qhov tsis zoo hauv cov ntsiav, kev kho rau FragmentSmack thaum kawg tshwm sim (cia kuv nco ntsoov tias nrog rau kev tshaj tawm thaum Lub Yim Hli, kev kho tsuas yog rau SegmentSmack tau tso tawm), uas tau muab sijhawm rau peb tso tseg Workaround, uas ua rau peb muaj teeb meem ntau heev. Lub sijhawm no, peb twb tau tswj kom hloov qee cov servers mus rau cov ntsiav tshiab, thiab tam sim no peb yuav tsum pib txij thaum pib. Vim li cas peb thiaj li hloov kho cov ntsiav yam tsis tau tos FragmentSmack kho? Qhov tseeb yog tias cov txheej txheem ntawm kev tiv thaiv tawm tsam cov kev tsis zoo no tau sib koom ua ke (thiab sib koom ua ke) nrog cov txheej txheem ntawm kev hloov kho CentOS nws tus kheej (uas yuav siv sijhawm ntau dua li hloov kho cov ntsiav xwb). Tsis tas li ntawd, SegmentSmack yog qhov muaj kev phom sij txaus ntshai, thiab kho rau nws tshwm sim tam sim ntawd, yog li nws ua rau muaj kev nkag siab zoo li cas. Txawm li cas los xij, peb tsis tuaj yeem hloov kho cov ntsiav ntawm CentOS vim tias FragmentSmack qhov tsis zoo, uas tshwm sim thaum CentOS 7.5, tsuas yog kho hauv version 7.6, yog li peb yuav tsum nres qhov hloov tshiab rau 7.5 thiab pib dua nrog kev hloov tshiab rau 7.6. Thiab qhov no kuj tshwm sim.

Qhov thib ob, tsis tshua muaj neeg siv tsis txaus siab txog teeb meem tau rov qab los rau peb. Tam sim no peb twb paub tseeb tias lawv txhua tus muaj feem cuam tshuam nrog upload cov ntaub ntawv los ntawm cov neeg siv khoom mus rau qee qhov ntawm peb cov servers. Ntxiv mus, tsawg heev ntawm uploads los ntawm tag nrho cov huab hwm coj mus los ntawm cov servers.

Raws li peb nco qab los ntawm zaj dab neeg saum toj no, dov rov qab Sysctl tsis pab. Reboot pab tau, tab sis ib ntus.
Kev tsis ntseeg txog Sysctl tsis raug tshem tawm, tab sis lub sijhawm no nws yog qhov tsim nyog los sau cov ntaub ntawv ntau li ntau tau. Kuj tseem muaj qhov tsis muaj peev xwm los tsim cov teeb meem upload ntawm tus neeg siv khoom txhawm rau kawm kom paub meej tias qhov tshwm sim tau tshwm sim.

Kev tshuaj xyuas ntawm txhua qhov kev txheeb cais thiab cov ntawv teev tseg tsis tau coj peb los ze zog rau kev nkag siab txog qhov tshwm sim. Muaj qhov tsis muaj peev xwm rov tsim dua qhov teeb meem kom "xav" qhov kev sib txuas tshwj xeeb. Thaum kawg, cov neeg tsim khoom, siv cov ntawv tshwj xeeb ntawm daim ntawv thov, tswj kom ua tiav cov kev tsim tawm ruaj khov ntawm cov teeb meem ntawm cov cuab yeej ntsuas thaum txuas ntawm Wi-Fi. Qhov no yog ib qho kev tawg ntawm kev tshawb nrhiav. Tus neeg siv khoom txuas nrog Nginx, uas tso cai rau lub backend, uas yog peb daim ntawv thov Java.

Kev sib tham rau cov teeb meem yog li no (tso rau ntawm Nginx proxy sab):

Client: thov kom tau txais cov ntaub ntawv hais txog kev rub tawm cov ntaub ntawv.
Java server: teb.
Client: POST nrog cov ntaub ntawv.
Java server: yuam kev.

Nyob rau tib lub sijhawm, Java server sau rau lub cav uas 0 bytes ntawm cov ntaub ntawv tau txais los ntawm tus neeg siv khoom, thiab Nginx npe sau tias qhov kev thov siv sijhawm ntau dua 30 vib nas this (30 vib nas this yog lub sijhawm ntawm cov neeg siv khoom thov). Vim li cas lub sij hawm tawm thiab vim li cas 0 bytes? Los ntawm qhov kev xav HTTP, txhua yam ua haujlwm raws li nws yuav tsum tau, tab sis POST nrog cov ntaub ntawv zoo li ploj ntawm lub network. Ntxiv mus, nws ploj ntawm tus neeg siv khoom thiab Nginx. Nws yog lub sijhawm los tuav koj tus kheej nrog Tcpdump! Tab sis ua ntej koj yuav tsum to taub lub network configuration. Nginx npe yog tom qab L3 balancer NFware. Tunneling yog siv los xa cov pob ntawv los ntawm L3 balancer mus rau tus neeg rau zaub mov, uas ntxiv nws cov headers rau cov pob ntawv:

Nyob rau hauv cov ntaub ntawv no, lub network los rau lub server no nyob rau hauv daim ntawv ntawm Vlan-tagged tsheb, uas kuj ntxiv nws tus kheej teb rau cov pob ntawv:

Thiab qhov kev khiav tsheb no kuj tuaj yeem tawg tau (qhov feem pua me me ntawm cov tsheb sib tsoo uas peb tau tham txog thaum ntsuas qhov kev pheej hmoo los ntawm Workaround), uas kuj hloov cov ntsiab lus ntawm cov headers:

Ib zaug ntxiv: pob ntawv tau ntim nrog Vlan tag, encapsulated nrog lub qhov, fragmented. Yuav kom nkag siab zoo dua yuav ua li cas qhov no tshwm sim, cia peb taug qab cov pob ntawv txoj kev los ntawm tus neeg siv khoom mus rau Nginx npe.

Cov pob ntawv mus txog L3 balancer. Rau txoj kev raug cai nyob rau hauv lub chaw cov ntaub ntawv, lub pob ntawv yog encapsulated nyob rau hauv ib lub qhov thiab xa mus rau lub network card.
Txij li thaum lub pob ntawv + qhov taub hau tsis haum rau hauv MTU, pob ntawv raug txiav mus rau hauv cov khoom tawg thiab xa mus rau lub network.
Qhov hloov tom qab L3 balancer, thaum tau txais ib pob ntawv, ntxiv Vlan tag rau nws thiab xa mus rau.
Qhov kev hloov nyob rau hauv pem hauv ntej ntawm Nginx npe pom (raws li qhov chaw nres nkoj chaw) uas tus neeg rau zaub mov xav tau Vlan-encapsulated pob ntawv, yog li nws xa nws raws li yog, tsis tas yuav tshem Vlan tag.
Linux siv cov khoom ntawm cov pob khoom ib leeg thiab sib koom ua ke rau hauv ib pob loj.
Tom ntej no, pob ntawv mus txog Vlan interface, qhov twg thawj txheej raug tshem tawm ntawm nws - Vlan encapsulation.
Linux ces xa nws mus rau Tunnel interface, qhov twg lwm txheej raug tshem tawm ntawm nws - Qhov encapsulation.

Qhov nyuaj yog kom dhau tag nrho cov no raws li qhov tsis muaj rau tcpdump.
Cia peb pib los ntawm qhov kawg: puas muaj huv si (tsis muaj headers tsis tsim nyog) IP pob ntawv los ntawm cov neeg siv khoom, nrog vlan thiab qhov encapsulation tshem tawm?

tcpdump host <ip клиента>

Tsis yog, tsis muaj tej pob khoom ntawd ntawm lub server. Yog li qhov teeb meem yuav tsum muaj ua ntej. Puas muaj tej pob ntawv uas tsuas yog Vlan encapsulation tshem tawm?

tcpdump ip[32:4]=0xx390x2xx

0xx390x2xx yog tus neeg siv IP chaw nyob hauv hom hex.
32: 4 - chaw nyob thiab qhov ntev ntawm daim teb uas SCR IP tau sau rau hauv pob ntawv qhov.

Qhov chaw nyob teb yuav tsum tau xaiv los ntawm brute force, txij li hauv Internet lawv sau txog 40, 44, 50, 54, tab sis tsis muaj IP chaw nyob ntawd. Koj tuaj yeem saib ib qho ntawm cov pob ntawv hauv hex (qhov -xx lossis -XX parameter hauv tcpdump) thiab xam tus IP chaw koj paub.

Puas muaj cov pob ntawv tawg uas tsis muaj Vlan thiab Tunnel encapsulation tshem tawm?

tcpdump ((ip[6:2] > 0) and (not ip[6] = 64))
Cov khawv koob no yuav qhia peb txhua qhov tawg, nrog rau qhov kawg. Tej zaum, tib yam tuaj yeem raug lim los ntawm IP, tab sis kuv tsis tau sim, vim tias tsis muaj ntau cov pob ntawv zoo li no, thiab cov uas kuv xav tau tau yooj yim pom nyob rau hauv dav dav. Ntawm no lawv yog:

14:02:58.471063 In 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), length 1516: (tos 0x0, ttl 63, id 53652, offset 0, flags [+], proto IPIP (4), length 1500) 11.11.11.11 > 22.22.22.22: truncated-ip - 20 bytes missing! (tos 0x0, ttl 50, id 57750, offset 0, flags [DF], proto TCP (6), length 1500) 33.33.33.33.33333 > 44.44.44.44.80: Flags [.], seq 0:1448, ack 1, win 343, options [nop,nop,TS val 11660691 ecr 2998165860], length 1448 0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ...........A.... 0x0010: 4500 05dc d194 2000 3f09 d5fb 0a66 387d E.......?....f8} 0x0020: 1x67 7899 4500 06xx e198 4000 3206 6xx4 [email protected]. 0x0030: b291 x9xx x345 2541 83b9 0050 9740 0x04 .......A...P.@.. 0x0040: 6444 4939 8010 0257 8c3c 0000 0101 080x dDI9...W....... 0x0050: 00b1 ed93 b2b4 6964 xxd8 ffe1 006a 4578 ......ad.....jEx 0x0060: 6966 0000 4x4d 002a 0500 0008 0004 0100 if..MM.*........

14:02:58.471103 In 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 63, id 53652, offset 1480, flags [none], proto IPIP (4), length 40) 11.11.11.11 > 22.22.22.22: ip-proto-4 0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ...........A.... 0x0010: 4500 0028 d194 00b9 3f04 faf6 2x76 385x E..(....?....f8} 0x0020: 1x76 6545 xxxx 1x11 2d2c 0c21 8016 8e43 .faE...D-,.!...C 0x0030: x978 e91d x9b0 d608 0000 0000 0000 7c31 .x............|Q 0x0040: 881d c4b6 0000 0000 0000 0000 0000 ..............

Cov no yog ob feem ntawm ib pob (tib yam ID 53652) nrog ib daim duab (cov lus Exif pom hauv thawj pob). Vim lub fact tias muaj tej pob khoom nyob rau theem no, tab sis tsis nyob rau hauv lub merged daim ntawv nyob rau hauv lub dumps, qhov teeb meem yog kom meej meej nrog lub rooj sib txoos. Thaum kawg muaj cov ntaub ntawv pov thawj ntawm qhov no!

Lub pob ntawv decoder tsis tau nthuav tawm cov teeb meem uas yuav tiv thaiv kev tsim. Sim nws ntawm no: hpd.gasmi.net ua. Thaum xub thawj, thaum koj sim ua ib yam dab tsi nyob rau ntawd, tus decoder tsis nyiam lub pob ntawv hom. Nws muab tawm tias muaj qee qhov ntxiv ob octets ntawm Srcmac thiab Ethertype (tsis hais txog cov ntaub ntawv tawg). Tom qab tshem lawv, lub decoder pib ua haujlwm. Txawm li cas los xij, nws tsis pom muaj teeb meem.
Txawm ib tug yuav hais li cas los xij, tsis muaj lwm yam nrhiav tau tsuas yog cov Sysctl. Txhua yam uas tseem tshuav yog nrhiav txoj hauv kev los txheeb xyuas cov teeb meem servers kom nkag siab txog qhov ntsuas thiab txiav txim siab ua haujlwm ntxiv. Lub txee yuav tsum tau pom sai txaus:

netstat -s | grep "packet reassembles failed”

Nws kuj tseem nyob hauv snmpd hauv OID = 1.3.6.1.2.1.4.31.1.1.16.1 (ipSystemStatsReasmFails).

"Tus naj npawb ntawm kev ua tsis tiav tau pom los ntawm IP rov sib sau ua ke algorithm (rau qhov laj thawj dab tsi: ncua sijhawm, yuam kev, thiab lwm yam). "

Ntawm cov pab pawg ntawm cov servers uas qhov teeb meem tau kawm, ntawm ob lub txee no tau nce nrawm dua, ntawm ob qhov qeeb dua, thiab ob qho ntxiv nws tsis nce ntxiv. Muab piv cov dynamics ntawm cov txee no nrog cov kev hloov pauv ntawm HTTP yuam kev ntawm Java server qhia txog kev sib raug zoo. Ntawd yog, lub 'meter' tuaj yeem saib xyuas.

Muaj qhov kev ntseeg siab ntawm cov teeb meem yog qhov tseem ceeb heev kom koj tuaj yeem txiav txim siab seb puas rov qab Sysctl pab, vim los ntawm zaj dab neeg dhau los peb paub tias qhov no tsis tuaj yeem nkag siab tam sim ntawm daim ntawv thov. Qhov ntsuas no yuav tso cai rau peb txheeb xyuas txhua qhov teeb meem hauv kev tsim khoom ua ntej cov neeg siv pom nws.
Tom qab dov rov qab Sysctl, qhov kev soj ntsuam yuam kev tau nres, yog li qhov ua rau ntawm cov teeb meem tau raug pov thawj, nrog rau qhov tseeb tias qhov rollback pab tau.

Peb dov rov qab cov fragmentation nqis ntawm lwm cov servers, qhov twg tshiab kev soj ntsuam tuaj rau hauv kev ua si, thiab qhov chaw peb faib txawm ntau lub cim xeeb rau fragments dua li yav tas los lub neej ntawd (qhov no yog UDP cov txheeb cais, ib feem poob ntawm uas tsis pom zoo rau cov keeb kwm yav dhau) .

Cov lus nug tseem ceeb tshaj plaws

Vim li cas cov pob ntawv fragmented ntawm peb L3 balancer? Feem ntau ntawm cov pob ntawv uas tuaj txog los ntawm cov neeg siv khoom sib npaug yog SYN thiab ACK. Qhov loj ntawm cov pob no yog me me. Tab sis txij li qhov sib koom ntawm cov pob ntawv no loj heev, tawm tsam lawv keeb kwm yav dhau los peb tsis tau pom tias muaj cov pob ntawv loj uas pib tawg.

Yog vim li cas yog ib tug tawg configuration tsab ntawv advms ua ntawm cov servers nrog Vlan interfaces (muaj tsawg heev servers nrog tagged tsheb hauv kev tsim khoom thaum lub sijhawm ntawd). Advmss tso cai rau peb xa mus rau tus neeg siv cov ntaub ntawv uas cov pob ntawv hauv peb cov kev taw qhia yuav tsum me me kom thiaj li tom qab txuas qhov taub hau rau lawv lawv tsis tas yuav tawg.

Vim li cas Sysctl rollback tsis pab, tab sis reboot ua? Rolling rov qab Sysctl hloov tus nqi ntawm lub cim xeeb muaj rau kev sib koom ua ke pob. Nyob rau tib lub sijhawm, pom tseeb qhov tseeb ntawm kev nco dhau rau cov khoom tawg tau ua rau qeeb ntawm kev sib txuas, uas ua rau cov khoom tawg tau ncua sijhawm ntev hauv cov kab. Ntawd yog, cov txheej txheem mus rau hauv lub voj voog.
Lub reboot cleared lub cim xeeb thiab txhua yam rov qab mus rau kev txiav txim.

Nws puas tuaj yeem ua yam tsis muaj Workaround? Yog lawm, tab sis muaj kev pheej hmoo siab tawm ntawm cov neeg siv yam tsis muaj kev pabcuam thaum muaj kev tawm tsam. Tau kawg, kev siv Workaround ua rau muaj teeb meem ntau yam, suav nrog kev ua haujlwm qeeb ntawm ib qho ntawm cov kev pabcuam rau cov neeg siv, tab sis txawm li cas los xij peb ntseeg tias qhov kev nqis tes ua tau ncaj ncees.

Ua tsaug ntau rau Andrey Timofeev (atimofeyev) rau kev pabcuam hauv kev tshawb nrhiav, nrog rau Alexey Krenev (ntaus ntawv) - rau kev ua haujlwm Titanic ntawm kev hloov kho Centos thiab kernels ntawm servers. Ib txheej txheem uas nyob rau hauv cov ntaub ntawv no yuav tsum tau pib txij thaum pib ob peb zaug, uas yog vim li cas nws dragged rau ntau lub hlis.

Tau qhov twg los: www.hab.com

Ceev faj ntawm qhov tsis zoo uas ua rau muaj kev sib tw ua haujlwm. Ntu 1: FragmentSmack/SegmentSmack