Ceev faj ntawm qhov tsis zoo uas ua rau muaj kev sib tw ua haujlwm. Ntu 1: FragmentSmack/SegmentSmack

Nyob zoo txhua tus! Kuv lub npe yog Dmitry Samsonov, thiab kuv ua haujlwm ua tus thawj coj ntawm Odnoklassniki. Peb muaj ntau dua 7 lub servers, 11 lub thawv hauv peb huab, thiab 200 daim ntawv thov, uas nyob rau hauv ntau yam kev teeb tsa tsim 700 pawg sib txawv. Feem ntau ntawm cov servers khiav CentOS 7.
Thaum Lub Yim Hli 14, 2018, cov ntaub ntawv hais txog qhov tsis muaj zog ntawm FragmentSmack tau tshaj tawm.
(CVE-2018-5391) thiab SegmentSmack (CVE-2018-5390). Cov no yog qhov tsis muaj zog nrog lub network tawm tsam vector thiab qhov qhab nia siab heev (7.5), hem tias yuav tsis kam lees kev pabcuam (DoS) vim yog cov peev txheej tsis txaus (CPU). Kev kho kernel rau FragmentSmack tsis tau raug thov thaum lub sijhawm ntawd; qhov tseeb, nws tau tso tawm ntau tom qab qhov tsis muaj zog tau tshaj tawm rau pej xeem. Kev hloov kho kernel tau pom zoo kom kho SegmentSmack. Lub pob hloov tshiab nws tus kheej tau tso tawm tib hnub ntawd; txhua yam uas tseem tshuav yog nruab nws.
Tsis yog, peb tsis tawm tsam qhov hloov tshiab kernel kiag li! Txawm li cas los xij, muaj qee qhov nuances ...

Peb hloov kho lub kernel li cas thaum tsim khoom

Feem ntau, tsis muaj dab tsi nyuaj:

1. Rub tawm cov pob khoom;
2. Nruab lawv rau ntawm ntau lub servers (suav nrog cov servers uas tuav peb huab);
3. Xyuas kom tseeb tias tsis muaj dab tsi tawg;
4. Xyuas kom tseeb tias txhua qhov chaw teeb tsa kernel txheem tau siv yam tsis muaj qhov yuam kev;
5. Tos ob peb hnub;
6. Tshawb xyuas cov ntsuas ntawm lub server;
7. Hloov kev xa tawm ntawm cov servers tshiab mus rau lub kernel tshiab;
8. Hloov kho txhua lub servers thoob plaws cov chaw khaws ntaub ntawv (ib lub chaw khaws ntaub ntawv ib zaug kom txo qhov cuam tshuam rau cov neeg siv yog tias muaj teeb meem);
9. Rov pib dua txhua lub servers.

Rov ua dua rau txhua ceg ntawm peb cov noob uas twb muaj lawm. Tam sim no, qhov no yog:

• Tshuag CentOS 7 3.10 - rau feem ntau cov servers ib txwm muaj;
• Vanilla 4.19 - rau peb ib-huab huab, vim peb xav tau BFQ, BBR, thiab lwm yam.;
• Elrepo kernel-ml 5.2 — rau cov neeg faib khoom uas muaj khoom ntau heev, vim tias 4.19 tsis ruaj khov ua ntej, tab sis xav tau cov yam ntxwv zoo ib yam.

Raws li koj yuav tau kwv yees, kev rov pib dua ntau txhiab lub servers siv sijhawm ntev tshaj plaws. Vim tias tsis yog txhua qhov tsis muaj zog yog qhov tseem ceeb rau txhua lub servers, peb tsuas yog rov pib dua cov uas nkag tau ncaj qha los ntawm internet. Hauv huab, kom tswj tau qhov yooj ywm, peb tsis xauv cov thawv sab nraud rau cov servers ib leeg khiav lub kernel tshiab, tab sis rov pib dua txhua lub hosts yam tsis muaj kev zam. Hmoov zoo, cov txheej txheem yooj yim dua li cov servers ib txwm muaj. Piv txwv li, cov thawv tsis muaj xeev tuaj yeem yooj yim tsiv mus rau lwm lub server thaum lub sijhawm rov pib dua.

Txawm li cas los xij, tseem muaj ntau yam haujlwm yuav tsum tau ua, thiab nws yuav siv sijhawm ntau lub lis piam, lossis txawm tias ntau lub hlis yog tias muaj teeb meem tshwm sim nrog cov version tshiab. Cov neeg tawm tsam paub zoo txog qhov no, yog li xav tau ib txoj kev npaj B.

Kev daws teeb meem ntawm FragmentSmack/SegmentSmack

Zoo hmoo, rau qee qhov tsis muaj zog, muaj Txoj Kev Npaj B, hu ua Workaround. Feem ntau, qhov no cuam tshuam nrog kev hloov pauv kernel lossis daim ntawv thov teeb tsa kom txo qhov cuam tshuam lossis tiv thaiv kev siv tsis raug.

Nyob rau hauv cov ntaub ntawv ntawm FragmentSmack / SegmentSmack tau muab tswv yim ib qho kev daws teeb meem zoo li no:

«Koj tuaj yeem hloov cov nqi qub ntawm 4MB thiab 3MB hauv net.ipv4.ipfrag_high_thresh thiab net.ipv4.ipfrag_low_thresh (thiab lawv cov IPv6 counterparts net.ipv6.ipfrag_high_thresh thiab net.ipv6.ipfrag_low_thresh) rau 256 kB thiab 192 kB, feem, lossis qis dua. Kev sim qhia me ntsis mus rau qhov tseem ceeb ntawm kev siv CPU thaum lub sijhawm tawm tsam, nyob ntawm kho vajtse, chaw, thiab cov xwm txheej. Txawm li cas los xij, tej zaum yuav muaj qee qhov kev cuam tshuam rau kev ua tau zoo vim yog ipfrag_high_thresh = 262144 bytes, vim tias tsuas yog ob daim 64K tuaj yeem haum rau hauv kab rov ua dua ib zaug. Piv txwv li, muaj kev pheej hmoo uas cov ntawv thov uas ua haujlwm nrog cov pob UDP loj yuav tawg.".

Cov parameters lawv tus kheej nyob rau hauv cov ntaub ntawv kernel piav qhia raws li nram no:

ipfrag_high_thresh - LONG INTEGER
    Maximum memory used to reassemble IP fragments.

ipfrag_low_thresh - LONG INTEGER
    Maximum memory used to reassemble IP fragments before the kernel
    begins to remove incomplete fragment queues to free up resources.
    The kernel still accepts new fragments for defragmentation.

Peb tsis muaj kev sib txuas UDP loj ntawm peb cov kev pabcuam tsim khoom. Tsis muaj tsheb khiav sib cais ntawm LAN, thiab muaj qee qhov ntawm WAN, tab sis nws tsis tseem ceeb. Zoo li tsis muaj dab tsi tsis ncaj ncees lawm - Workaround npaj txhij rau dov tawm!

FragmentSmack/SegmentSmack. Ntshav Thawj Zaug

Qhov teeb meem thawj zaug uas peb ntsib yog tias cov thawv huab qee zaum tsuas yog siv ib feem ntawm cov chaw tshiab (tsuas yog ipfrag_low_thresh), thiab qee zaum tsis siv lawv kiag li, tsuas yog poob thaum pib. Peb tsis tuaj yeem rov ua qhov teeb meem no dua (kev siv tag nrho cov chaw yooj yim). Kev nkag siab tias vim li cas lub thawv poob thaum pib tsis yooj yim: tsis pom qhov yuam kev. Muaj ib qho tseeb: dov rov qab cov chaw teeb tsa daws qhov teeb meem poob ntawm lub thawv.

Vim li cas ho tsis txaus siv Syssctl ntawm tus tswv tsev? Lub thawv nyob hauv nws lub npe network tshwj xeeb, yog li tsawg kawg ib feem ntawm cov kev teeb tsa Sysctl hauv network nyob rau hauv lub thawv yuav txawv ntawm tus tswv tsev.

Cov chaw teeb tsa Syssctl siv li cas hauv lub thawv? Vim tias peb cov thawv tsis muaj cai, hloov pauv txhua qhov chaw teeb tsa Syssctl los ntawm hauv lub thawv nws tus kheej yog qhov tsis yooj yim sua - peb tsuas yog tsis muaj kev tso cai tsim nyog. Lub sijhawm ntawd, peb huab siv Docker los tso cov thawv (tam sim no podman). Cov kev teeb tsa ntawm lub thawv tshiab, suav nrog cov chaw teeb tsa Syssctl uas xav tau, tau raug xa mus rau Docker ntawm API.
Tom qab sim ntau hom versions, nws pom tseeb tias Docker API tsis rov qab tag nrho cov yuam kev (tsawg kawg hauv version 1.10). Thaum peb sim tso lub thawv siv "docker run," peb thaum kawg pom qee yam:

write /proc/sys/net/ipv4/ipfrag_high_thresh: invalid argument docker: Error response from daemon: Cannot start container <...>: [9] System error: could not synchronise with container process.

Tus nqi parameter tsis raug. Tab sis vim li cas? Thiab vim li cas nws tsuas yog tsis raug qee zaum xwb? Nws hloov tawm tias Docker tsis lav qhov kev txiav txim uas Sysctl parameters raug siv (qhov version kawg tau txheeb xyuas yog 1.13.1), yog li qee zaum ipfrag_high_thresh yuav sim teeb tsa nws rau 256K thaum ipfrag_low_thresh tseem yog 3M. Qhov no txhais tau tias qhov ciam teb sab saud qis dua qhov ciam teb qis dua, uas ua rau muaj qhov yuam kev.

Lub sijhawm ntawd, peb twb muaj peb tus kheej txoj hauv kev rau kev kho dua lub thawv tom qab tso tawm (khov lub thawv tom qab lub tub yees cgroup thiab ua cov lus txib hauv lub thawv lub npe ntawm cov ip netns), thiab peb kuj tau ntxiv lub peev xwm los teev cov kev teeb tsa Syssctl rau ntu no. Qhov teeb meem tau daws lawm.

FragmentSmack/SegmentSmack. Ntshav Thawj Zaug 2

Thaum peb pib tau txais ob peb qhov kev tsis txaus siab los ntawm cov neeg siv thawj zaug, peb tsis tau nkag siab txog kev siv Workaround hauv huab. Thaum ntawd, ob peb lub lis piam dhau los txij li Workaround tau muab tso rau ntawm thawj cov servers. Kev tshawb nrhiav thawj zaug qhia tau tias cov lus tsis txaus siab tau los ntawm cov kev pabcuam tshwj xeeb, tsis yog txhua lub servers hauv cov kev pabcuam ntawd. Qhov teeb meem tau dhau los ua qhov tsis meej heev.

Ua ntej tshaj plaws, peb tau sim dov rov qab cov chaw teeb tsa Sysctl, tab sis qhov ntawd tsis muaj txiaj ntsig. Ntau yam kev hloov pauv ntawm lub server thiab daim ntawv thov teeb tsa kuj tsis tau pab. Kev rov pib dua tau pab. Rov pib dua rau Linux tsis zoo li qub li nws yog ib qho mob ib txwm rau kev ua haujlwm nrog Windows Yav tas los. Txawm li cas los xij, nws ua haujlwm tau zoo, thiab peb suav tias nws yog "kernel glitch" thaum siv cov chaw Syssctl tshiab. Peb ruam kawg li...

Peb lub lis piam tom qab, qhov teeb meem rov tshwm sim dua. Kev teeb tsa ntawm cov servers no yooj yim heev: Nginx hauv hom proxy/balancer. Tsheb khiav tsawg. Cov ntaub ntawv tshiab: tus lej ntawm 504 qhov yuam kev ntawm cov neeg siv khoom tau nce ntxiv txhua hnub.Lub Sijhawm Gateway). Daim duab qhia tus lej ntawm 504 qhov yuam kev hauv ib hnub rau qhov kev pabcuam no:

Tag nrho cov yuam kev muaj feem cuam tshuam nrog tib lub backend - qhov nyob hauv huab. Daim duab qhia txog kev siv lub cim xeeb rau cov pob ntawv ntawm lub backend no zoo li no:

Qhov no yog ib qho ntawm cov teeb meem pom tseeb tshaj plaws ntawm cov duab qhia txog lub operating system. Hauv huab, lwm qhov teeb meem network nrog QoS (Traffic Control) chaw tau kho tib lub sijhawm. Nws zoo ib yam li ntawm daim duab qhia txog kev siv lub cim xeeb ntawm pob ntawv:

Qhov kev xav yooj yim xwb: yog tias lawv zoo ib yam ntawm cov duab kos, ces lawv muaj tib qho laj thawj. Ntxiv mus, teeb meem nrog hom kev nco no tsis tshua muaj.

Qhov teeb meem uas peb kho tau yog tias peb siv lub fq packet scheduler rau QoS nrog cov chaw teeb tsa ua ntej. Los ntawm lub neej ntawd, nws tso cai rau 100 pob ntawv kom raug teem rau ib qho kev sib txuas, thiab qee qhov kev sib txuas yuav pib sau cov kab kom puv thaum bandwidth tsawg. Hauv qhov no, cov pob ntawv yuav raug tso tseg. Qhov no pom tau hauv tc cov ntaub ntawv txheeb cais (tc -s qdisc):

qdisc fq 2c6c: parent 1:2c6c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
 Sent 454701676345 bytes 491683359 pkt (dropped 464545, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  1024 flows (1021 inactive, 0 throttled)
  0 gc, 0 highprio, 0 throttled, 464545 flows_plimit

"464545 flows_plimit" sawv cev rau cov pob ntawv poob vim yog tshaj qhov txwv ntawm kab rau ib qho kev sib txuas, thiab "poob 464545" yog qhov sib npaug ntawm tag nrho cov pob ntawv poob rau lub sijhawm teem sijhawm no. Tom qab nce qhov ntev ntawm kab mus rau 1 thiab rov pib dua cov thawv, qhov teeb meem ploj mus. Tam sim no nws yog lub sijhawm los zaum rov qab thiab muaj smoothie.

FragmentSmack/SegmentSmack. Ntshav Kawg

Ua ntej, ob peb lub hlis tom qab cov qhov tsis zoo ntawm lub kernel tau tshaj tawm, qhov kev kho rau FragmentSmack thaum kawg tau tso tawm (nco ntsoov, qhov kev tshaj tawm thaum Lub Yim Hli tsuas yog tso tawm qhov kev kho rau SegmentSmack), uas tau muab sijhawm rau peb tso tseg Workaround, uas tau ua rau peb muaj teeb meem ntau heev. Peb twb tau tsiv qee lub servers mus rau lub kernel tshiab thaum lub sijhawm no, thiab tam sim no peb yuav tsum pib los ntawm kos. Vim li cas peb thiaj hloov kho lub kernel yam tsis tau tos qhov kev kho FragmentSmack? Qhov tseeb yog tias cov txheej txheem ntawm kev tiv thaiv cov qhov tsis zoo no tau sib xws (thiab sib koom ua ke) nrog cov txheej txheem ntawm kev hloov kho Workaround nws tus kheej. CentOS (uas siv sijhawm ntev dua li hloov kho lub kernel xwb). Dhau li ntawm qhov ntawd, SegmentSmack yog qhov tsis muaj zog txaus ntshai dua, thiab kev kho rau nws tau muaj tam sim ntawd, yog li nws ua rau muaj kev nkag siab txawm li cas los xij. Txawm li cas los xij, tsuas yog hloov kho lub kernel CentOS peb ua tsis tau vim yog qhov tsis muaj zog ntawm FragmentSmack uas tau tshwm sim thaum lub sijhawm CentOS 7.5 tsuas yog kho tau hauv version 7.6 xwb, yog li peb yuav tsum tso tseg qhov hloov tshiab mus rau 7.5 thiab pib dua nrog qhov hloov tshiab mus rau 7.6. Qhov no kuj tshwm sim.

Qhov thib ob, peb tau pib tau txais cov lus tsis txaus siab uas tsis tshua muaj neeg siv txog cov teeb meem. Tam sim no peb paub tseeb tias lawv txhua tus muaj feem cuam tshuam nrog cov ntaub ntawv upload los ntawm cov neeg siv khoom mus rau qee qhov ntawm peb cov servers. Ntxiv mus, cov servers no suav nrog ib feem pua ​​​​​​me me ntawm tag nrho cov uploads.

Raws li peb nco qab los ntawm zaj dab neeg saum toj no, kev dov rov qab Syssctl tsis tau pab. Kev rov pib dua tau pab, tab sis tsuas yog ib ntus xwb.
Sysctl tseem raug liam tias ua txhaum, tab sis lub sijhawm no nws yog qhov tseem ceeb heev uas yuav tsum tau sau cov ntaub ntawv ntau li ntau tau. Tsis tas li ntawd, qhov xav tau heev yog lub peev xwm rov ua dua qhov teeb meem upload ntawm tus neeg siv khoom kom paub meej ntxiv txog qhov xwm txheej.

Kev tshuaj xyuas txhua yam ntaub ntawv thiab cov cav uas muaj tsis tau coj peb los ze rau kev nkag siab txog qhov xwm txheej. Peb xav tau ib txoj hauv kev los rov ua qhov teeb meem thiab tshawb xyuas qhov kev sib txuas tshwj xeeb. Thaum kawg, cov neeg tsim khoom tau tswj hwm kom rov ua qhov teeb meem ntawm lub cuab yeej sim siv cov version tshwj xeeb ntawm lub app thaum txuas nrog ntawm Wi-Fi. Qhov no yog qhov kev tawg hauv kev tshawb nrhiav. Tus neeg siv khoom txuas nrog Nginx, uas proxying rau backend, uas yog peb daim ntawv thov Java.

Cov lus sib tham thaum muaj teeb meem yog raws li nram no (sau tseg rau ntawm Nginx proxy sab):

1. Tus Neeg Siv Khoom: thov kom tau txais cov ntaub ntawv hais txog kev rub tawm cov ntaub ntawv tiav.
2. Java server: cov lus teb.
3. Tus Neeg Siv Khoom: POST nrog cov ntaub ntawv.
4. Java server: yuam kev.

Lub Java server sau tias 0 bytes ntawm cov ntaub ntawv tau txais los ntawm tus neeg siv khoom, thaum Nginx proxy sau tias qhov kev thov siv sijhawm ntau dua 30 vib nas this (30 vib nas this yog lub sijhawm ntawm daim ntawv thov tus neeg siv khoom). Vim li cas lub sijhawm tas, thiab vim li cas 0 bytes? Los ntawm HTTP qhov kev xav, txhua yam ua haujlwm raws li qhov xav tau, tab sis POST nrog cov ntaub ntawv zoo li ploj ntawm lub network. Ntxiv mus, nws ploj ntawm tus neeg siv khoom thiab Nginx. Nws yog lub sijhawm los ntes Tcpdump! Tab sis ua ntej, peb yuav tsum nkag siab txog kev teeb tsa network. Lub Nginx proxy nyob tom qab L3 load balancer. NFwareTunneling siv los xa cov pob ntawv los ntawm L3 load balancer mus rau lub server, uas ntxiv nws cov headers rau cov pob ntawv:

Hauv qhov no, lub network tuaj txog rau lub server no hauv daim ntawv ntawm VLAN-tagged traffic, uas kuj ntxiv nws cov teb rau cov pob ntawv:

Cov tsheb khiav no kuj tseem tuaj yeem raug faib ua ntu zus (tib feem pua ​​me me ntawm cov tsheb khiav uas tau faib ua ntu zus uas peb tau tham hauv qhov kev ntsuam xyuas kev pheej hmoo Workaround), uas kuj hloov cov ntsiab lus ntawm cov headers:

Ib zaug ntxiv: cov pob ntawv raug kaw nrog VLAN tag, kaw los ntawm lub qhov av, thiab sib cais. Yuav kom nkag siab zoo dua txog qhov no tshwm sim li cas, cia peb taug qab cov pob ntawv txoj kev los ntawm tus neeg siv khoom mus rau Nginx proxy.

1. Lub pob ntawv tuaj txog ntawm L3 load balancer. Txhawm rau kom ntseeg tau tias muaj kev xa mus rau hauv lub chaw khaws ntaub ntawv kom raug, lub pob ntawv raug kaw hauv qhov av thiab xa mus rau daim npav network.
2. Vim tias cov headers ntawm pob ntawv + qhov av tsis haum rau hauv MTU, pob ntawv raug txiav ua tej daim me me thiab xa mus rau lub network.
3. Tus hloov tom qab L3 balancer ntxiv ib daim VLAN tag rau ib pob ntawv thaum tau txais nws thiab xa nws mus ntxiv.
4. Tus hloov nyob rau pem hauv ntej ntawm Nginx proxy pom (raws li qhov chaw teeb tsa chaw nres nkoj) tias lub server xav tau pob ntawv VLAN-encapsulated, yog li nws xa nws raws li nws yog, yam tsis tau tshem tawm VLAN tag.
5. Linux txais cov khoom me me ntawm cov pob khoom ib leeg thiab nplaum lawv rau hauv ib pob loj.
6. Tom ntej no, lub pob ntawv mus rau VLAN interface, qhov twg thawj txheej - VLAN encapsulation - raug tshem tawm.
7. ces Linux xa nws mus rau qhov Tunnel interface, qhov twg lwm txheej raug tshem tawm ntawm nws - Tunnel encapsulation.

Qhov nyuaj yog nyob rau hauv kev xa tag nrho cov no ua cov kev cai rau tcpdump.
Cia peb pib ntawm qhov kawg: puas muaj cov pob IP huv si (tsis muaj cov headers tsis tsim nyog) los ntawm cov neeg siv khoom, nrog VLAN thiab qhov tunnel encapsulation raug tshem tawm?

tcpdump host <ip клиента>

Tsis yog, tsis muaj cov pob ntawv zoo li no ntawm lub server. Yog li qhov teeb meem yuav tsum tau muaj ua ntej lawm. Puas muaj cov pob ntawv twg uas tsuas yog tshem tawm VLAN encapsulation?

tcpdump ip[32:4]=0xx390x2xx

0xx390x2xx yog tus neeg siv khoom IP chaw nyob hauv hom ntawv hex.
32:4 — qhov chaw nyob thiab ntev ntawm daim teb uas SCR IP sau rau hauv pob ntawv Tunnel.

Kuv yuav tsum tau siv zog ua kom lub chaw nyob raug, vim tias hauv internet tau tshaj tawm 40, 44, 50, thiab 54, tab sis tsis muaj chaw nyob IP. Koj tuaj yeem saib ib qho ntawm cov pob ntawv hauv hex (tus parameter -xx lossis -XX hauv tcpdump) thiab xam seb qhov chaw nyob twg sib raug rau qhov chaw nyob IP paub.

Puas muaj cov pob ntawv tawg uas tsis tau tshem tawm Vlan thiab Tunnel encapsulation?

tcpdump ((ip[6:2] > 0) and (not ip[6] = 64))

Daim khawv koob no yuav qhia peb txhua daim, suav nrog daim kawg. Tej zaum yuav ua tau los lim tib yam los ntawm IP, tab sis kuv tsis tau sim vim tias tsis muaj ntau cov pob ntawv zoo li no, thiab cov uas kuv xav tau yooj yim nrhiav tau hauv cov kwj deg dav dav. Nov yog lawv:

14:02:58.471063 In 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), length 1516: (tos 0x0, ttl 63, id 53652, offset 0, flags [+], proto IPIP (4), length 1500)
    11.11.11.11 > 22.22.22.22: truncated-ip - 20 bytes missing! (tos 0x0, ttl 50, id 57750, offset 0, flags [DF], proto TCP (6), length 1500)
    33.33.33.33.33333 > 44.44.44.44.80: Flags [.], seq 0:1448, ack 1, win 343, options [nop,nop,TS val 11660691 ecr 2998165860], length 1448
        0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ...........A....
        0x0010: 4500 05dc d194 2000 3f09 d5fb 0a66 387d E.......?....f8}
        0x0020: 1x67 7899 4500 06xx e198 4000 3206 6xx4 .faEE.....@.2.m.
        0x0030: b291 x9xx x345 2541 83b9 0050 9740 0x04 .......A...P.@..
        0x0040: 6444 4939 8010 0257 8c3c 0000 0101 080x dDI9...W.......
        0x0050: 00b1 ed93 b2b4 6964 xxd8 ffe1 006a 4578 ......ad.....jEx
        0x0060: 6966 0000 4x4d 002a 0500 0008 0004 0100 if..MM.*........

14:02:58.471103 Hauv 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), ntev 62: (tos 0x0, ttl 63, ID 53652, qhov sib txawv 1480, chij [tsis muaj], proto IPIP (4), ntev 40)
    11.11.11.11 > 22.22.22.22: ip-proto-4
        0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ..........A....
        0x0010: 4500 0028 d194 00b9 3f04 faf6 2x76 385x E..(....?....f8}
        0x0020: 1x76 6545 xxxx 1x11 2d2c 0c21 8016 8e43 .faE...D-,.!...C
        0x0030: x978 e91d x9b0 d608 0000 0000 0000 7c31 .x............|Q
        0x0040: 881d c4b6 0000 0000 0000 0000 0000 .............

Cov no yog ob daim ntawm tib lub pob ntawv (tib lub ID 53652) nrog ib daim duab (lo lus Exif pom tau hauv thawj pob ntawv). Txij li thaum cov pob ntawv muaj nyob rau theem no, tab sis tsis nyob hauv cov pob khoom sib koom ua ke, muaj teeb meem nrog kev sib dhos. Thaum kawg, muaj qee qhov kev lees paub sau tseg!

Tus decoder pob ntawv tsis pom muaj teeb meem dab tsi uas tiv thaiv kev tsim. Kuv sim nws ntawm no: hpd.gasmi.netThaum xub thawj, thaum sim ntxig ib yam dab tsi rau hauv nws, tus decoder tsis nyiam hom ntawv pob ntawv. Nws tau tshwm sim tias muaj ob lub octets ntxiv ntawm Srcmac thiab Ethertype (tsis muaj feem cuam tshuam rau cov ntaub ntawv tawg). Tom qab tshem lawv tawm, tus decoder ua haujlwm. Txawm li cas los xij, nws tsis qhia txog teeb meem dab tsi.
Txawm koj saib nws li cas los xij, tsis muaj dab tsi ntxiv pom tsuas yog cov Syssctl parameters xwb. Txhua yam uas tshuav yog nrhiav txoj hauv kev los txheeb xyuas cov servers uas muaj teeb meem kom nkag siab txog qhov ntsuas thiab txiav txim siab ua ntxiv. Lub counter tsim nyog tau pom sai sai:

netstat -s | grep "packet reassembles failed”

Nws kuj tseem nyob hauv snmpd hauv qab OID = 1.3.6.1.2.1.4.31.1.1.16.1 (ipSystemStatsReasmFails).

Tus naj npawb ntawm cov kev ua tsis tiav uas tau pom los ntawm IP re-assembly algorithm (vim li cas los xij: lub sijhawm tas, qhov yuam kev, thiab lwm yam)."

Ntawm cov servers uas tau kawm, qhov kev suav no nce sai dua ntawm ob, qeeb dua ntawm ob, thiab tsis nce kiag li ntawm ob. Kev sib piv qhov dynamics ntawm qhov kev suav no nrog qhov dynamics ntawm HTTP yuam kev ntawm Java server tau qhia txog kev sib raug zoo. Qhov no txhais tau tias qhov kev suav tuaj yeem raug saib xyuas.

Muaj ib qho qhia txog teeb meem uas ntseeg tau yog qhov tseem ceeb heev rau kev txiav txim siab kom raug seb Sysctl rollback puas pab tau, raws li peb tau kawm los ntawm kev sib tham dhau los tias qhov no tsis pom tseeb tam sim ntawd los ntawm daim ntawv thov. Qhov qhia no yuav tso cai rau peb txheeb xyuas txhua qhov teeb meem hauv kev tsim khoom ua ntej cov neeg siv pom lawv.
Tom qab dov rov qab Syssctl, qhov yuam kev saib xyuas tau nres, yog li qhov ua rau muaj teeb meem tau ua pov thawj, nrog rau qhov tseeb tias qhov dov rov qab pab tau.

Peb tau dov rov qab cov chaw teeb tsa fragmentation ntawm lwm lub servers uas qhov kev saib xyuas tshiab tau pib, thiab qee qhov chaw peb txawm tias tau faib ntau lub cim xeeb rau cov fragments dua li yav dhau los (qhov no yog rau UDP cov ntaub ntawv, qhov poob ib nrab uas tsis pom tseeb hauv cov ntsiab lus tag nrho).

Cov lus nug tseem ceeb tshaj plaws

Vim li cas cov pob ntawv thiaj li tawg ua tej daim me me ntawm peb lub L3 load balancer? Feem ntau ntawm cov pob ntawv uas los ntawm cov neeg siv mus rau lub load balancer yog SYN thiab ACK. Cov pob ntawv no me me. Txawm li cas los xij, vim tias cov pob ntawv no muaj feem ntau heev, peb tsis tau pom cov pob ntawv loj uas tawg ua tej daim me me.

Qhov ua rau yog ib daim ntawv teeb tsa tsis zoo. advmss Ntawm cov servers nrog VLAN interfaces (lub sijhawm ntawd, muaj ob peb lub servers hauv kev tsim khoom nrog cov tsheb khiav tagged). Advmss tso cai rau peb qhia rau tus neeg siv khoom tias cov pob ntawv uas peb mus yuav tsum me dua kom lawv tsis tas yuav tsum tau fragmented tom qab appending tunnel headers.

Vim li cas ho tsis rov qab siv Sysctl, tab sis kev rov pib dua ua tau? Kev rov qab siv Sysctl tau hloov qhov ntau ntawm lub cim xeeb uas muaj rau kev sib txuas ntawm cov pob ntawv. Ntxiv mus, nws zoo li qhov tseeb tias lub cim xeeb rau cov khoom seem puv ua rau kev sib txuas qeeb, uas ua rau cov khoom seem raug khaws cia rau hauv kab ntev. Hauv lwm lo lus, cov txheej txheem tau daig hauv lub voj voog.
Qhov reboot tau tshem tawm lub cim xeeb thiab txhua yam rov qab los rau qhov qub.

Puas ua tau kom tsis txhob muaj Workaround? Yog lawm, tab sis muaj kev pheej hmoo siab uas yuav ua rau cov neeg siv tsis muaj kev pabcuam yog tias muaj kev tawm tsam. Txawm hais tias siv Workaround ua rau muaj ntau yam teeb meem, suav nrog kev qeeb rau cov neeg siv ntawm ib qho ntawm cov kev pabcuam, peb tseem ntseeg tias cov kev ntsuas no yog qhov raug.

Ua tsaug ntau rau Andrey Timofeev (Atimofeev) rau kev pab hauv kev tshawb nrhiav, nrog rau Alexey Krenev (devicex) - rau txoj haujlwm titanic ntawm kev hloov kho tshiab Centos thiab cov cores ntawm lub server. Hauv qhov no, cov txheej txheem yuav tsum tau rov pib dua ntau zaus, ua rau nws siv sijhawm ntau lub hlis.

Tau qhov twg los: www.hab.com