Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Lumelang bohle! Lebitso la ka ke Dmitry Samsonov, ke sebetsa ke le mookameli oa tsamaiso ea Odnoklassniki. Re na le li-server tse fetang likete tse 7, lijana tse likete tse 11 marung a rona le lits'ebetso tse 200, tseo ka litlhophiso tse fapaneng li theha lihlopha tse fapaneng tse 700. Boholo ba li-server li tsamaisa CentOS 7.
Ka la 14 Phato, 2018, ho ile ha phatlalatsoa lintlha tse mabapi le ho ba kotsing ha FragmentSmack
(CVE-2018-5391) le SegmentSmack (CVE-2018-5390). Tsena ke bofokoli bo nang le vector ea tlhaselo ea marang-rang le lintlha tse phahameng hantle (7.5), tse sokelang ho haneloa ha litšebeletso (DoS) ka lebaka la ho felloa ke matla ha lisebelisoa (CPU). Tokiso ea kernel bakeng sa FragmentSmack e ne e sa hlahisoa ka nako eo; ho feta moo, e ile ea tsoa kamora nako e telele ho feta phatlalatso ea tlhahisoleseling mabapi le ho ba kotsing. Ho felisa SegmentSmack, ho khothalelitsoe ho nchafatsa kernel. Sephutheloana sa ntlafatso ka boeona se ile sa lokolloa ka letsatsi lona leo, ho neng ho setse feela ho e kenya.
Che, ha re khahlanong le ho nchafatsa kernel ho hang! Leha ho le joalo, ho na le li-nuances ...

Re ntlafatsa kernel joang tlhahiso

Ka kakaretso, ha ho letho le rarahaneng:

  1. Khoasolla liphutheloana;
  2. Li kenye ho li-server tse 'maloa (ho kenyeletsoa le li-server tse amohelang leru la rona);
  3. Etsa bonnete ba hore ha ho letho le robehileng;
  4. Etsa bonnete ba hore litlhophiso tsohle tse tloaelehileng tsa kernel li sebelisoa ntle le liphoso;
  5. Ema matsatsi a seng makae;
  6. Hlahloba tshebetso ya seva;
  7. Fetolela phepelo ea li-server tse ncha ho kernel e ncha;
  8. Ntlafatsa li-server tsohle ka setsi sa data (setsi se le seng sa data ka nako ho fokotsa phello ho basebelisi haeba ho na le mathata);
  9. Qala hape li-server tsohle.

Pheta bakeng sa makala ohle a lithollo tseo re nang le tsona. Hajoale ke:

  • Stock CentOS 7 3.10 - bakeng sa li-server tse ngata tse tloaelehileng;
  • Vanilla 4.19 - bakeng sa rona maru a leru le le leng, hobane re hloka BFQ, BBR, joalo-joalo;
  • Elrepo kernel-ml 5.2 - bakeng sa baphatlalatsi ba imetsoeng haholo, hobane 4.19 e ne e tloaetse ho itšoara ka mokhoa o sa tsitsang, empa likarolo tse tšoanang lia hlokahala.

Joalokaha u ka be u nahanne, ho qala li-server tse likete ho nka nako e telele ka ho fetisisa. Kaha ha se mathata ohle a bohlokoa bakeng sa li-server tsohle, re qala feela tse fumanehang marang-rang. Lerung, e le hore re se ke ra fokotsa ho feto-fetoha ha maemo, ha re tlamelle lijana tse fumanehang ka ntle ho li-server ka bomong tse nang le kernel e ncha, empa re qalisa mabotho ohle ntle le mokhelo. Ka lehlohonolo, ts'ebetso ea moo e bonolo ho feta ka li-server tse tloaelehileng. Mohlala, lijana tse se nang palo li ka fetela ho seva se seng nakong ea ho qala bocha.

Leha ho le joalo, ho ntse ho e-na le mosebetsi o mongata, 'me ho ka nka libeke tse' maloa, 'me haeba ho na le mathata leha e le afe ka phetolelo e ncha, ho fihlela likhoeli tse' maloa. Bahlaseli ba utloisisa sena hantle, kahoo ba hloka moralo oa B.

FragmentSmack/SegmentSmack. Mokhoa oa ho lokisa

Ka lehlohonolo, bakeng sa bofokoli bo bong leano le joalo la B le teng, 'me le bitsoa Workaround. Hangata, sena ke phetoho litlhophisong tsa kernel/application tse ka fokotsang phello e ka bang teng kapa tsa felisa ka botlalo tšebeliso ea bofokoli.

Tabeng ea FragmentSmack/SegmentSmack e hlahisitsoe Phetoho e joalo:

«O ka fetola maemo a kamehla a 4MB le 3MB ho net.ipv4.ipfrag_high_thresh le net.ipv4.ipfrag_low_thresh (le balekane ba bona bakeng sa ipv6 net.ipv6.ipfrag_high_thresh le net.ipv6.ipv256.ipfrag192.ipfrag 262144 kapa 64 kBlowXNUMX) tlase. Liteko li bonts'a marotholi a manyane ho isa ho a bohlokoa ts'ebelisong ea CPU nakong ea tlhaselo ho latela lisebelisoa, litlhophiso le maemo. Leha ho le joalo, ho ka 'na ha e-ba le tšusumetso e itseng ea ts'ebetso ka lebaka la ipfrag_high_thresh=XNUMX bytes, kaha likhechana tse peli feela tsa XNUMXK li ka kena moleng oa ho kopanya hape ka nako. Ka mohlala, ho na le kotsi ea hore likopo tse sebetsang ka lipakete tse kholo tsa UDP li tla robeha".

Mekhahlelo ka bo eona litokomaneng tsa kernel e hlalositsoe ka tsela e latelang:

ipfrag_high_thresh - LONG INTEGER
    Maximum memory used to reassemble IP fragments.

ipfrag_low_thresh - LONG INTEGER
    Maximum memory used to reassemble IP fragments before the kernel
    begins to remove incomplete fragment queues to free up resources.
    The kernel still accepts new fragments for defragmentation.

Ha re na li-UDP tse kholo litšebeletsong tsa tlhahiso. Ha ho sephethe-phethe se arohaneng ho LAN; ho na le sephethephethe se arohaneng ho WAN, empa ha se bohlokoa. Ha ho na matšoao - o ka qala Workaround!

FragmentSmack/SegmentSmack. Mali a pele

Bothata ba pele boo re kopaneng le bona ke hore ka linako tse ling li-container tsa maru li ne li sebelisa litlhophiso tse ncha hanyane feela (feela ipfrag_low_thresh), 'me ka linako tse ling li ne li sa li sebelise ho hang - li ne li oela qalong feela. Ho ne ho sa khonehe ho hlahisa bothata ka mokhoa o tsitsitseng (litlhophiso tsohle li ne li sebelisoa ka letsoho ntle le mathata leha e le afe). Ho utloisisa hore na ke hobane'ng ha setshelo se soahlamana qalong ha ho bonolo haholo: ha ho liphoso tse fumanoeng. Ho ne ho na le ntho e le 'ngoe e tiileng: ho khutlisetsa morao litlhophiso ho rarolla bothata ka ho senyeha ha setshelo.

Hobaneng ho sa lekana ho sebelisa Sysctl ho moamoheli? Setshelo se lula sebakeng sa sona sa marang-rang se inehetseng sa Namespace, kahoo bonyane karolo ea marang-rang Sysctl parameters ka setshelo e ka 'na ea fapana le moamoheli.

Litlhophiso tsa Sysctl li sebelisoa joang ka har'a setshelo? Kaha lijana tsa rona ha li na tokelo, u ke ke ua khona ho fetola maemo afe kapa afe a Sysctl ka ho kena ka sejaneng ka boeona - ha u na litokelo tse lekaneng. Ho tsamaisa lijana, leru la rona ka nako eo le ne le sebelisa Docker (hona joale podman). Mekhahlelo ea setshelo se secha e fetiselitsoe ho Docker ka API, ho kenyelletsa le litlhophiso tse hlokahalang tsa Sysctl.
Ha ke ntse ke batla liphetolelong, ho ile ha fumaneha hore Docker API ha ea ka ea khutlisa liphoso tsohle (bonyane ka mofuta oa 1.10). Ha re leka ho qala setshelo ka "docker run", qetellong re bone bonyane ho hong:

write /proc/sys/net/ipv4/ipfrag_high_thresh: invalid argument docker: Error response from daemon: Cannot start container <...>: [9] System error: could not synchronise with container process.

Boleng ba paramethara ha bo sebetse. Empa hobaneng? Hona ke hobane’ng ha e sa sebetse ka linako tse ling feela? Ho ile ha fumaneha hore Docker ha e fane ka tiiso ea tatellano eo liparamente tsa Sysctl li sebelisoang ka eona (phetolelo ea morao-rao e lekiloeng ke 1.13.1), kahoo ka linako tse ling ipfrag_high_thresh e ne e leka ho behoa ho 256K ha ipfrag_low_thresh e ntse e le 3M, ke hore, moeli o ka holimo o ne o le tlaase. ho feta moeli o ka tlaase, o lebisitseng phosong.

Ka nako eo, re ne re se re sebelisitse mochini oa rona oa ho lokisa setshelo hape kamora ho qala (ho homisa setshelo kamora moo sehatsetsi sa sehlopha le ho phethahatsa litaelo sebakeng sa mabitso sa setshelo ka ip marang-rang), hape re kentse ho ngola liparamente tsa Sysctl karolong ena. Bothata bo ile ba rarolloa.

FragmentSmack/SegmentSmack. Mali a Pele 2

Pele re e-ba le nako ea ho utloisisa tšebeliso ea Workaround ka leru, litletlebo tsa pele tse sa tloaelehang tse tsoang ho basebelisi li ile tsa qala ho fihla. Ka nako eo, ho ne ho se ho fetile libeke tse 'maloa ho tloha ha ho qala ho sebelisoa Workaround ho li-server tsa pele. Phuputso ea pele e bontšitse hore litletlebo li ile tsa amoheloa khahlanong le litšebeletso tsa motho ka mong, eseng li-server tsohle tsa litšebeletso tsena. Bothata bo boetse bo fetohile bo sa tsitsang haholo.

Pele ho tsohle, ehlile, re lekile ho khutlisa litlhophiso tsa Sysctl, empa sena ha se na phello. Liphetoho tse fapaneng ka seva le litlhophiso tsa ts'ebeliso le tsona ha lia ka tsa thusa. Restart e thusitse. Ho qala Linux bocha ha se ntho ea tlhaho joalo ka ha e ne e tloaelehile bakeng sa Windows matsatsing a khale. Leha ho le joalo, e ile ea thusa, 'me re ile ra e etsa "kernel glitch" ha re sebelisa litlhophiso tse ncha ho Sysctl. E ne e le bohlanya hakakang...

Libeke tse tharo hamorao bothata bo ile ba boela ba hlaha. Tlhophiso ea li-server tsena e ne e le bonolo haholo: Nginx ka mokhoa oa proxy/balancer. Sephethephethe se sengata. Tsebiso e ncha ea selelekela: palo ea liphoso tse 504 ho bareki e ntse e eketseha letsatsi le letsatsi (Gateway Timeout). Kerafo e bonts'a palo ea liphoso tse 504 ka letsatsi bakeng sa ts'ebeletso ena:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Liphoso tsohle li batla li tšoana le backend - mabapi le e leng marung. Kerafo ea tšebeliso ea memori bakeng sa likhechana tsa sephutheloana sebakeng sena se ka morao e ne e shebahala tjena:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Ena ke e 'ngoe ea liponahatso tse hlakileng tsa bothata ho li-graph tsa sistimi e sebetsang. Lerung, ka nako e ts'oanang, bothata bo bong ba marang-rang bo nang le litlhophiso tsa QoS (Traffic Control) bo ne bo tsitsitse. Ho graph ea tšebeliso ea memori bakeng sa likhechana tsa pakete, e ne e shebahala e ts'oana hantle:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Maikutlo a ne a le bonolo: haeba a shebahala a tšoana lirapeng, joale a na le lebaka le tšoanang. Ho feta moo, mathata leha e le afe a mofuta ona oa ho hopola ha a fumanehe haholo.

Moko-taba oa bothata bo tsitsitseng ke hore re sebelisitse fq pakete scheduler ka litlhophiso tsa kamehla ho QoS. Ka ho sa feleng, bakeng sa khokahanyo e le 'ngoe, e u lumella ho eketsa lipakete tse 100 moleng,' me likhokahano tse ling, maemong a khaello ea mocha, li ile tsa qala ho koala mokoloko ho fihlela matla. Tabeng ena, lipakete li theoha. Lipalong tsa tc (tc -s qdisc) e ka bonoa tjena:

qdisc fq 2c6c: parent 1:2c6c limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
 Sent 454701676345 bytes 491683359 pkt (dropped 464545, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  1024 flows (1021 inactive, 0 throttled)
  0 gc, 0 highprio, 0 throttled, 464545 flows_plimit

"464545 flows_plimit" ke lipakete tse theohileng ka lebaka la ho feta moeli oa khokahanyo e le 'ngoe, 'me "theoha 464545" ke kakaretso ea liphutheloana tsohle tse theohileng tsa kemiso ena. Ka mor'a ho eketsa bolelele ba queu ho 1 sekete le ho tsosolosa lijana, bothata bo ile ba khaotsa ho etsahala. U ka lula fatše 'me ua noa smoothie.

FragmentSmack/SegmentSmack. Madi a ho Qetela

Taba ea pele, likhoeli tse 'maloa ka mor'a phatlalatso ea bofokoli ka har'a kernel, qetellong ho ile ha hlaha tokiso ea FragmentSmack (e re ke u hopotse hore hammoho le phatlalatso ka Phato, ho ile ha lokolloa tokiso ea SegmentSmack feela), e re fileng monyetla oa ho tlohela Workaround, e leng se ileng sa re bakela mathata a mangata. Nakong ena, re ne re se re khonne ho fetisetsa tse ling tsa li-server ho kernel e ncha, 'me joale re ne re tlameha ho qala ho tloha qalong. Hobaneng re ile ra ntlafatsa kernel ntle le ho emela tokiso ea FragmentSmack? Taba ke hore ts'ebetso ea ho itšireletsa khahlanong le bofokoli bona e tsamaellana (mme e kopantsoe) le ts'ebetso ea ho nchafatsa CentOS ka boeona (e nkang nako e telele ho feta ho nchafatsa kernel feela). Ntle le moo, SegmentSmack ke tlokotsi e kotsi le ho feta, 'me tokiso ea eona e ile ea hlaha hang-hang, kahoo e ne e utloahala. Leha ho le joalo, ha rea ​​​​ka ra khona ho nchafatsa kernel ho CentOS hobane ts'oaetso ea FragmentSmack, e hlahileng nakong ea CentOS 7.5, e ne e tsitsitse feela ka mofuta oa 7.6, kahoo re ile ra tlameha ho emisa ntlafatso ho 7.5 mme re qale hape ka ntlafatso ea 7.6. 'Me sena se boetse se etsahala.

Taba ea bobeli, litletlebo tse sa tloaelehang tsa basebelisi mabapi le mathata li khutletse ho rona. Hona joale re se re tseba hantle hore kaofela li amana le ho kenya lifaele ho tloha ho bareki ho ea ho tse ling tsa lisebelisoa tsa rona. Ho feta moo, palo e nyane haholo ea li-uploads tse tsoang ho bongata bohle li ile tsa feta har'a li-server tsena.

Joalo ka ha re hopola paleng e kaholimo, ho khutlela morao Sysctl ha hoa ka ha thusa. Reboot e thusitse, empa ka nakoana.
Lipelaelo tse mabapi le Sysctl ha lia ka tsa tlosoa, empa lekhetlong lena ho ne ho hlokahala hore ho bokelloe boitsebiso bo bongata kamoo ho ka khonehang. Hape ho ne ho e-na le khaello e kholo ea bokhoni ba ho hlahisa bothata ba ho kenya mochine ho moreki hore a ithute ka nepo se etsahalang.

Tlhahlobo ea lipalo-palo tsohle tse teng le likutu ha ea ka ea re atametsa ho utloisisa se neng se etsahala. Ho ne ho e-na le khaello e matla ea bokhoni ba ho hlahisa bothata hape e le ho "ikutloa" khokahano e itseng. Qetellong, bahlahisi, ba sebelisa mofuta o khethehileng oa kopo, ba khonne ho fihlela ho hlahisa mathata ka mokhoa o tsitsitseng sesebelisoa sa teko ha ba hokahantsoe ka Wi-Fi. Ena e bile khatelo-pele lipatlisisong. Moreki o hokahane le Nginx, e neng e le proxy ho backend, e neng e le ts'ebeliso ea rona ea Java.

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Puisano ea mathata e ne e le tjena (e lokisitsoe ka lehlakoreng la moemeli oa Nginx):

  1. Moreki: kopa ho fumana leseli mabapi le ho jarolla faele.
  2. Seva ea Java: karabo.
  3. Moreki: POST ka faele.
  4. Seva ea Java: phoso.

Ka nako e ts'oanang, seva sa Java se ngolla logi hore li-byte tse 0 tsa data li amohetsoe ho tsoa ho moreki, mme moemeli oa Nginx o ngola hore kopo e nkile metsotsoana e fetang 30 (metsotsoana e 30 ke nako ea ts'ebeliso ea bareki). Ke hobane'ng ha nako e felile 'me ke hobane'ng ha li-byte tse 0? Ho latela pono ea HTTP, ntho e 'ngoe le e' ngoe e sebetsa kamoo e lokelang, empa POST e nang le faele e bonahala e nyamela marang-rang. Ho feta moo, e nyamela pakeng tsa moreki le Nginx. Ke nako ea ho itlhomella ka Tcpdump! Empa pele u hloka ho utloisisa tlhophiso ea marang-rang. Nginx proxy e ka morao ho L3 balancer NFware. Tunneling e sebelisetsoa ho fana ka lipakete ho tloha ho balancer ea L3 ho ea ho seva, e leng se eketsang lihlooho tsa eona ho lipakete:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Tabeng ena, marang-rang a tla ho seva sena ka mokhoa oa sephethephethe sa Vlan-tagged, se eketsang masimo a sona ho lipakete:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

'Me sephethephethe sena le sona se ka aroloa (e leng eona karolo e nyane ea sephethephethe se kenang se arohaneng seo re buileng ka sona ha re hlahloba likotsi tse tsoang ho Workaround), e fetolang litaba tsa lihlooho:

Hlokomela bofokoli bo bakang mathata a mosebetsi. Karolo ea 1: FragmentSmack/SegmentSmack

Hape hape: lipakete li koaletsoe ka tag ea Vlan, e kentsoeng ka kotopo, e arohane. Ho utloisisa hamolemo hore na sena se etsahala joang, ha re lateleng tsela ea pakete ho tloha ho moreki ho ea ho moemeli oa Nginx.

  1. Pakete e fihla ho balancer ea L3. Bakeng sa ho tsamaisa ka nepo ka har'a setsi sa data, pakete e kentsoe ka har'a kotopo ebe e romelloa kareteng ea marang-rang.
  2. Kaha lihlooho tsa pakete + tsa kotopo ha li kenelle ho MTU, pakete e khaola likotoana ebe e romeloa ho marang-rang.
  3. Phetoho ka mor'a tekanyo ea L3, ha e amohela pakete, e eketsa tag ea Vlan ho eona ebe e e romela.
  4. Phetoho e ka pel'a moemeli oa Nginx e bona (e ipapisitse le litlhophiso tsa boema-kepe) hore seva e lebelletse pakete e kentsoeng ea Vlan, ka hona e e romella joalo ka ha e le joalo, ntle le ho tlosa tag ea Vlan.
  5. Linux e nka likhechana tsa liphutheloana ka bomong ebe e li kopanya hore e be sephutheloana se le seng se seholo.
  6. Ka mor'a moo, pakete e fihla sebopeho sa Vlan, moo ho tlosoang lera la pele ho eona - Vlan encapsulation.
  7. Ka mor'a moo, Linux e e romela ho Tunnel interface, moo lera le leng le tlosoang ho eona - Tunnel encapsulation.

Bothata ke ho fetisa tsena tsohle e le liparamente ho tcpdump.
A re qaleng ho tloha qetellong: na ho na le lipakete tse hloekileng (ntle le lihlooho tse sa hlokahaleng) lipakete tsa IP tse tsoang ho bareki, tse nang le vlan le tunnel encapsulation e tlosoa?

tcpdump host <ip клиента>

Che, ho ne ho se na liphutheloana tse joalo ho seva. Kahoo bothata bo tlameha ho ba teng pele ho nako. Na ho na le lipakete tse nang le Vlan encapsulation feela e tlositsoeng?

tcpdump ip[32:4]=0xx390x2xx

0xx390x2xx ke aterese ea IP ea bareki ka sebopeho sa hex.
32:4 - aterese le bolelele ba sebaka seo SCR IP e ngotsoeng ka har'a pakete ea Tunnel.

Aterese ea tšimo e ne e lokela ho khethoa ka matla a sehlōhō, kaha Inthaneteng ba ngola ka 40, 44, 50, 54, empa ho ne ho se na aterese ea IP moo. U ka boela ua sheba e 'ngoe ea lipakete ka hex (parameter ea -xx kapa -XX ho tcpdump) 'me u bale aterese ea IP eo u e tsebang.

Na ho na le likhechana tsa lipakete ntle le Vlan le Tunnel encapsulation e tlositsoeng?

tcpdump ((ip[6:2] > 0) and (not ip[6] = 64))

Boselamose bona bo tla re bontša likhechana tsohle, ho kenyelletsa le ea ho qetela. Mohlomong, ntho e ts'oanang e ka hloekisoa ke IP, empa ha kea ka ka leka, hobane ha ho na lipakete tse joalo tse ngata, 'me tseo ke neng ke li hloka li ne li fumanoa habonolo ka phallo e akaretsang. Ke tsena:

14:02:58.471063 In 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), length 1516: (tos 0x0, ttl 63, id 53652, offset 0, flags [+], proto IPIP (4), length 1500)
    11.11.11.11 > 22.22.22.22: truncated-ip - 20 bytes missing! (tos 0x0, ttl 50, id 57750, offset 0, flags [DF], proto TCP (6), length 1500)
    33.33.33.33.33333 > 44.44.44.44.80: Flags [.], seq 0:1448, ack 1, win 343, options [nop,nop,TS val 11660691 ecr 2998165860], length 1448
        0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ...........A....
        0x0010: 4500 05dc d194 2000 3f09 d5fb 0a66 387d E.......?....f8}
        0x0020: 1x67 7899 4500 06xx e198 4000 3206 6xx4 [email protected].
        0x0030: b291 x9xx x345 2541 83b9 0050 9740 0x04 .......A...P.@..
        0x0040: 6444 4939 8010 0257 8c3c 0000 0101 080x dDI9...W.......
        0x0050: 00b1 ed93 b2b4 6964 xxd8 ffe1 006a 4578 ......ad.....jEx
        0x0060: 6966 0000 4x4d 002a 0500 0008 0004 0100 if..MM.*........

14:02:58.471103 In 00:de:ff:1a:94:11 ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 63, id 53652, offset 1480, flags [none], proto IPIP (4), length 40)
    11.11.11.11 > 22.22.22.22: ip-proto-4
        0x0000: 0000 0001 0006 00de fb1a 9441 0000 0800 ...........A....
        0x0010: 4500 0028 d194 00b9 3f04 faf6 2x76 385x E..(....?....f8}
        0x0020: 1x76 6545 xxxx 1x11 2d2c 0c21 8016 8e43 .faE...D-,.!...C
        0x0030: x978 e91d x9b0 d608 0000 0000 0000 7c31 .x............|Q
        0x0040: 881d c4b6 0000 0000 0000 0000 0000 ..............

Tsena ke likhechana tse peli tsa sephutheloana se le seng (ID e tšoanang 53652) e nang le foto (lentsoe Exif le bonahala ka har'a sephutheloana sa pele). Ka lebaka la hore ho na le liphutheloana boemong bona, empa eseng ka foromo e kopantsoeng ka har'a lithōle, bothata bo hlakile ka kopano. Qetellong ho na le bopaki ba litokomane tsa sena!

Decoder ea pakete ha ea ka ea senola mathata leha e le afe a ka thibelang mohaho. E lekile mona: hpd.gasmi.net. Qalong, ha u leka ho kenya ntho e itseng moo, decoder ha e rate sebopeho sa pakete. Ho ile ha fumaneha hore ho na le li-octet tse ling tse peli lipakeng tsa Srcmac le Ethertype (tse sa amaneng le tlhahisoleseling ea sekhechana). Kamora ho li tlosa, decoder e ile ea qala ho sebetsa. Leha ho le joalo, ha ea ka ea bontša mathata.
Eng kapa eng eo motho a ka e buang, ha ho letho le leng le ileng la fumanoa haese li-Systl. Sohle se neng se setse e ne e le ho fumana mokhoa oa ho khetholla li-server tsa bothata e le hore u utloisise sekala le ho etsa qeto ka liketso tse ling. Khaontara e hlokahalang e fumanoe kapele ho lekana:

netstat -s | grep "packet reassembles failed”

E boetse e ho snmpd tlasa OID=1.3.6.1.2.1.4.31.1.1.16.1 (ipSystemStatsReasmFails).

"Palo ea mefokolo e fumanoeng ke algorithm ea ho kopanya hape IP (ka lebaka lefe kapa lefe: nako e felile, liphoso, joalo-joalo)."

Har'a sehlopha sa li-server tseo bothata bo ithutoang ho tsona, ho tse peli k'hamphani ena e ile ea eketseha ka potlako, ho tse peli butle-butle, 'me ho tse ling tse peli ha ea ka ea eketseha ho hang. Ho bapisa matla a k'haontareng ena le matla a liphoso tsa HTTP ho seva sa Java ho senotse khokahano. Ke hore meter e ne e ka behoa leihlo.

Ho ba le sesupo se tšepahalang sa mathata ho bohlokoa haholo e le hore o ka tseba ka nepo hore na ho khutlisa Sysctl hoa thusa, hobane ho tloha pale e fetileng rea tseba hore sena se ke ke sa utloisisoa hanghang ho tsoa ts'ebelisong. Letšoao lena le tla re lumella ho tseba libaka tsohle tsa mathata tlhahisong pele basebelisi ba e fumana.
Ka mor'a ho khutlisa Sysctl, liphoso tsa ho shebella li ile tsa emisa, kahoo sesosa sa mathata se ile sa pakoa, hammoho le taba ea hore rollback e thusa.

Re ile ra khutlisa litlhophiso tsa karohano ho li-server tse ling, moo tlhahlobo e ncha e ileng ea sebetsa, 'me kae-kae re ile ra fana ka mohopolo o mongata oa likhechana ho feta oo pele e neng e le oa kamehla (ena e ne e le lipalo-palo tsa UDP, tahlehelo ea karolo e itseng ea eona e neng e sa bonahale khahlano le semelo se akaretsang) .

Lipotso tsa bohlokoa ka ho fetisisa

Hobaneng ha lipakete li arotsoe ho lekanse ea rona ea L3? Boholo ba lipakete tse tsoang ho basebelisi ho ea ho li-balancers ke SYN le ACK. Boholo ba liphutheloana tsena bo nyane. Empa kaha karolo ea lipakete tse joalo e kholo haholo, khahlanong le semelo sa bona ha rea ​​ka ra hlokomela ho ba teng ha lipakete tse kholo tse ileng tsa qala ho arohana.

Lebaka e ne e le sengoloa sa tlhophiso se robehileng advmss ho li-server tse nang le li-interfaces tsa Vlan (ho ne ho e-na le li-server tse fokolang haholo tse nang le sephethephethe sa tlhahiso ka nako eo). Advmss e re lumella hore re fetisetse ho moreki tlhahisoleseding eo lipakete ka lehlakoreng la rona e lokelang ho ba tse nyenyane ka boholo e le hore ka mor'a ho khomarela lihlooho tsa kotopo ho tsona li se ke tsa tlameha ho arohana.

Hobaneng ha Sysctl rollback e sa ka ea thusa, empa reboot e ile ea thusa? Rolling back Sysctl e fetotse palo ea memori e fumanehang bakeng sa ho kopanya liphutheloana. Ka nako e ts'oanang, ho hlakile hore 'nete ea memori e khaphatsehang bakeng sa likhechana e lebisitse ho theoheng ha likhokahano, e leng se ileng sa etsa hore likhechana li liehe nako e telele moleng. Ke hore, mokhoa ona o ile oa tsamaea ka lipotoloho.
Reboot e ile ea hlakola memori mme tsohle li khutlela ho tatellano.

Na ho ne ho khoneha ho etsa ntle le Workaround? E, empa ho na le kotsi e kholo ea ho siea basebelisi ntle le ts'ebeletso nakong ea tlhaselo. Ha e le hantle, tšebeliso ea Workaround e ile ea fella ka mathata a sa tšoaneng, ho kenyelletsa le ho fokotseha ha e 'ngoe ea litšebeletso bakeng sa basebelisi, empa leha ho le joalo re lumela hore liketso li ne li nepahetse.

Ke leboha haholo Andrey Timofeev (atimofeyev) bakeng sa thuso ea ho etsa lipatlisiso, hammoho le Alexey Krenev (sesebelisoax) - bakeng sa mosebetsi oa titanic oa ho nchafatsa Centos le li-kernel ho li-server. Ts'ebetso eo tabeng ena e neng e tlameha ho qalisoa ho tloha qalong ka makhetlo a 'maloa, ke ka lebaka leo e ileng ea hula ka likhoeli tse ngata.

Source: www.habr.com

Eketsa ka tlhaloso