Ndi2019, uye isu hatisati tave neyakajairwa mhinduro yekuunganidza log muKubernetes. Muchinyorwa chino, tinoda, tichishandisa mienzaniso kubva kumaitiro chaiwo, kugovera zvatinotsvaga, matambudziko akasangana nemhinduro dzawo.
Nekudaro, chekutanga, ini ndichachengeta kuti vatengi vakasiyana vanonzwisisa zvinhu zvakasiyana zvakanyanya nekuunganidza matanda:
- mumwe munhu anoda kuona chengetedzo nekuongorora matanda;
- mumwe munhu - centralized kutema kwese kwezvivakwa;
- uye kune vamwe, zvakakwana kuunganidza chete matanda ekushandisa, kusasanganisa, semuenzaniso, vanoenzanisa.
Pazasi pane kuchekwa pazasi nezve mashandisiro atakaita akasiyana "zvishuwo zvinyorwa" uye ndeapi matambudziko atakasangana nawo.
Dzidziso: nezve maturusi ekutema matanda
Background pane zvikamu zvelogging system
Kutema matanda kwakauya kure, semhedzisiro yekuti ndedzipi nzira dzekuunganidza nekuongorora matanda dzakagadzirwa, ndizvo zvatinoshandisa nhasi. Kare muma1950s, Fortran yakaunza analogue yeyakajairwa yekupinda / kubuda hova, iyo yakabatsira mugadziri wepurogiramu kugadzirisa chirongwa chake. Aya aive ekutanga matanda emakombuta aiita kuti hupenyu huve nyore kune vanogadzira zvirongwa zvenguva idzodzo. Nhasi tinoona mavari chikamu chekutanga chekutema matanda - tsime kana "mugadziri" wematanda.
Sayenzi yekombuta haina kumira: macomputer network akaonekwa, masumbu ekutanga ... Complex masisitimu anoumbwa nemakomputa akawanda akatanga kushanda. Iye zvino vatariri vehurongwa vakamanikidzwa kuunganidza matanda kubva kumakina akati wandei, uye muzviitiko zvakakosha vaigona kuwedzera OS kernel meseji kana vachida kuongorora kutadza kwehurongwa. Kurondedzera centralized log collection systems, mukutanga 2000s yakabudiswa , iyo yakamisa kure_syslog. Aya ndiwo mabudiro akaita chimwe chikamu chakakosha: log collector uye kuchengetedza kwavo.
Nekuwedzera kwehuwandu hwematanda uye kuiswa kwakapararira kwewebhu tekinoroji, mubvunzo wakamuka wekuti ndeapi matanda anoda kuratidzwa zviri nyore kune vashandisi. Maturusi akareruka ekunyaradza (awk/sed/grep) akatsiviwa nemamwe epamberi log vataridzi - chikamu chechitatu.
Pamusana pekuwedzera kwehuwandu hwematanda, chimwe chinhu chakave chakajeka: matanda anodiwa, asi kwete ose. Uye matanda akasiyana anoda mazinga akasiyana ekuchengetedza: mamwe anogona kurasika muzuva, asi mamwe anoda kuchengetwa kwemakore mashanu. Saka, chikamu chekusefa uye kufambisa data kuyerera chakawedzerwa kune iyo yekutema matanda - ngatiidaidze. sefa.
Kuchengetera kwaitawo kusvetuka kukuru: kubva kumafaira enguva dzose kuenda kune ehukama dhatabhesi, uyezve kune magwaro-anotariswa kuchengetedza (semuenzaniso, Elasticsearch). Saka chengetedzo yakaparadzaniswa nemuunganidzi.
Pakupedzisira, iyo pfungwa chaiyo yegiyo yakawedzera kusvika kune imwe mhando yeabstract rwizi rwezviitiko izvo isu tinoda kuchengetedza nhoroondo. Kana kuti, kana iwe uchida kuferefeta kana kudhirowa rondedzero yekuongorora ...
Nekuda kweizvozvo, munguva pfupi pfupi yenguva, kuunganidzwa kwelogi kwakagadziridzwa kuita yakakosha subsystem, iyo inogona kunzi chimwe chezvikamu muBig Data.
Kana pane imwe nguva madhindindi akajairwa angave akakwana kune "hurongwa hwekutema miti," ikozvino mamiriro achinja zvakanyanya.
Kubernetes uye logs
Kubernetes paakauya kune zvivakwa, dambudziko raivepo rekuunganidza matanda harina kuripfuura. Mune dzimwe nzira, zvakatonyanya kurwadza: kubata chikuva chezvivakwa hakuna kungorerutswa, asiwo kwakaoma panguva imwe chete. Mazhinji masevhisi ekare atanga kutamira kune microservices. Mumamiriro ezvinhu ematanda, izvi zvinoratidzwa muhuwandu huri kukura hwezvinyorwa zvinyorwa, mararamiro avo anokosha, uye kukosha kwekutevera hukama hwezvinhu zvose zvehurongwa kuburikidza nematanda ...
Ndichitarisa kumberi, ndinogona kutaura kuti ikozvino, zvinosuruvarisa, hapana yakamisikidzwa yekutema matanda sarudzo yeKubernetes inogona kuenzaniswa zvakanaka nevamwe vese. Zvirongwa zvinonyanya kufarirwa munharaunda ndeizvi:
- mumwe munhu anosunungura murwi EFK (Elasticsearch, Fluentd, Kibana);
- mumwe munhu ari kuyedza ichangobva kuburitswa kana kushandisa ;
- Π½Π°Ρ (uye pamwe kwete isu chete?..) Ini ndinogutsikana zvakanyanya nekukura kwangu - ...
Semutemo, isu tinoshandisa anotevera masumbu mumasumbu eK8s (yekuzvigadzirira mhinduro):
- ;
- .
Nekudaro, ini handisi kuzogara pane mirairo yekumisikidzwa kwavo nekugadzirisa. Pane kudaro, ini ndichaisa pfungwa pazvikanganiso zvavo uye mamwe mhedziso yepasi rose pamusoro pemamiriro ezvinhu nematanda muhuwandu.
Dzidzira nematanda muK8s
"Everyday logs", vangani venyu varipo?..
Centralized kuunganidzwa kwematanda kubva kune yakakura zvakaringana zvivakwa zvinoda zvakati wandei zviwanikwa, izvo zvichashandiswa mukuunganidza, kuchengetedza uye kugadzirisa matanda. Panguva yekushanda kwezvirongwa zvakasiyana-siyana, takatarisana nezvinodiwa zvakasiyana-siyana uye matambudziko ekushanda aibva mazviri.
Ngatiedzei ClickHouse
Ngatitarisei nzvimbo yekuchengetera yepakati pane purojekiti ine application inogadzira matanda zvakanyatsoshanda: inodarika 5000 mitsetse pasekondi. Ngatitange kushanda nematanda ake, tichiawedzera kuClickHouse.
Pakangodiwa nguva chaiyo chaiyo, iyo 4-core server neClickHouse inenge yatove yakawandisa pane disk subsystem:
Iyi mhando yekurodha imhaka yekuti tiri kuyedza kunyora muClickHouse nekukurumidza sezvinobvira. Uye iyo dhatabhesi inobata kune izvi nekuwedzera dhisiki mutoro, izvo zvinogona kukonzera zvinotevera zvikanganiso:
DB::Exception: Too many parts (300). Merges are processing significantly slower than inserts
Point ndechokuti muClickHouse (ivo vane log data) vane matambudziko avo panguva yekunyora mabasa. Iyo data yakaiswa mavari inogadzira chikamu chechinguvana, icho chinozobatanidzwa netafura huru. Nekuda kweizvozvo, kurekodha kunoratidza kuve kwakanyanya kudiwa pa diski, uye zvakare iri pasi pemiganho yatakagamuchira chiziviso pamusoro apa: hapana anopfuura mazana matatu madiki anogona kubatanidzwa musekondi imwe (chaizvoizvo, izvi 1 kuisa. pasekondi).
Kuti udzivise hunhu uhu, muzvidimbu zvakakura sezvinobvira uye kwete kupfuura 1 nguva ose 2 seconds. Nekudaro, kunyora mukuputika kukuru kunoratidza kuti isu tinofanirwa kunyora zvishoma kazhinji muClickHouse. Izvi, zvakare, zvinogona kutungamirira kune buffer kufashukira uye kurasikirwa kwematanda. Mhinduro ndeyekuwedzera Fluentd buffer, asi ipapo iyo ndangariro kushandiswa kuchawedzerawo.
taura pfungwa: Chimwe chinonetsa chemhinduro yedu neClickHouse yaive nechokuita nenyaya yekuti kupatsanura mune yedu (loghouse) kunoitwa kuburikidza nematafura ekunze akabatana. . Izvi zvinotungamira kune chokwadi chekuti kana sampling hombe dzenguva, yakawandisa RAM inodiwa, sezvo metatable iterates kuburikidza nezvikamu zvese - kunyangwe izvo zviri pachena kuti hazvina data rinodiwa. Nekudaro, ikozvino nzira iyi inogona kuziviswa zvakachengeteka kusashanda kune zvazvino shanduro dzeClickHouse (c ).
Nekuda kweizvozvo, zvinova pachena kuti haisi purojekiti yese ine zviwanikwa zvakakwana zvekuunganidza matanda munguva chaiyo muClickHouse (kunyanya, kugovera kwavo hakuzokodzeri). Mukuwedzera, iwe unofanirwa kushandisa Π°ΠΊΠΊΡΠΌΡΠ»ΡΡΠΎΡ, kwatinozodzoka gare gare. Nyaya inotsanangurwa pamusoro apa ndeyechokwadi. Uye panguva iyoyo hatina kukwanisa kupa mhinduro yakavimbika uye yakagadzikana yaizoenderana nemutengi uye kutibvumira kuunganidza matanda nekunonoka kudiki ...
Zvakadini neElasticsearch?
Elasticsearch inozivikanwa kubata basa rinorema. Ngatiedzei muchirongwa chimwe chete. Zvino mutoro unotaridzika seizvi:
Elasticsearch yakakwanisa kugaya iyo data rwizi, zvisinei, kunyora mavhoriyamu akadaro kwairi kunoshandisa zvakanyanya CPU. Izvi zvinosarudzwa nekuronga sumbu. Nehunyanzvi, iri harisi dambudziko, asi zvinozoitika kuti kungoshandisa iyo log yekuunganidza sisitimu isu tinotoshandisa anenge 8 cores uye tine chimwe chekuwedzera chakaremerwa chikamu muhurongwa...
Pazasi mutsara: iyi sarudzo inogona kururamiswa, asi chete kana purojekiti yakakura uye manejimendi ayo akagadzirira kushandisa zvakakosha zviwanikwa pane yepakati matanda system.
Ipapo mubvunzo wepanyama unomuka:
Ndeapi matanda anodiwa chaizvo?
Ngatiedzei kushandura nzira yacho pachayo: matanda anofanira panguva imwe chete achidzidzisa uye kwete kuvhara mumwe nomumwe chiitiko mu system.
Ngatitii tine chitoro chendaneti chakabudirira. Ndezvipi matanda zvakakosha? Kuunganidza ruzivo rwakawanda sezvinobvira, somuenzaniso, kubva pamusuwo wekubhadhara, ipfungwa huru. Asi haasi ese matanda kubva kumufananidzo slicing sevhisi mune yechigadzirwa catalog akakosha kwatiri: zvikanganiso chete uye yepamberi yekutarisisa inokwana (semuenzaniso, iyo muzana ye500 zvikanganiso zvinogadzirwa nechikamu ichi).
Saka takasvika pakugumisa kuti kutema miti nepakati hakuiti nguva dzose kwakarurama. Kazhinji kazhinji mutengi anoda kuunganidza matanda ese munzvimbo imwechete, kunyangwe hazvo, kubva parogi rese, chete 5% ine mameseji akakosha kune bhizinesi anodiwa:
- Dzimwe nguva zvakakwana kugadzirisa, toti, saizi chete yegiyo regi uye muunganidzi wekukanganisa (semuenzaniso, Sentry).
- Chiziviso chemhosho uye logi hombe yenzvimbo pachayo inogona kunge yakakwana kuongorora zviitiko.
- Isu takanga tine mapurojekiti aiitwa nemayedzo anoshanda chete uye masisitimu ekuunganidza zvikanganiso. Mugadziri aisada matanda akadaro - ivo vakaona zvese kubva pakukanganisa.
Mufananidzo kubva muupenyu
Imwe nyaya inogona kushanda semuenzaniso wakanaka. Takagamuchira chikumbiro kubva kuchikwata chekuchengetedza chemumwe wevatengi vedu anga achitoshandisa mhinduro yekutengesa iyo yakagadziridzwa kare kusati kwaunzwa Kubernetes.
Zvaive zvakakodzera "kuita shamwari" yepakati pekuunganidza log system neiyo kambani yekuona dambudziko sensor - QRadar. Iyi sisitimu inogona kugamuchira matanda kuburikidza ne syslog protocol uye inotora kubva kuFTP. Nekudaro, hazvina kuita nekukurumidza kuibatanidza neremote_syslog plugin yekunyatsonzwisisa (sezvazvakaita, ). Matambudziko nekumisikidza QRadar akazove kudivi retimu yekuchengetedza yemutengi.
Nekuda kweizvozvo, chikamu cheiyo bhizinesi-yakakosha matanda yakaiswa kuFTP QRadar, uye chimwe chikamu chakadzoserwa kuburikidza neiri kure syslog zvakananga kubva kumanodhi. Nokuda kwaizvozvi takatonyorawo - zvichida zvichabatsira mumwe munhu kugadzirisa dambudziko rakafanana ... Nekuda kwechirongwa chakaguma, mutengi pachake akagamuchira uye akaongorora matanda anokosha (achishandisa midziyo yake yaanofarira), uye takakwanisa kuderedza mari yekugadzira matanda, kuchengetedza chete mwedzi wapfuura.
Mumwe muenzaniso unoratidza zvausingafaniri kuita. Imwe yevatengi vedu yekugadzirisa cheumwe neumwe zviitiko zvinouya kubva kumushandisi, zvakaitwa multiline isina kugadzirwa ruzivo mu log. Sezvaungafungidzira, matanda akadaro ainyanya kunetsa kuverenga nekuchengeta.
Maitiro ematanda
Mienzaniso yakadaro inotungamira kumhedziso yekuti kunze kwekusarudza iyo log yekuunganidza system, iwe unofanirwa gadzirawo matanda pachawo! Ndezvipi zvinodiwa pano?
- Malog anofanirwa kunge ari mufomati inoverengwa nemuchina (semuenzaniso, JSON).
- Marogi anofanirwa kuve akabatana uye nekugona kushandura dhigirii rekutema matanda kuitira kugadzirisa matambudziko angangoitika. Panguva imwecheteyo, munzvimbo dzekugadzira iwe unofanirwa kumhanyisa masisitimu ane nhanho yekutema senge yambiro kana mhosho.
- Marogi anofanirwa kuve akajairwa, ndiko kuti, muchinhu chelogi, mitsetse yese inofanirwa kuve nemhando yemunda yakafanana.
Miti isina kurongeka inogona kutungamirira kumatambudziko nekutakura matanda mukuchengetedza uye kumira kwakakwana mukugadzirisa kwavo. Semufananidzo, heino muenzaniso nemhosho 400, iyo yakawanda yakanyatso sangana nayo mumatanda akajeka:
2019-10-29 13:10:43 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"
Iko kukanganisa kunoreva kuti uri kutumira munda une mhando isina kugadzikana kune index ine mepu yakagadzirwa. Muenzaniso wakapfava imunda mune nginx log ine shanduko $upstream_status. Inogona kuva nenhamba kana tambo. Semuyenzaniso:
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "17ee8a579e833b5ab9843a0aca10b941", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staffs/265.png", "protocol": "HTTP/1.1", "status": "200", "body_size": "906", "referrer": "https://example.com/staff", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.001", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "127.0.0.1:9000", "upstream_status": "200", "upstream_response_length": "906", "location": "staff"}
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "47fe42807f2a7d8d5467511d7d553a1b", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staff", "protocol": "HTTP/1.1", "status": "200", "body_size": "2984", "referrer": "-", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.010", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "10.100.0.10:9000, 10.100.0.11:9000", "upstream_status": "404, 200", "upstream_response_length": "0, 2984", "location": "staff"}
Matanda anoratidza kuti sevha 10.100.0.10 yakapindura nekanganiso ye404 uye chikumbiro chakatumirwa kune imwe chengetedzo yemukati. Nekuda kweizvozvo, kukosha mumatanda kwakava seizvi:
"upstream_response_time": "0.001, 0.007"
Mamiriro ezvinhu aya akajairika zvekuti anotofanira kuparadzaniswa .
Zvakadini nokuvimbika?
Pane dzimwe nguva apo matanda ese pasina kusarudzika akakosha. Uye neizvi, zvakajairwa matanda ekuunganidza zvirongwa zveK8s zvakatsanangurwa / zvakurukurwa pamusoro zvine matambudziko.
Semuenzaniso, ane ruzivo haakwanise kuunganidza matanda kubva mumidziyo yenguva pfupi. Mune imwe yemapurojekiti edu, dhatabhesi yekufambisa mudziyo yaigara kweasingasviki 4 masekondi uye ichibva yadzimwa - zvinoenderana neanofananidzira chirevo:
"helm.sh/hook-delete-policy": hook-succeeded
Nekuda kweizvi, irogi rekuita zvekutama harina kuverengerwa mudura. Zvematongerwo enyika zvinogona kubatsira munyaya iyi. before-hook-creation.
Mumwe muenzaniso ndeye Docker log kutenderera. Ngatitii pane application inoshingairira kunyora kumatanda. Mumamiriro ezvinhu akajairwa, isu tinokwanisa kugadzirisa matanda ese, asi nekukurumidza kana dambudziko richinge raonekwa - semuenzaniso, sezvakatsanangurwa pamusoro nefomati isina kururama - kugadzirisa kunomira, uye Docker inotenderera faira. Mhedzisiro ndeyekuti matanda-akakosha matanda anogona kurasika.
Ndicho chikonzero zvakakosha kuparadzanisa hova dzelogi, kupinza kutumira iyo inonyanya kukosha yakananga mukushandisa kuti ive nechokwadi chekuchengetedza kwavo. Mukuwedzera, zvingave zvisinganyanyi kukosha kugadzira zvimwe "accumulator" yematanda, iyo inogona kupona kusawanikwa kwekuchengetedza kwenguva pfupi uchichengetedza mameseji akakosha.
Pakupedzisira, hatifaniri kukanganwa izvozvo Zvakakosha kutarisa chero subsystem zvakanaka. Zvikasadaro, zviri nyore kumhanya mumamiriro ezvinhu umo mutsetse ari mune imwe nyika CrashLoopBackOff uye haatumire chero chinhu, uye izvi zvinovimbisa kurasikirwa kwemashoko anokosha.
zvakawanikwa
Muchikamu chino, hatisi kutarisa SaaS mhinduro seDatadog. Mazhinji ematambudziko anotsanangurwa pano akatogadziriswa neimwe nzira nemakambani ekutengesa ane hunyanzvi mukuunganidza matanda, asi havasi vese vanogona kushandisa SaaS nekuda kwezvikonzero zvakasiyana. (iyo huru imari uye kutevedzera 152-FZ).
Centralized log collection pakutanga inoita sebasa riri nyore, asi harina zvachose. Zvakakosha kuyeuka kuti:
- Zvinhu zvakakosha chete ndizvo zvinofanirwa kunyorwa zvakadzama, nepo kutarisa uye kuunganidza kukanganisa kunogona kugadzirwa kune mamwe masisitimu.
- Marogi mukugadzira anofanirwa kuchengetwa mashoma kuitira kuti asawedzera mutoro usingakoshi.
- Marogi anofanirwa kuverengeka muchina, akajairwa, uye aine fomati yakasimba.
- Zvechokwadi matanda anokosha anofanira kutumirwa mune imwe nzira yakasiyana, iyo inofanira kuparadzaniswa kubva kune huru.
- Izvo zvakakosha kufunga nezve log accumulator, iyo inogona kukuchengetedza kubva pakuputika kwemutoro wakakwira uye kuita kuti mutoro uri pakuchengetedza uwedzere yunifomu.
Iyi mitemo yakapfava, kana ikashandiswa kwese kwese, yaizobvumira masekete anotsanangurwa pamusoro apa kuti ashande - kunyangwe arikushaya zvakakosha zvikamu (bhatiri). Kana iwe usingatevedzeri kune misimboti yakadaro, basa racho richakutungamirira iwe nyore uye zvivakwa kune imwe yakanyanya kuremerwa (uye panguva imwechete isingashande) chikamu chehurongwa.
PS
Verenga zvakare pablog yedu:
- Β«";
- Β«";
- Β«".
Source: www.habr.com