{"id":92073,"date":"2020-08-22T19:41:56","date_gmt":"2020-08-22T17:41:56","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io"},"modified":"2020-08-22T19:41:56","modified_gmt":"2020-08-22T17:41:56","slug":"post-mortem-po-nedostupnosti-quay-io","status":"publish","type":"post","link":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io","title":{"rendered":"Post Mortem par Quay.io nepieejam\u012bbu","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Piez\u012bme:<\/b>: augusta s\u0101kum\u0101 Red Hat publiski pazi\u0146oja par pieejam\u012bbas probl\u0113mu risin\u0101\u0161anu, kas rad\u0101s iepriek\u0161\u0113j\u0101 m\u0113ne\u0161os t\u0101s pakalpojuma lietot\u0101jiem. <noindex><a rel=\"nofollow\" href=\"http:\/\/quay.io\/\">Quay.io<\/a><\/noindex> (t\u0101 pamat\u0101 ir konteineru att\u0113lu re\u0123istrs, kas uz\u0146\u0113muma \u012bpa\u0161um\u0101 non\u0101ca kop\u0101 ar CoreOS ieg\u0101di). Neatkar\u012bgi no j\u016bsu ieinteres\u0113t\u012bbas \u0161aj\u0101 pakalpojum\u0101, ir m\u0101c\u012b\u0161an\u0101s v\u0113rts ce\u013c\u0161, pa kuru uz\u0146\u0113muma SRE in\u017eenieri dev\u0101s, lai diagnostic\u0113tu un nov\u0113rstu av\u0101rijas c\u0113lo\u0146us.<\/i><\/p>\n<p><img decoding=\"async\" alt=\"Post Mortem par Quay.io nepieejam\u012bbu\" src=\"\/wp-content\/uploads\/2020\/08\/67ef7fddee25448ae68ae7f4700bb25b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n19. maij\u0101, agr\u0101 r\u012bt\u0101 (p\u0113c vasaras zieme\u013camerik\u0101\u0146u austrumu laika, EDT), serviss quay.io piedz\u012bvoja kritumu. Av\u0101rija sk\u0101ra gan quay.io klientus, gan Open Source projektus, kas izmanto quay.io k\u0101 platformu programmat\u016bras izstr\u0101dei un izplat\u012b\u0161anai. Red Hat nov\u0113rt\u0113 uztic\u012bbu gan vieniem, gan otriem.<\/p>\n<p>SRE in\u017eenieru komanda uzreiz \u0137\u0113r\u0101s pie darba un cent\u0101s p\u0113c iesp\u0113jas \u0101tr\u0101k stabiliz\u0113t Quay pakalpojuma darb\u012bbu. Tom\u0113r kam\u0113r vi\u0146i to dar\u012bja, klienti zaud\u0113ja iesp\u0113ju p\u0101rs\u016bt\u012bt jaunus att\u0113lus un tikai periodiski var\u0113ja lejupiel\u0101d\u0113t eso\u0161os. Nezin\u0101mu iemeslu d\u0113\u013c quay.io datu b\u0101ze tika blo\u0137\u0113ta p\u0113c pakalpojuma papla\u0161in\u0101\u0161anas l\u012bdz pilnai jaudai.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>\u00ab<b>Kas main\u012bj\u0101s?<\/b>\u00bb \u2014 tas ir pirmais jaut\u0101jums, ko parasti uzdod \u0161\u0101dos gad\u012bjumos. M\u0113s paman\u012bj\u0101m, ka neilgi pirms probl\u0113mas OpenShift Dedicated klasteris (uz kura darbojas quay.io) s\u0101ka atjaunin\u0101\u0161anu uz versiju 4.3.19. T\u0101 k\u0101 quay.io darbojas uz Red Hat OpenShift Dedicated (OSD), regul\u0101rie atjaunin\u0101jumi bija ikdieni\u0161\u0137a oper\u0101cija un nekad nevienu neaptur\u0113ja. Turkl\u0101t iepriek\u0161\u0113j\u0101 pusgad\u0101 m\u0113s vair\u0101kas reizes atjaunin\u0101j\u0101m Quay klasterus bez jebk\u0101d\u0101m apkalpo\u0161anas p\u0101rtraukumiem.<\/p>\n<p>Kam\u0113r m\u0113s cent\u0101mies atjaunot pakalpojuma darb\u012bbu, citi in\u017eenieri s\u0101ka gatavot jaunu OSD klasteri ar iepriek\u0161\u0113jo programmat\u016bras versiju, lai, ja nepiecie\u0161ams, visu izvietotu uz t\u0101.<\/p>\n<h2>Sak\u0146u c\u0113lo\u0146u anal\u012bze<\/h2>\n<p>\nGalvenais av\u0101rijas simptoms bija desmitiem t\u016bksto\u0161u savienojumu uz DB lav\u012bna, d\u0113\u013c kuras MySQL instants faktiski k\u013cuva nedarbsp\u0113j\u012bgs. T\u0101 rezult\u0101t\u0101 bija gr\u016bti diagnostic\u0113t probl\u0113mu. M\u0113s ievies\u0101m ierobe\u017eojumu uz maksim\u0101lo klientu savienojumu skaitu, lai pal\u012bdz\u0113tu SRE komandai nov\u0113rt\u0113t probl\u0113mu. Nek\u0101du neparastu datu b\u0101zes satiksmi nepaman\u012bj\u0101m: paties\u012bb\u0101 vairums piepras\u012bjumu bija uz las\u012b\u0161anu, bet tikai neliels skaits \u2013 uz rakst\u012b\u0161anu.<\/p>\n<p>M\u0113s ar\u012b cent\u0101mies noteikt mode\u013cus datu b\u0101zes trafika pl\u016bsm\u0101, kas var\u0113tu izrais\u012bt \u0161o lav\u012bnu. Tom\u0113r nebija iesp\u0113jams atrast nek\u0101das likumsak\u0101r\u012bbas \u017eurn\u0101los. Gaidot jauno klasteri ar OSD 4.3.18 gatav\u012bbu, m\u0113s turpin\u0101j\u0101m m\u0113\u0123in\u0101jumus palaist podus quay.io. Katru reizi, kad klasteris sasniedza pilnu jaudu, datu b\u0101ze iestr\u0113ga. Tas noz\u012bm\u0113ja, ka bija nepiecie\u0161ams restart\u0113t RDS instanci papildus visiem quay.io podiem.<\/p>\n<p>Vakara beig\u0101s m\u0113s stabiliz\u0113j\u0101m servisu las\u012b\u0161anas re\u017e\u012bm\u0101 un atsl\u0113dz\u0101m maksim\u0101lo skaitu nesvar\u012bgu funkciju (piem\u0113ram, atkritumu v\u0101k\u0161ana nosaukumu telp\u0101), lai samazin\u0101tu slodzi uz datu b\u0101zi. Iestr\u0113gumi apst\u0101j\u0101s, <b>bet iemesls t\u0101 ar\u012b netika atrasts<\/b>. Jaunais OSD klasteris bija gatavs, un m\u0113s p\u0101rc\u0113l\u0101m servisu, piesl\u0113dz\u0101m trafiku un turpin\u0101j\u0101m uzraudz\u012bbu.<\/p>\n<p>Quay.io stabila darboj\u0101s jaunaj\u0101 OSD klaster\u012b, t\u0101p\u0113c m\u0113s atgriez\u0101mies pie datu b\u0101zes \u017eurn\u0101liem, bet t\u0101 ar\u012b neizdev\u0101s atrast korel\u0101ciju, kas izskaidrotu blo\u0137\u0113jumus. OpenShift in\u017eenieri str\u0101d\u0101ja kop\u0101 ar mums, cen\u0161oties saprast, vai izmai\u0146as Red Hat OpenShift 4.3.19 var\u0113ja izrais\u012bt probl\u0113mas ar Quay. Tom\u0113r neko neatkl\u0101j\u0101m, un <b>problemu laboratorijas apst\u0101k\u013cos reproduc\u0113t neizdev\u0101s<\/b>.<\/p>\n<h2>Otrais trauc\u0113jums<\/h2>\n<p>\n28. maij\u0101, neilgi pirms pusdienlaika EDT, quay.io atkal sabruka ar tiem pa\u0161iem simptomiem: datu b\u0101zes darb\u012bba tika blo\u0137\u0113ta. Atkal m\u0113s velt\u012bj\u0101m visus sp\u0113kus izmekl\u0113\u0161anai. Vispirms bija nepiecie\u0161ams atjaunot servisa darb\u012bbu. Tom\u0113r <b>\u0161oreiz RDS restart\u0113\u0161ana un quay.io podu restart\u0113\u0161ana neko neietekm\u0113ja<\/b>: n\u0101kam\u0101 lav\u012bna savienojumu p\u0101rpl\u016bdin\u0101ja datu b\u0101zi. Bet k\u0101p\u0113c?<\/p>\n<p>Quay ir uzrakst\u012bts Python valod\u0101 un katrs pods darbojas k\u0101 vienots monol\u012bts konteiners. Konteinera izpildes vid\u0113 vienlaikus tiek izpild\u012btas daudzas paral\u0113las oper\u0101cijas. M\u0113s izmantojam bibliot\u0113ku <code>gevent<\/code> Roskomnadzora. <code>gunicorn<\/code> lai apstr\u0101d\u0101tu t\u012bmek\u013ca piepras\u012bjumus. Kad Quay sa\u0146em piepras\u012bjumu (caur m\u016bsu API vai Docker API), tam tiek piesaist\u012bts gevent darbinieks. Parasti \u0161im darbiniekam vajadz\u0113tu sazin\u0101ties ar datu b\u0101zi. Pirm\u0101 trauc\u0113juma laik\u0101 m\u0113s atkl\u0101j\u0101m, ka gevent darbinieki piek\u013cuva datu b\u0101zei, izmantojot noklus\u0113juma iestat\u012bjumus.<\/p>\n<p>\u0145emot v\u0113r\u0101 iev\u0113rojamo Quay podu skaitu un t\u016bksto\u0161iem piepras\u012bjumu sekund\u0113, lielais datu b\u0101zes savienojumu skaits teor\u0113tiski var\u0113ja noslogot MySQL eksempl\u0101ru. Pateicoties uzraudz\u012bbai, bija zin\u0101ms, ka Quay vid\u0113ji apstr\u0101d\u0101 5000 piepras\u012bjumus sekund\u0113. Aptuveni tikpat bija ar\u012b datu b\u0101zes savienojumu skaits. 5000 savienojumu p\u0101rst\u0101ja ierasto RDS eksempl\u0101ra kapacit\u0101ti (ko nevar teikt par desmit t\u016bksto\u0161iem). <b>K\u0101du iemeslu d\u0113\u013c notika negaid\u012bti piepras\u012bjumu savienojumu uzplaiksn\u012bjumi.<\/b>, tom\u0113r m\u0113s nepaman\u012bj\u0101m nek\u0101du korel\u0101ciju ar ien\u0101ko\u0161ajiem piepras\u012bjumiem.<\/p>\n<p>\u0160oreiz m\u0113s stingri izl\u0113m\u0101m atrast un nov\u0113rst probl\u0113mas avotu, nevis apmierin\u0101ties ar restart\u0113\u0161anu. Quay kodu b\u0101z\u0113 <b>tika veiktas izmai\u0146as, ierobe\u017eojot datu b\u0101zes savienojumu skaitu katram worker'iem<\/b> . \u0160is skaits k\u013cuvis par parametru konfigur\u0101cij\u0101: to var\u0113ja main\u012bt \u201clidojuma laik\u0101\u201d, neizveidojot jaunu konteineru att\u0113lu. Lai noskaidrotu, cik daudz savienojumu patiesi var apstr\u0101d\u0101t, tika veikti da\u017ei testi ar staging vid\u0113m, nosakot da\u017e\u0101das v\u0113rt\u012bbas, lai redz\u0113tu, k\u0101 tas ietekm\u0113 slodzes test\u0113\u0161anas scen\u0101rijus. Galu gal\u0101 izr\u0101d\u012bj\u0101s, ka <b>Quay s\u0101k izdot 502 k\u013c\u016bdas, kad savienojumu skaits p\u0101rsniedz 10 t\u016bksto\u0161us.<\/b><\/p>\n<p>M\u0113s nekav\u0113joties izv\u0113r\u0161am \u0161o jauno versiju ra\u017eo\u0161an\u0101 un s\u0101k\u0101m uzraudz\u012bt datu b\u0101zes savienojumu grafiku. Agr\u0101k b\u0101ze tika blo\u0137\u0113ta aptuveni p\u0113c 20 min\u016bt\u0113m. Pusstundas bez probl\u0113m\u0101m mums rad\u0101s cer\u012bba, un p\u0113c stundas \u2014 p\u0101rliec\u012bba. M\u0113s atjaunoj\u0101m rakst\u012b\u0161anas trafiku vietn\u0113 un uzs\u0101k\u0101m postmortem anal\u012bzi.<\/p>\n<p>Mums izdev\u0101s apiet blo\u0137\u0113\u0161anas probl\u0113mu, <b>m\u0113s nenoskaidroj\u0101m t\u0101s \u012bst\u0101s c\u0113lo\u0146us<\/b>. Apstiprin\u0101j\u0101s, ka t\u0101 nav saist\u012bta ar k\u0101diem izmai\u0146\u0101m OpenShift 4.3.19, jo tas pats notika ar\u012b 4.3.18 versij\u0101, kas iepriek\u0161 darboj\u0101s ar Quay bez jebk\u0101d\u0101m probl\u0113m\u0101m.<\/p>\n<p>Kl\u0101ster\u012b ac\u012bmredzot sl\u0113p\u0101s v\u0113l kaut kas.<\/p>\n<h2>Detaliz\u0113ta izp\u0113te<\/h2>\n<p>\nQuay.io se\u0161i gadus izmantoja noklus\u0113juma iestat\u012bjumus datub\u0101zes savienojumam bez jebk\u0101d\u0101m probl\u0113m\u0101m. Kas main\u012bj\u0101s? Ir skaidrs, ka visu \u0161o laiku satiksme uz quay.io nep\u0101rtraukti pieauga. M\u016bsu gad\u012bjum\u0101 viss izskat\u012bj\u0101s t\u0101, it k\u0101 tiktu sasniegta k\u0101da sliek\u0161\u0146a v\u0113rt\u012bba, kas kalpoja par trigeri savienojumu lav\u012bnai. M\u0113s turpin\u0101j\u0101m izp\u0113t\u012bt datub\u0101zes \u017eurn\u0101lus p\u0113c otr\u0101 sabrukuma, bet neatzin\u0101m nek\u0101das likumsakar\u012bbas vai ac\u012bmredzamas saiknes.<\/p>\n<p>Tikm\u0113r SRE komanda str\u0101d\u0101ja pie uzlabojumiem saist\u012bb\u0101 ar piepras\u012bjumu nov\u0113rojam\u012bbu Quay un visp\u0101r\u0113jo pakalpojuma vesel\u012bbu. <b>Tika ieviestas jaunas metrisk\u0101s un uzraudz\u012bbas paneli<\/b>, kas r\u0101da, kuras Quay da\u013cas ieg\u016bst visliel\u0101ko piepras\u012bjumu no klientiem.<\/p>\n<p>Quay.io parasti darboj\u0101s l\u012bdz 9. j\u016bnijam. No r\u012bta (p\u0113c EDT) m\u0113s atkal k\u013cuv\u0101m par lieciniekiem iev\u0113rojamam savienojumu skaita pieaugumam ar datub\u0101zi. <b>\u0160oreiz d\u012bkst\u0101ves nenotika<\/b>, jo jauns parametrs ierobe\u017eoja to skaitu un ne\u013c\u0101va p\u0101rsniegt MySQL caurlaidsp\u0113ju. Tom\u0113r apm\u0113ram pusstundas laik\u0101 daudzi lietot\u0101ji zi\u0146oja par quay.io l\u0113nu darb\u012bbu. M\u0113s \u0101tri apkopoju\u0161i visus iesp\u0113jamos datus, izmantoj\u0101m pievienotos uzraudz\u012bbas r\u012bkus. P\u0113k\u0161\u0146i par\u0101d\u012bj\u0101s likumsakar\u012bba.<\/p>\n<p><b>Pirms paaugstin\u0101juma savienojumu skaita liela da\u013ca piepras\u012bjumu tika sa\u0146emta uz App Registry API<\/b>. App Registry ir maz zina funkcija quay.io. T\u0101 \u013cauj uzglab\u0101t t\u0101das lietas k\u0101 Helm chartus un konteinerus ar bag\u0101t\u012bgu (rich) metadatu. Liel\u0101k\u0101 da\u013ca quay.io lietot\u0101ju neizmanto \u0161o funkciju, tom\u0113r to akt\u012bvi izmanto Red Hat OpenShift. OperatorHub, kas ir da\u013ca no OpenShift, uzglab\u0101 visus operatorus App Registry. \u0160ie operatori veido pamatu OpenShift darba slodzes ekosist\u0113mai un oper\u0101ciju modelim (Dienas 2 oper\u0101ciju ietvaros), kas orient\u0113ts uz partneriem.<\/p>\n<p>Katrs OpenShift 4 klasteris izmanto operatorus no ieb\u016bv\u0113t\u0101 OperatorHub, lai public\u0113tu pieejamo operatoru katalogu instal\u0101cijai un sniegtu atjaunin\u0101jumus jau instal\u0113tiem operatoriem. Pieaugot OpenShift 4 popularit\u0101tei, ir palielin\u0101jies ar\u012b klasteru skaits vis\u0101 pasaul\u0113. Katrs no \u0161iem klasteriem iel\u0101d\u0113 operatoru saturu, lai aktiviz\u0113tu ieb\u016bv\u0113to OperatorHub, izmantojot App Registry iek\u0161\u0113jo quay.io k\u0101 backend. <b>Probl\u0113mas avota mekl\u0113jumos m\u0113s nepaman\u012bj\u0101m, ka ar OpenShift popularit\u0101tes pieaugumu pieauga ar\u012b slodze uz vienu no reti izmantotaj\u0101m quay.io funkcij\u0101m.<\/b>.<\/p>\n<p>M\u0113s veic\u0101m datu pl\u016bsmas anal\u012bzi par App Registry piepras\u012bjumiem un iel\u016bkoj\u0101mies re\u0123istra kod\u0101. Tas uzreiz atkl\u0101ja tr\u016bkumus, kas padar\u012bja datu b\u0101zes piepras\u012bjumus neoptim\u0101lus. Zem\u0101 slodze nerad\u012bja nek\u0101das probl\u0113mas, bet t\u0101s pieauguma gad\u012bjum\u0101 k\u013cuva par probl\u0113mu avotu. App Registry izr\u0101d\u012bj\u0101s divi probl\u0113mu endpoint'i, kas slikti rea\u0123\u0113ja uz slodzes pieaugumu: pirmais sniedza visu pakot\u0146u sarakstu kr\u0101tuv\u0113, bet otrais atgrieza visus blob'us pakotnei.<\/p>\n<h2>Probl\u0113mu nov\u0113r\u0161ana.<\/h2>\n<p>\nN\u0101kamo ned\u0113\u013cu m\u0113s velt\u012bj\u0101m App Registry koda un t\u0101 apk\u0101rt\u0113jo vides optimiz\u0101cijai. Tika p\u0101rstr\u0101d\u0101ti ac\u012bmredzami neefekt\u012bvi SQL piepras\u012bjumi, nov\u0113rsti liekie komandu izsaukumi (tie tika aktiviz\u0113ti, katru reizi ieg\u016bstot blob'us), kur iesp\u0113jams, tika ieviesta ke\u0161atmi\u0146a. P\u0113c tam tika veikts pla\u0161s veiktsp\u0113jas tests un sal\u012bdzin\u0101ta App Registry darb\u012bba pirms un p\u0113c izmai\u0146\u0101m. <code>tar<\/code> API piepras\u012bjumi, kas agr\u0101k ilga l\u012bdz pusmin\u016btei, tagad izpild\u012bj\u0101s milisekund\u0113s.<\/p>\n<p><b>. N\u0101kamaj\u0101 ned\u0113\u013c\u0101 m\u0113s ievies\u0101m izmai\u0146as ra\u017eo\u0161an\u0101, un kop\u0161 t\u0101 laika quay.io darbojas stabil\u0101 re\u017e\u012bm\u0101. \u0160aj\u0101 laik\u0101 m\u0113s nov\u0113roj\u0101m vair\u0101kus straujus datu pl\u016bsmas pieaugumus App Registry endpoint'\u0101, ta\u010du veiktie uzlabojumi nov\u0113rsa datu b\u0101zes darb\u012bbas p\u0101rtraukumus.<\/b>Skaidrs, ka jebkura pakalpojuma sniedz\u0113js cen\u0161as izvair\u012bties no d\u012bkst\u0101ves. M\u016bsu gad\u012bjum\u0101 m\u0113s ticam, ka nesenie trauc\u0113jumi pal\u012bdz\u0113ja padar\u012bt quay.io lab\u0101ku. M\u0113s izdar\u012bj\u0101m da\u017eus svar\u012bgus secin\u0101jumus, kurus v\u0113lamies dal\u012bties:<\/p>\n<h2>Ko m\u0113s esam m\u0101c\u012bju\u0161ies?<\/h2>\n<p>\nDati par to, kas un k\u0101 izmanto j\u016bsu pakalpojumu, nekad nav lieki.<\/p>\n<ol>\n<li> <b>. Kam\u0113r Quay \"vienk\u0101r\u0161i str\u0101d\u0101ja\", mums nebija nepiecie\u0161am\u012bbas t\u0113r\u0113t laiku datu pl\u016bsmas optimiz\u0101cijai un slodzes p\u0101rvald\u012bbai. Tas rad\u012bja mald\u012bgu dro\u0161\u012bbas saj\u016btu, ka pakalpojums var tikt papla\u0161in\u0101ts bezgal\u012bgi.<\/b>Kad pakalpojums p\u0101rtr\u016bkst,<\/li>\n<li> atjauno\u0161anas process ir galven\u0101 priorit\u0101te. <b>t\u0101s atjauno\u0161ana \u2014 galven\u0101 priorit\u0101te<\/b>. Kam\u0113r Quay turpin\u0101ja ciest no blo\u0137\u0113t\u0101s datu b\u0101zes pirmaj\u0101 av\u0101rij\u0101, m\u016bsu standarta proced\u016bras neizr\u0101d\u012bj\u0101s paredz\u0113taj\u0101 efekt\u0101, un m\u0113s nevar\u0113j\u0101m atjaunot pakalpojuma darb\u012bbu, izmantojot t\u0101s. Tas noveda pie situ\u0101cijas, kad bija j\u0101laiko dati un j\u0101analiz\u0113, lai cer\u0113tu atrast pamata c\u0113loni \u2013 viet\u0101, lai visus sp\u0113kus velt\u012btu atjauno\u0161anas darb\u012bb\u0101m.<\/li>\n<li> <b>Nov\u0113rt\u0113jiet katras pakalpojuma funkcijas ietekmi.<\/b>. Klienti reti izmantoja App Registry, t\u0101p\u0113c tas nebija m\u016bsu komandas priorit\u0101te. Kad da\u017eas produkta funkcijas tiek gandr\u012bz nelietotas, to k\u013c\u016bdas \u201epar\u0101d\u0101s\u201d reti, un izstr\u0101d\u0101t\u0101ji p\u0101rst\u0101j sekot kodeksam. Ir viegli k\u013c\u016bt par maldino\u0161o priek\u0161statu, ka t\u0101 tam j\u0101b\u016bt \u2013 l\u012bdz br\u012bdim, kad \u0161\u012b funkcija neizsist p\u0101rv\u0113rt\u0113\u0161anas d\u0113\u013c liel\u0101 incidenta centr\u0101.<\/li>\n<\/ol>\n<p><\/p>\n<h2>Kas t\u0101l\u0101k?<\/h2>\n<p>\nDarbs pie pakalpojuma stabilit\u0101tes nodro\u0161in\u0101\u0161anas nekad neapst\u0101jas, un m\u0113s past\u0101v\u012bgi to uzlabojam. Trafika apjomi uz quay.io turpina pieaugt, un m\u0113s apzin\u0101mies, ka mums j\u0101dara viss iesp\u0113jamais, lai attaisnotu klientu uztic\u0113\u0161anos. T\u0101p\u0113c pa\u0161laik str\u0101d\u0101jam pie \u0161\u0101diem uzdevumiem:<\/p>\n<ol>\n<li> Las\u0101m\u0101s datu b\u0101zes repliku izvieto\u0161ana, lai pal\u012bdz\u0113tu pakalpojumam apstr\u0101d\u0101t attiec\u012bgo trafiku probl\u0113mu gad\u012bjum\u0101 ar galveno RDS instanci.<\/li>\n<li> RDS instances atjaunin\u0101\u0161ana. Eso\u0161\u0101 versija pati par sevi nav probl\u0113ma. Dr\u012bz\u0101k m\u0113s vienk\u0101r\u0161i v\u0113lamies nov\u0113rst nepatiesu pavedienu (kura virzien\u0101 m\u0113s dev\u0101mies av\u0101rijas laik\u0101); programmat\u016bras uztur\u0113\u0161ana aktu\u0101l\u0101 st\u0101vokl\u012b \u013caus nov\u0113rst v\u0113l vienu faktoru n\u0101kotnes trauc\u0113jumiem.<\/li>\n<li> Papildu ke\u0161atmi\u0146a vis\u0101 klaster\u012b. M\u0113s turpin\u0101m mekl\u0113t jomas, kur ke\u0161atmi\u0146a var\u0113tu samazin\u0101t noslodzi uz datu b\u0101zi.<\/li>\n<li> Web aplik\u0101ciju ugunsm\u016bra (WAF) pievieno\u0161ana, lai redz\u0113tu, kas un k\u0101p\u0113c piesl\u0113dzas pie quay.io.<\/li>\n<li> S\u0101kot ar n\u0101kamo laidienu, Red Hat OpenShift klasteri atteiksies no App Registry par labu operatoru katalogiem (Operator Catalogs), kas balst\u012bti uz konteineru att\u0113liem, kas pieejami uz quay.io.<\/li>\n<li> Ilgtermi\u0146a aizvietot\u0101js App Registry var\u0113tu b\u016bt atbalsts Open Container Initiative (OCI) artefaktu specifik\u0101cij\u0101m. Patlaban tas tiek \u012bstenots k\u0101 Quay dzimtais funkcionalit\u0101te un b\u016bs pieejams lietot\u0101jiem, kad specifik\u0101cija tiks noteikti apstiprin\u0101ta.<\/li>\n<\/ol>\n<p>\nViss iepriek\u0161 min\u0113tais ir da\u013ca no turpin\u0101jumiem, ko Red Hat iegulda quay.io, p\u0101rejot no nelielas \"s\u0101kuma komandas\" uz nobriedu\u0161u platformu, ko p\u0101rvalda SRE. M\u0113s apzin\u0101mies, ka daudzi m\u016bsu klienti pa\u013caujas uz quay.io sav\u0101 ikdienas darb\u012bb\u0101 (tai skait\u0101 Red Hat!) un cen\u0161amies b\u016bt maksim\u0101li atkl\u0101ti par nesenajiem trauc\u0113jumiem un nep\u0101rtrauktajiem centieniem k\u013c\u016bt lab\u0101kiem.<\/p>\n<h2>P.S. no tulka<\/h2>\n<p>\nLasiet ar\u012b m\u016bsu emu\u0101ra rakstos:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/news\/t\/475716\/\/\">Red Hat ir atv\u0113rusi CoreOS konteineru att\u0113lu re\u0123istra kodu \u2014 Quay<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/510486\/\">Praktiskas st\u0101sti no m\u016bsu SRE ikdienas. Otr\u0101 da\u013ca<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/461807\/\">K\u0101 podu priorit\u0101tes Kubernetes izrais\u012bja Grafana Labs d\u012bkst\u0101vi<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Avots: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/515932\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u043c. \u043f\u0435\u0440\u0435\u0432.: \u0432 \u043d\u0430\u0447\u0430\u043b\u0435 \u0430\u0432\u0433\u0443\u0441\u0442\u0430 Red Hat \u043f\u0443\u0431\u043b\u0438\u0447\u043d\u043e \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u043b\u0430 \u043e \u0440\u0435\u0448\u0435\u043d\u0438\u0438 \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438, \u0447\u0442\u043e \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u043b\u0438 \u0432 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0435 \u043c\u0435\u0441\u044f\u0446\u044b \u0443 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u0435\u0439 \u0435\u0451 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 Quay.io (\u0432 \u0435\u0433\u043e \u043e\u0441\u043d\u043e\u0432\u0435 \u2014 \u0440\u0435\u0435\u0441\u0442\u0440 \u0434\u043b\u044f \u043e\u0431\u0440\u0430\u0437\u043e\u0432 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u0434\u043e\u0441\u0442\u0430\u0432\u0448\u0438\u0439\u0441\u044f \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0432\u043c\u0435\u0441\u0442\u0435 \u0441 \u043f\u043e\u043a\u0443\u043f\u043a\u043e\u0439 CoreOS). \u0412\u043d\u0435 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0438 \u043e\u0442 \u0432\u0430\u0448\u0435\u0439 \u0437\u0430\u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043e\u0432\u0430\u043d\u043d\u043e\u0441\u0442\u0438 \u0432 \u044d\u0442\u043e\u043c \u0441\u0435\u0440\u0432\u0438\u0441\u0435 \u043a\u0430\u043a \u0442\u0430\u043a\u043e\u0432\u043e\u043c, \u043f\u043e\u0443\u0447\u0438\u0442\u0435\u043b\u0435\u043d \u0441\u0430\u043c \u043f\u0443\u0442\u044c, \u043f\u043e \u043a\u043e\u0442\u043e\u0440\u043e\u043c\u0443 \u043f\u0440\u043e\u0448\u043b\u0438 SRE-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u044b \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":92074,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-92073","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u043c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5\" \/>\n\t\t<meta property=\"og:locale\" content=\"lv_LV\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Post Mortem \u043f\u043e \u043d\u0435\u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438 Quay.io | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u043c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-22T17:41:56+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-22T17:41:56+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Post Mortem par Quay.io pieejam\u012bbas tr\u016bkumu | ProHoster","description":"Piez.","canonical_url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"lv_LV","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Post Mortem \u043f\u043e \u043d\u0435\u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0441\u0442\u0438 Quay.io | ProHoster","og:description":"\u041f\u0440\u0438\u043c.","og:url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-22T17:41:56+00:00","article:modified_time":"2020-08-22T17:41:56+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"92073","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:15:36","updated":"2022-10-02 22:37:13","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/92073","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/comments?post=92073"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/92073\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media\/92074"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media?parent=92073"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/categories?post=92073"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/tags?post=92073"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}