{"id":56260,"date":"2020-02-08T00:00:00","date_gmt":"2020-02-07T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com"},"modified":"2020-02-18T14:04:30","modified_gmt":"2020-02-18T11:04:30","slug":"slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","title":{"rendered":"SRE Slurm. Komplexn\u00fd experiment s odborn\u00edkmi z Booking.com a Google.com","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

N\u00e1\u0161 t\u00edm miluje experimentovanie. Ka\u017ed\u00fd Slurm nie je statick\u00e9 opakovanie predch\u00e1dzaj\u00faceho, ale zamyslenie nad sk\u00fasenos\u0165ou a prechod od dobr\u00e9ho k lep\u0161iemu. Ale s Slurmom SRE<\/a><\/noindex> sme sa rozhodli aplikova\u0165 \u00faplne nov\u00fd form\u00e1t \u2014 da\u0165 \u00fa\u010dastn\u00edkom podmienky, maxim\u00e1lne pribl\u00ed\u017een\u00e9 k \"bojov\u00fdm\". <\/p>\n

<\/p>\n

Ak by sme stru\u010dne na\u010drtli, \u010domu sme sa na intenz\u00edvnom \u0161kolen\u00ed venovali: \"Staviame, rozb\u00edjame, opravujeme,
\n\u0161tudujeme\". SRE nem\u00e1 ve\u013ek\u00fa hodnotu v hol\u00e9 te\u00f3rii \u2014 iba prax, re\u00e1lne rie\u0161enia, re\u00e1lne probl\u00e9my. <\/p>\n

<\/p>\n

\u00da\u010dastn\u00edci boli rozdelen\u00ed do t\u00edmov, aby \u017eiv\u00fd s\u00fa\u0165a\u017en\u00fd duch nikoho nenechal zaspa\u0165 alebo nespustil \"Angry Birds\" na iPhone ako napr\u00edklad Dmitrij Anatoljevi\u010d. <\/p>\n

<\/p>\n

Probl\u00e9my, chyby, bugy a \u00falohy zabezpe\u010dovali \u00fa\u010dastn\u00edkom \u0161tyria mentori. Ivan Kruglov, hlavn\u00fdm v\u00fdvoj\u00e1rom v Booking.com (Holandsko). Ben Tyler, hlavn\u00fdm v\u00fdvoj\u00e1rom v Booking.com (USA). Eduard Medvedev, CTO vo Tungsten Labs (Nemecko). Jevgenij Varavva, \u0161irokospektr\u00e1lny v\u00fdvoj\u00e1r v Google (San Francisco).<\/p>\n

<\/p>\n

A e\u0161te aj \u00fa\u010dastn\u00edci rozdelen\u00ed do t\u00edmov \u2014 a s\u00fa\u0165a\u017eia navz\u00e1jom. Zauj\u00edmav\u00e9? <\/p>\n

<\/p>\n

\"SRE
\nIvan, Ben, Eduard a Jevgenij s dobrou Leninovou pokorou pozeraj\u00fa na chud\u00e1kov \u00fa\u010dastn\u00edkov Slurm SRE pred za\u010diatkom s\u00fa\u0165a\u017ee.<\/em><\/p>\n

<\/p>\n

<\/a><\/noindex>Tak\u017ee \u00faloha: <\/p>\n

<\/p>\n

My sme, my nov\u00fd svet postav\u00edme\u2026<\/strong><\/p>\n

<\/p>\n

Existuje webov\u00fd agreg\u00e1tor l\u00edstkov do kina. Incidenty vym\u00fd\u0161\u013eaj\u00fa mentori v vopred spracovanom scen\u00e1ri (aj ke\u010f nikto nevylu\u010duje extra sofistikovan\u00fa a zlo\u017eit\u00fa improviz\u00e1ciu), funk\u010dnos\u0165 webu je popisovan\u00e1 r\u00f4znymi metrikami. Probl\u00e9my m\u00f4\u017eu by\u0165 r\u00f4zne: l\u00edstky na divadeln\u00e9 predstavenie \"Moulin Rouge\" sa nena\u010d\u00edtavaj\u00fa do datab\u00e1zy; plag\u00e1ty filmov a predstaven\u00ed sa na\u010d\u00edtavaj\u00fa do datab\u00e1zy dlh\u0161ie ako 10 sek\u00fand; popis jednotliv\u00e9ho filmu sa <\/p>\n

<\/p>\n

\"SRE
\nzasek\u00e1va; 0,1% objedn\u00e1vok sa dost\u00e1va na u\u017e rezervovan\u00e9 miesta; ob\u010das sa na sladk\u00e9 syst\u00e9m spracovania platieb odpoj\u00ed na min\u00fatu-dve. A ve\u013ea, ve\u013ea nepr\u00edjemnost\u00ed, ktor\u00e9 m\u00f4\u017ee \u00fa\u010dastn\u00edkovi Slurma SRE spadn\u00fa\u0165 na jeho re\u00e1lnu pr\u00e1cu.<\/em><\/p>\n

<\/p>\n

Na\u0161a \u0165a\u017eko sk\u00fa\u0161an\u00e1 str\u00e1nka pozost\u00e1va z viacer\u00fdch mikroservisov. Jej \u00falohou je agreg\u00e1cia \u00fadajov o sedeniach, cen\u00e1ch a vo\u013en\u00fdch miestach zo v\u0161etk\u00fdch k\u00edn; zobrazuje up\u00fatavky na filmy, umo\u017e\u0148uje vybra\u0165 kino, sedenie, s\u00e1lu a miesto, rezervova\u0165 a zaplati\u0165 vstupenky. Skr\u00e1tka, v\u0161etko, o \u010dom div\u00e1k len sn\u00edva. Len\u017ee pou\u017e\u00edvate\u013e ani netu\u0161\u00ed, ak\u00fd tit\u00e1nsky boj prebieha za stabilitu a dostupnos\u0165 str\u00e1nky vo vn\u00fatri. <\/p>\n

<\/p>\n

Pre str\u00e1nku na intenz\u00edve sme formulovali ukazovatele SLO, SLI, SLA, vyvinuli sme architekt\u00faru a infra\u0161trukt\u00faru, nasadili sme str\u00e1nku, nastavili monitoring a alerting. A za\u010dalo sa. <\/p>\n

\nSLO, SLI, SLA<\/b><\/p>\n

SLI \u2014 ukazovatele \u00farovne slu\u017eieb. SLO \u2014 ciele \u00farovne slu\u017eieb. SLA \u2014 zmluvy o \u00farovni slu\u017eieb. <\/p>\n

SLA \u2014 term\u00edn metodol\u00f3gie ITIL, ozna\u010duj\u00faci form\u00e1lnu zmluvu medzi z\u00e1kazn\u00edkom slu\u017eby a jej poskytovate\u013eom, ktor\u00e1 obsahuje popis slu\u017eby, pr\u00e1va a povinnosti str\u00e1n a, \u010do je najd\u00f4le\u017eitej\u0161ie, dohodnut\u00fa \u00farove\u0148 kvality poskytovanej slu\u017eby.<\/p>\n

SLO \u2014 cie\u013e \u00farovne slu\u017eieb: cie\u013eov\u00e1 hodnota alebo rozsah hodn\u00f4t pre \u00farove\u0148 slu\u017eieb, ktor\u00fd meria SLI. Norm\u00e1lnou hodnotou pre SLO je \"SLI \u2264 cie\u013eov\u00e1 hodnota\" alebo \"doln\u00e1 hranica \u2264 SLI \u2264 horn\u00e1 hranica.\" <\/p>\n

SLI je ukazovate\u013e \u00farovne slu\u017eieb \u2014 starostlivo definovan\u00e9 kvantitat\u00edvne meranie jedn\u00e9ho z aspektov poskytovanej \u00farovne slu\u017eieb. Pre v\u00e4\u010d\u0161inu slu\u017eieb sa ako k\u013e\u00fa\u010dov\u00fd SLI pova\u017euje oneskorenie po\u017eiadavky \u2014 ko\u013eko \u010dasu je potrebn\u00e9 na vr\u00e1tenie odpovede na po\u017eiadavku. In\u00e9 be\u017en\u00e9 SLI zahr\u0148uj\u00fa frekvenciu ch\u00fdb, \u010dasto vyjadren\u00fa ako podiel v\u0161etk\u00fdch prijat\u00fdch po\u017eiadaviek, a priepustnos\u0165 syst\u00e9mu, zvy\u010dajne meran\u00fa v po\u017eiadavk\u00e1ch za sekundu.<\/p><\/blockquote>\n

<\/p>\n

Najprv zni\u010d\u00edme lietadl\u00e1, no a diev\u010dat\u00e1, diev\u010dat\u00e1 potom\u2026<\/strong><\/p>\n

<\/p>\n

Intern\u00e9 a extern\u00e9 faktory od prv\u00fdch min\u00fat za\u010dali \"kazit\" SLO. Na hlavy administr\u00e1torov sa zosypalo v\u0161etko \u2014 a chyby v\u00fdvoj\u00e1rov, a zlyhania infra\u0161trukt\u00fary, a pr\u00edliv n\u00e1v\u0161tevn\u00edkov, a DDoS \u00fatoky. V\u0161etko, \u010do zhor\u0161uje SLO. <\/p>\n

<\/p>\n

\"SRE
\n\"- Drah\u00ed \u00fa\u010dastn\u00edci, pon\u00e1h\u013eam sa v\u00e1s pote\u0161i\u0165, ako prv\u00e9 v\u00e1m pad\u00e1\u2026 v\u0161etko!\"<\/em><\/p>\n

<\/p>\n

Po\u010das akcie predn\u00e1\u0161aj\u00faci rozoberali odolnos\u0165, rozpo\u010det ch\u00fdb, prax testovania, riadenie preru\u0161en\u00ed a prev\u00e1dzkov\u00fa z\u00e1\u0165a\u017e.<\/p>\n

<\/p>\n

Nie sme v\u00fdhybk\u00e1ri, nie sme tes\u00e1ri\u2026<\/strong><\/p>\n

<\/p>\n

Tu \u00fa\u010dastn\u00edci za\u010dali opravova\u0165 \u2014 hlavn\u00e9 je pochopi\u0165, \u010do uchopi\u0165 ako prv\u00e9. <\/p>\n

<\/p>\n

\"SRE
\n\u201e- Bo\u017ee, nikdy som nevidel, \u017ee by sa to pokazilo takto, v takom stave a v takej poz\u00edcii!\u201c<\/em><\/p>\n

<\/p>\n

Tak\u017ee do\u0161lo k hav\u00e1rii. Slu\u017eba spracovania platieb zlyhala. Ako postupova\u0165, aby sme obnovili funk\u010dnos\u0165 v minim\u00e1lnom \u010dase?<\/p>\n

<\/p>\n

\"SRE
\nOdborn\u00edci s l\u00e1skypln\u00fdm poh\u013eadom na \u00fa\u010dastn\u00edkov pripravuj\u00fa \u010fal\u0161\u00ed trik.<\/em><\/p>\n

<\/p>\n

Ka\u017ed\u00fd t\u00edm organizuje pr\u00e1cu skupiny na odstr\u00e1nenie hav\u00e1rie \u2014 zap\u00e1ja kolegov, informuje zainteresovan\u00fdch (stakeholders). Z\u00e1rove\u0148 sa stanovuj\u00fa priority. Tak \u00fa\u010dastn\u00edci tr\u00e9novali pr\u00e1cu pod tlakom v podmienkach extr\u00e9mne obmedzen\u00e9ho \u010dasu.<\/p>\n

<\/p>\n

\"SRE
\n\u201e- \u010co je to za hr\u00f4zu, \u010do sa vyliahlo?!\u201c<\/em><\/p>\n

<\/p>\n

Vyd\u00fdchli sme... a dokon\u010dili cvi\u010denie.<\/strong><\/p>\n

<\/p>\n

Spolu s predn\u00e1\u0161aj\u00facimi po ka\u017edom vyrie\u0161enom probl\u00e9me a do\u010dasne stabilizovanej str\u00e1nke t\u00edm \u0161tudoval incidenty z poh\u013eadu SRE. Detailne analyzovali probl\u00e9my \u2014 pr\u00ed\u010diny vzniku, priebeh odstr\u00e1nenia. Potom t\u00edmovo aj kolekt\u00edvne prij\u00edmali rozhodnutia o ich \u010fal\u0161om predch\u00e1dzan\u00ed: ako zlep\u0161i\u0165 monitoring, ako spr\u00e1vne zmeni\u0165 architekt\u00faru, ako upravi\u0165 pr\u00edstup k v\u00fdvoju a prev\u00e1dzke, ako opravi\u0165 predpisy. Predn\u00e1\u0161aj\u00faci demon\u0161trovali prax vykon\u00e1vania post-mortem. <\/p>\n

<\/p>\n

\"SRE
\n\u201e- Kto chce \u010fal\u0161ie mu\u010denie! \u2014 Ja!\u201c<\/em><\/p>\n

<\/p>\n

Na elektronickej tabuli sa striktne a jasne zaznamen\u00e1vali \u00faspechy t\u00edmov. <\/p>\n

<\/p>\n

\"SRE<\/p>\n

<\/p>\n

Za prv\u00e9 miesta \u2014 odmena od stakeholderov.<\/p>\n

<\/p>\n

\"SRE<\/p>\n

Zdroj: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430 \u043b\u044e\u0431\u0438\u0442 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u041a\u0430\u0436\u0434\u044b\u0439 \u0421\u043b\u0451\u0440\u043c \u2014 \u044d\u0442\u043e \u043d\u0435 \u0441\u0442\u0430\u0442\u0438\u0447\u043d\u043e\u0435 \u043f\u043e\u0432\u0442\u043e\u0440\u0435\u043d\u0438\u0435 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0445, \u0430 \u043e\u0441\u043c\u044b\u0441\u043b\u0435\u043d\u0438\u0435 \u043e\u043f\u044b\u0442\u0430 \u0438 \u043f\u0435\u0440\u0435\u0445\u043e\u0434 \u043e\u0442 \u0445\u043e\u0440\u043e\u0448\u0435\u0433\u043e \u043a \u043b\u0443\u0447\u0448\u0435\u043c\u0443. \u041d\u043e \u0441\u043e \u0421\u043b\u0451\u0440\u043c\u043e\u043c SRE \u043c\u044b \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u0442\u044c \u0430\u0431\u0441\u043e\u043b\u044e\u0442\u043d\u043e \u043d\u043e\u0432\u044b\u0439 \u0444\u043e\u0440\u043c\u0430\u0442 \u2014 \u0434\u0430\u0442\u044c \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u0430\u043c \u0443\u0441\u043b\u043e\u0432\u0438\u044f, \u043c\u0430\u043a\u0441\u0438\u043c\u0430\u043b\u044c\u043d\u043e \u043f\u0440\u0438\u0431\u043b\u0438\u0436\u0451\u043d\u043d\u044b\u0435 \u043a \u00ab\u0431\u043e\u0435\u0432\u044b\u043c\u00bb. \u0415\u0441\u043b\u0438 \u043a\u0440\u0430\u0442\u043a\u043e \u043e\u0431\u0440\u0438\u0441\u043e\u0432\u0430\u0442\u044c, \u0447\u0435\u043c \u043c\u044b \u0437\u0430\u043d\u0438\u043c\u0430\u043b\u0438\u0441\u044c \u043d\u0430 \u0438\u043d\u0442\u0435\u043d\u0441\u0438\u0432\u0435: \u00ab\u0421\u0442\u0440\u043e\u0438\u043c, \u043b\u043e\u043c\u0430\u0435\u043c, \u0447\u0438\u043d\u0438\u043c, \u0438\u0437\u0443\u0447\u0430\u0435\u043c\u00bb. SRE \u043c\u0430\u043b\u043e \u0447\u0435\u0433\u043e \u0441\u0442\u043e\u0438\u0442 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-56260","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Sloerm SRE. Nepretr\u017eit\u00fd experiment s odborn\u00edkmi z Booking.com a Google.com | ProHoster","description":"N\u00e1\u0161 t\u00edm miluje experimenty. Ka\u017ed\u00fd Sloerm nie je statick\u00e9 opakovanie predch\u00e1dzaj\u00facich, ale prenikanie do sk\u00fasenosti a prechod od dobr\u00e9ho k lep\u0161iemu.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0421\u043b\u0451\u0440\u043c SRE. \u0421\u043f\u043b\u043e\u0448\u043d\u043e\u0439 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442 c \u044d\u043a\u0441\u043f\u0435\u0440\u0442\u0430\u043c\u0438 \u0438\u0437 Booking.com \u0438 Google.com | ProHoster","og:description":"\u041d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430 \u043b\u044e\u0431\u0438\u0442 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u041a\u0430\u0436\u0434\u044b\u0439 \u0421\u043b\u0451\u0440\u043c \u2014 \u044d\u0442\u043e \u043d\u0435 \u0441\u0442\u0430\u0442\u0438\u0447\u043d\u043e\u0435 \u043f\u043e\u0432\u0442\u043e\u0440\u0435\u043d\u0438\u0435 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0445, \u0430 \u043e\u0441\u043c\u044b\u0441\u043b\u0435\u043d\u0438\u0435 \u043e\u043f\u044b\u0442\u0430 \u0438 \u043f\u0435\u0440\u0435\u0445\u043e\u0434 \u043e\u0442 \u0445\u043e\u0440\u043e\u0448\u0435\u0433\u043e \u043a \u043b\u0443\u0447\u0448\u0435\u043c\u0443.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-07T21:00:00+00:00","article:modified_time":"2020-02-18T11:04:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"56260","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:28:23","updated":"2022-09-29 15:08:36","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/56260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=56260"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/56260\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=56260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=56260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=56260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}