{"id":56260,"date":"2020-02-08T00:00:00","date_gmt":"2020-02-07T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com"},"modified":"2020-02-18T14:04:30","modified_gmt":"2020-02-18T11:04:30","slug":"slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","title":{"rendered":"Slurm SRE. A continuous experiment with experts from Booking.com and Google.com","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Our team loves experimentation. Each SLRM is not a static repetition of the previous ones, but a reflection on experience and a transition from good to better. But with SLRM SRE<\/a><\/noindex> we decided to implement a completely new format \u2014 to provide participants with conditions as close to 'combat' as possible. <\/p>\n

<\/p>\n

To briefly outline what we did during the intensive: 'Building, breaking, fixing,
\nlearning.' SRE doesn't amount to much in pure theory \u2014 only practice, real solutions, real problems. <\/p>\n

<\/p>\n

Participants were divided into teams so that the lively competitive spirit would keep everyone awake and prevent anyone from launching 'Angry Birds' on their iPhone, like Dmitry Anatolyevich did. <\/p>\n

<\/p>\n

Problems, glitches, bugs, and tasks were managed by four mentors: Ivan Kruglov, Principal Developer at Booking.com (Netherlands); Ben Tyler, Principal Developer at Booking.com (USA); Eduard Medvedev, CTO at Tungsten Labs (Germany); Evgeny Varavva, full-stack developer at Google (San Francisco).<\/p>\n

<\/p>\n

And participants were divided into teams \u2014 competing against each other. Interesting? <\/p>\n

<\/p>\n

\"Slurm
\nIvan, Ben, Eduard, and Evgeny, with a kind Lenin-like squint, look at the poor participants of SLRM SRE before the competition starts.<\/em><\/p>\n

<\/p>\n

<\/a><\/noindex>So the task is: <\/p>\n

<\/p>\n

We shall build our own, a new world\u2026<\/strong><\/p>\n

<\/p>\n

There is a ticket aggregator website for movies. Incidents are devised by mentors within a well-prepared scenario (though no one excludes particularly sophisticated and cunning improvisation). The website's performance is described by various metrics. Problems can vary widely: tickets for the 'Moulin Rouge' theater do not load into the database; movie and play posters take more than 10 seconds to load into the database; the description of a particular film hangs; 0.1% of orders end up on already reserved seats; occasionally, the payment processing system goes down for a minute or two. And so many unpleasant things can fall on the participant of SLRM SRE in their real job. <\/p>\n

<\/p>\n

\"Slurm
\nWe are ready to handle everything\u2026 and everyone.<\/em><\/p>\n

<\/p>\n

Our beleaguered website consists of several microservices. Its task is to aggregate data about sessions, prices, and available seats from all cinemas, showing movie trailers, allowing users to select a cinema, session, hall, and seat, and to book and pay for tickets. In short, everything a viewer could possibly dream of. Yet users have no idea about the titanic struggle for the website's stability and availability that takes place behind the scenes. <\/p>\n

<\/p>\n

For the intensive website, we defined SLO, SLI, SLA metrics, developed the architecture and infrastructure, deployed the site, and set up monitoring and alerting. And then it began. <\/p>\n

\nSLO, SLI, SLA<\/b><\/p>\n

SLI \u2014 service level indicators. SLO \u2014 service level objectives. SLA \u2014 service level agreements. <\/p>\n

SLA is a term from the ITIL methodology that denotes a formal contract between the service customer and the service provider, containing a description of the service, the rights and responsibilities of the parties, and, most importantly, the agreed level of quality of the service provided.<\/p>\n

SLO is a service level objective: a target value or range of values for the service level measured by SLI. A normal value for SLO is \"SLI \u2264 target value\" or \"lower boundary \u2264 SLI \u2264 upper boundary.\" <\/p>\n

SLI is a service level indicator \u2014 a carefully defined quantitative measure of one aspect of the service level provided. For most services, the key SLI is request latency \u2014 how long it takes to return a response to a request. Other common SLIs include error rate, often expressed as a fraction of all received requests, and system throughput, usually measured in requests per second.<\/p><\/blockquote>\n

<\/p>\n

First of all, we will break the planes, and the girls, well, the girls will come later\u2026<\/strong><\/p>\n

<\/p>\n

Internal and external factors began to 'spoil' the SLO from the very first minute. Everything fell on the admins \u2014 both developer errors, infrastructure failures, surges in visitors, and DDoS attacks. All of these affect the SLO negatively. <\/p>\n

<\/p>\n

\"Slurm
\n\"- Dear participants, I am pleased to inform you that, first of all, everything is crashing... everything!\"<\/em><\/p>\n

<\/p>\n

Along the way, the speakers discussed resilience, error budgets, testing practices, managing interruptions, and operational load.<\/p>\n

<\/p>\n

We are not stokers, we are not carpenters\u2026<\/strong><\/p>\n

<\/p>\n

Here, participants started to fix things \u2014 the main thing is to understand what to tackle first. <\/p>\n

<\/p>\n

\"Slurm
\n\"- God, I've never seen something break like this, in this form and position!\"<\/em><\/p>\n

<\/p>\n

So, an accident occurred. The payment processing service went down. What should we do to restore functionality as quickly as possible?<\/p>\n

<\/p>\n

\"Slurm
\nExperts are watching the participants kindly while preparing their next trick.<\/em><\/p>\n

<\/p>\n

Each team organizes the group's work to mitigate the incident \u2014 connecting colleagues, notifying stakeholders. Additionally, they prioritize tasks. Thus, participants practiced working under pressure in extremely limited time conditions.<\/p>\n

<\/p>\n

\"Slurm
\n\"- What horror is that?!\"<\/em><\/p>\n

<\/p>\n

They sighed... and finished the exercise.<\/strong><\/p>\n

<\/p>\n

Together with the speakers, after each resolved issue and temporary stabilization of the site, teams studied incidents from an SRE perspective. They analyzed problems in detail \u2014 the causes of occurrence, the course of resolution. Afterwards, both individually and collectively, they made decisions on how to prevent them in the future: how to improve monitoring, how to properly change architecture, how to adjust the approach to development and operation, how to correct regulations. Speakers demonstrated the practice of conducting post-mortems. <\/p>\n

<\/p>\n

\"Slurm
\n\"- Who else wants some suffering! \u2014 I do!\"<\/em><\/p>\n

<\/p>\n

The electronic scoreboard clearly and accurately recorded the teams' progress. <\/p>\n

<\/p>\n

\"Slurm<\/p>\n

<\/p>\n

For the first places \u2014 a prize from the stakeholders.<\/p>\n

<\/p>\n

\"Slurm<\/p>\n

Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430 \u043b\u044e\u0431\u0438\u0442 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u041a\u0430\u0436\u0434\u044b\u0439 \u0421\u043b\u0451\u0440\u043c \u2014 \u044d\u0442\u043e \u043d\u0435 \u0441\u0442\u0430\u0442\u0438\u0447\u043d\u043e\u0435 \u043f\u043e\u0432\u0442\u043e\u0440\u0435\u043d\u0438\u0435 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0445, \u0430 \u043e\u0441\u043c\u044b\u0441\u043b\u0435\u043d\u0438\u0435 \u043e\u043f\u044b\u0442\u0430 \u0438 \u043f\u0435\u0440\u0435\u0445\u043e\u0434 \u043e\u0442 \u0445\u043e\u0440\u043e\u0448\u0435\u0433\u043e \u043a \u043b\u0443\u0447\u0448\u0435\u043c\u0443. \u041d\u043e \u0441\u043e \u0421\u043b\u0451\u0440\u043c\u043e\u043c SRE \u043c\u044b \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438\u043c\u0435\u043d\u0438\u0442\u044c \u0430\u0431\u0441\u043e\u043b\u044e\u0442\u043d\u043e \u043d\u043e\u0432\u044b\u0439 \u0444\u043e\u0440\u043c\u0430\u0442 \u2014 \u0434\u0430\u0442\u044c \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u0430\u043c \u0443\u0441\u043b\u043e\u0432\u0438\u044f, \u043c\u0430\u043a\u0441\u0438\u043c\u0430\u043b\u044c\u043d\u043e \u043f\u0440\u0438\u0431\u043b\u0438\u0436\u0451\u043d\u043d\u044b\u0435 \u043a \u00ab\u0431\u043e\u0435\u0432\u044b\u043c\u00bb. \u0415\u0441\u043b\u0438 \u043a\u0440\u0430\u0442\u043a\u043e \u043e\u0431\u0440\u0438\u0441\u043e\u0432\u0430\u0442\u044c, \u0447\u0435\u043c \u043c\u044b \u0437\u0430\u043d\u0438\u043c\u0430\u043b\u0438\u0441\u044c \u043d\u0430 \u0438\u043d\u0442\u0435\u043d\u0441\u0438\u0432\u0435: \u00ab\u0421\u0442\u0440\u043e\u0438\u043c, \u043b\u043e\u043c\u0430\u0435\u043c, \u0447\u0438\u043d\u0438\u043c, \u0438\u0437\u0443\u0447\u0430\u0435\u043c\u00bb. SRE \u043c\u0430\u043b\u043e \u0447\u0435\u0433\u043e \u0441\u0442\u043e\u0438\u0442 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-56260","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Slorm SRE. A continuous experiment with experts from Booking.com and Google.com | ProHoster","description":"Our team loves experimentation. Each Slorm is not a static repetition of the previous ones, but a reflection on experience and a transition from good to better.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0421\u043b\u0451\u0440\u043c SRE. \u0421\u043f\u043b\u043e\u0448\u043d\u043e\u0439 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442 c \u044d\u043a\u0441\u043f\u0435\u0440\u0442\u0430\u043c\u0438 \u0438\u0437 Booking.com \u0438 Google.com | ProHoster","og:description":"\u041d\u0430\u0448\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u0430 \u043b\u044e\u0431\u0438\u0442 \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u044b. \u041a\u0430\u0436\u0434\u044b\u0439 \u0421\u043b\u0451\u0440\u043c \u2014 \u044d\u0442\u043e \u043d\u0435 \u0441\u0442\u0430\u0442\u0438\u0447\u043d\u043e\u0435 \u043f\u043e\u0432\u0442\u043e\u0440\u0435\u043d\u0438\u0435 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0438\u0445, \u0430 \u043e\u0441\u043c\u044b\u0441\u043b\u0435\u043d\u0438\u0435 \u043e\u043f\u044b\u0442\u0430 \u0438 \u043f\u0435\u0440\u0435\u0445\u043e\u0434 \u043e\u0442 \u0445\u043e\u0440\u043e\u0448\u0435\u0433\u043e \u043a \u043b\u0443\u0447\u0448\u0435\u043c\u0443.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/slyorm-sre-sploshnoj-eksperiment-c-ekspertami-iz-booking-com-i-google-com","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-07T21:00:00+00:00","article:modified_time":"2020-02-18T11:04:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"56260","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:28:23","updated":"2022-09-29 15:08:36","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/56260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=56260"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/56260\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=56260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=56260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=56260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}