Our team loves experimentation. Each SLRM is not a static repetition of the previous ones, but a reflection on experience and a transition from good to better. But with we decided to implement a completely new format — to provide participants with conditions as close to 'combat' as possible.
To briefly outline what we did during the intensive: 'Building, breaking, fixing,
learning.' SRE doesn't amount to much in pure theory — only practice, real solutions, real problems.
Participants were divided into teams so that the lively competitive spirit would keep everyone awake and prevent anyone from launching 'Angry Birds' on their iPhone, like Dmitry Anatolyevich did.
Problems, glitches, bugs, and tasks were managed by four mentors: Ivan Kruglov, Principal Developer at Booking.com (Netherlands); Ben Tyler, Principal Developer at Booking.com (USA); Eduard Medvedev, CTO at Tungsten Labs (Germany); Evgeny Varavva, full-stack developer at Google (San Francisco).
And participants were divided into teams — competing against each other. Interesting?
Ivan, Ben, Eduard, and Evgeny, with a kind Lenin-like squint, look at the poor participants of SLRM SRE before the competition starts.
So the task is:
We will build our new world…
There is a ticket aggregator website for movies. Incidents are devised by mentors within a well-prepared scenario (though no one excludes particularly sophisticated and cunning improvisation). The website's performance is described by various metrics. Problems can vary widely: tickets for the 'Moulin Rouge' theater do not load into the database; movie and play posters take more than 10 seconds to load into the database; the description of a particular film hangs; 0.1% of orders end up on already reserved seats; occasionally, the payment processing system goes down for a minute or two. And so many unpleasant things can fall on the participant of SLRM SRE in their real job.
We are ready to handle everything… and everyone.
Our beleaguered website consists of several microservices. Its task is to aggregate data about sessions, prices, and available seats from all cinemas, showing movie trailers, allowing users to select a cinema, session, hall, and seat, and to book and pay for tickets. In short, everything a viewer could possibly dream of. Yet users have no idea about the titanic struggle for the website's stability and availability that takes place behind the scenes.
For the intensive website, we defined SLO, SLI, SLA metrics, developed the architecture and infrastructure, deployed the site, and set up monitoring and alerting. And then it began.
SLO, SLI, SLA
SLI — service level indicators. SLO — service level objectives. SLA — service level agreements.
SLA is a term from the ITIL methodology that denotes a formal contract between the service customer and the service provider, containing a description of the service, the rights and responsibilities of the parties, and, most importantly, the agreed level of quality of the service provided.
SLO is a service level objective: a target value or range of values for the service level measured by SLI. A normal value for SLO is "SLI ≤ target value" or "lower boundary ≤ SLI ≤ upper boundary."
SLI is a service level indicator — a carefully defined quantitative measure of one aspect of the service level provided. For most services, the key SLI is request latency — how long it takes to return a response to a request. Other common SLIs include error rate, often expressed as a fraction of all received requests, and system throughput, usually measured in requests per second.
First, we will break the planes, and the girls, well, the girls later...
Internal and external factors began to 'spoil' the SLO from the very first minute. Everything fell on the admins — both developer errors, infrastructure failures, surges in visitors, and DDoS attacks. All of these affect the SLO negatively.
"- Dear participants, I am pleased to inform you that, first of all, everything is crashing... everything!"
Along the way, the speakers discussed resilience, error budgets, testing practices, managing interruptions, and operational load.
We are neither stokers nor carpenters...
Here, participants started to fix things — the main thing is to understand what to tackle first.
"- God, I've never seen something break like this, in this form and position!"
So, an accident occurred. The payment processing service went down. What should we do to restore functionality as quickly as possible?
Experts are watching the participants kindly while preparing their next trick.
Each team organizes the group's work to mitigate the incident — connecting colleagues, notifying stakeholders. Additionally, they prioritize tasks. Thus, participants practiced working under pressure in extremely limited time conditions.
"- What horror is that?!"
They sighed... and finished the exercise.
Together with the speakers, after each resolved issue and temporary stabilization of the site, teams studied incidents from an SRE perspective. They analyzed problems in detail — the causes of occurrence, the course of resolution. Afterwards, both individually and collectively, they made decisions on how to prevent them in the future: how to improve monitoring, how to properly change architecture, how to adjust the approach to development and operation, how to correct regulations. Speakers demonstrated the practice of conducting post-mortems.
"- Who else wants some suffering! — I do!"
The electronic scoreboard clearly and accurately recorded the teams' progress.
For the first places — a prize from the stakeholders.
Source: habr.com
