{"id":54449,"date":"2019-12-26T00:00:00","date_gmt":"2019-12-25T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-ya-nedelyu-byl-stazherom-sre-inzhenera-dezhurstvo-glazami-inzhenera-po"},"modified":"2020-02-18T14:02:27","modified_gmt":"2020-02-18T11:02:27","slug":"kak-ya-nedelyu-byl-stazherom-sre-inzhenera-dezhurstvo-glazami-inzhenera-po","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-ya-nedelyu-byl-stazherom-sre-inzhenera-dezhurstvo-glazami-inzhenera-po","title":{"rendered":"How I spent a week as an SRE engineer intern. Duty through the eyes of a software engineer.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"How<\/p>\n

<\/p>\n

SRE Engineer - Intern<\/h2>\n

<\/p>\n

Let me introduce myself. I am @tristan.read<\/a><\/noindex>, a frontend engineer in the Monitor::Health<\/a><\/noindex> GitLab. Last week I had the honor of being an intern with one of our on-call SRE engineers. The goal was to observe daily how the on-call engineer responds to incidents and gain real work experience. We would like our engineers to better understand user needs. functions<\/a><\/noindex> Monitor::Health.<\/p>\n

<\/p>\n

I was to follow the SRE engineer everywhere for a week. This meant I attended the handover of duty, observed the same alert channels, and responded to incidents if and when they occurred.<\/p>\n

<\/a><\/noindex><\/p>\n

Incidents<\/h2>\n

<\/p>\n

During the week, two incidents occurred.<\/p>\n

<\/p>\n

1. Crypto Miner<\/h3>\n

<\/p>\n

On Wednesday, a spike in the usage of GitLab.com was recorded, caused by attempts to use runner minutes for cryptocurrency mining. The incident was resolved using our own violation neutralization tool, which stops runner jobs and removes the related project and account. GitLab Runner<\/a><\/noindex>caused by attempts to use runner minutes for cryptocurrency mining. The incident was addressed with our own violation neutralization tool that stops runner tasks and removes the associated project and account. <\/p>\n

<\/p>\n

2. Degradation of Performance for Canary and Main Applications<\/p>\n

<\/p>\n

The incident was triggered by slowdowns and increased error rates in the canary and main web applications on GitLab.com. Several Apdex values were violated.<\/h3>\n

<\/p>\n

Open incident task:<\/p>\n

<\/p>\n

Key Takeaways https:\/\/gitlab.com\/gitlab-com\/gl-infra\/production\/issues\/1442<\/a><\/noindex><\/p>\n

<\/p>\n

Here are a few points I learned during the week on duty.<\/h2>\n

<\/p>\n

1. Alerts are most useful when they detect deviations from normal behavior.<\/p>\n

<\/p>\n

Alerts can be divided into several types:<\/h3>\n

<\/p>\n

Alerts based on a specific threshold, such as '10 5xx errors occurred per second.'<\/p>\n

<\/p>\n