SRE Engineer - Intern
Let me introduce myself. I am , a frontend engineer in the team at GitLab. Last week, I had the privilege of being an intern with one of our on-call SRE engineers. The goal was to observe daily how the on-call engineer reacts to incidents and gain real work experience. We want our engineers to better understand user needs. Monitor::Health.
I was to follow the SRE engineer everywhere for a week. This meant I attended the handover of duty, observed the same alert channels, and responded to incidents if and when they occurred.
Incidents
During the week, two incidents occurred.
1. Crypto Miner
On Wednesday, a spike in the usage of GitLab.com was recorded, caused by attempts to use runner minutes for cryptocurrency mining. The incident was resolved using our own violation neutralization tool, which stops runner jobs and removes the related project and account. Had this event not been noticed, an automated tool would have caught it, but in this case, the SRE engineer noticed the violation first. An incident task was created, but the information regarding it is closed.
2. Degradation of Performance for Canary and Main Applications
The incident was triggered by slowdowns and increased error rates in the canary and main web applications on GitLab.com. Several Apdex values were violated.
Open incident task:
Key Takeaways
Here are a few points I learned during the week on duty.
1. Alerts are most useful when they detect deviations from normal behavior.
Alerts can be divided into several types:
Alerts based on a specific threshold, such as '10 5xx errors occurred per second.'
- Alerts where the threshold is a percentage, like 'the frequency of 5xx errors is 10% of the total request volume over a specified time.'
- Alerts based on a historical mean value, such as '5xx errors in the 90th percentile.'
- Generally speaking, the 2nd and 3rd types are more useful for on-call SREs as they reveal deviations from normal behavior in the process.
2. Many alerts do not escalate to incidents.
SRE engineers deal with a constant stream of alerts, many of which are not actually critical.
SR engineers are faced with a constant flow of alerts, many of which are not critical.
So, why not limit alerts to only the truly important ones? With this approach, however, one might miss the early signs of something that can snowball into a real problem, threatening significant damage.
The duty of an SRE on call is to determine which alerts genuinely indicate something serious, and whether they need to be escalated and investigated. I suspect this is also due to the rigidity of alerts: it would be better if multiple levels or 'smart' ways of configuring alerts according to the situation described above were introduced.
Feature suggestion:
3. Our on-call SREs use many tools
Internal:
- GitLab infra project: this is where Runbooks, on-call handoffs for shifts/weeks, and incident response tasks are stored.
- GitLab issues: investigations, reviews, and maintenance are also tracked in issues.
- GitLab labels: automation tasks are triggered by specific labels that bots monitor for task activity.
External:
- PagerDuty: alerts
- Slack: this is where the stream of PagerDuty/AlertManager messages is directed. Integration with slash commands for executing various tasks, such as closing alerts or escalating to an incident.
- Grafana: visualization of metrics with a focus on long-term trends.
- Kibana: provides visualization/search in logs, allowing deeper dives into specific events.
- Zoom: there is a constantly running 'discussion room' in Zoom. This allows SRE engineers to quickly discuss events without wasting precious time creating a room and sending out links to participants.
And much, much more.
4. Monitoring GitLab.com with GitLab is a single point of failure
If there is a major service outage on GitLab.com, we would not want that to affect our ability to resolve the issue. It can be mitigated by launching a second instance of GitLab to manage GitLab.com. In fact, this is already operational for us: .
5. Several features to consider adding to GitLab
- , similar to Google Docs. This would assist in incident tasks during an event, as well as in review tasks. In both cases, multiple participants may need to add something in real-time.
- More webhooks for tasks. The ability to trigger different steps of the GitLab workflow internally will help reduce reliance on Slack integrations. For example, the option to enable notifications in PagerDuty through a slash command in a GitLab task.
Conclusion
SRE engineers face many challenges. It would be great to see more GitLab products addressing these issues. We are already working on some additions to the product that will ease the workflows mentioned above. Details are available in .
In 2020, we are expanding the team to build all these amazing features. If you're interested, please check out , and feel free to reach out to anyone on our team with any questions.
Source: habr.com
