Hello everyone. I work as a lead system administrator at OK and I'm responsible for the stable operation of the portal. I want to share how we structured the process of automatic disk replacement, and then how we eliminated the administrator from this process and replaced them with a bot.
This article is a kind of transliteration at HighLoad+ 2018
Building the Disk Replacement Process
First, some numbers
OK is a massive service used by millions of people. It is supported by about 7,000 servers located in 4 different data centers. These servers house over 70,000 disks. If we stack them on top of each other, they would form a tower more than 1 km high.
Hard drives are the component of the server that fails most often. With such volumes, we have to replace about 30 disks a week, and this procedure has become a rather unpleasant routine.
Incidents
Our company has implemented a comprehensive incident management system. We log each incident in Jira and then resolve and analyze it. If an incident had an effect on users, we always hold a meeting to discuss how to respond more quickly in such cases, how to reduce the impact, and of course how to prevent recurrence.
Drives are no exception. Their status is monitored by Zabbix. We monitor Syslog messages for read/write errors, analyze the status of HW/SW RAID systems, and keep an eye on SMART, calculating wear for SSDs.
How Drives Were Replaced Previously
When any trigger lights up in Zabbix, an incident is created in Jira and is automatically assigned to the corresponding engineers in the data centers. We do this for all HW incidents, meaning those that require any physical work with the equipment in the data center.
A data center engineer is a person who addresses hardware-related issues, responsible for the installation, maintenance, and deinstallation of servers. Upon receiving a ticket, the engineer begins work. They independently replace disks in the disk shelves. However, if they lack access to the necessary device, the engineer reaches out to the on-duty system administrators for assistance. First, the disk needs to be taken out of rotation. For this, necessary changes must be made on the server, applications stopped, and the disk unmounted.
The on-duty system administrator is responsible for the operation of the entire portal during their working shift. They investigate incidents, carry out repairs, and assist developers with small tasks. They do not handle hard drives.
Previously, data center engineers communicated with the system administrator in chat. Engineers sent links to Jira tickets, and the administrator would check them, keeping a work log in a notebook. However, chats are inconvenient for such tasks: the information is unstructured and quickly gets lost. Moreover, the administrator might step away from the computer and not respond to requests for a while, leaving the engineer waiting at the server with a stack of disks.
The worst part was that administrators could not see the overall picture: what disk incidents existed and where potential problems could arise. This is because we assign all hardware incidents to engineers. Yes, it was possible to display all incidents on the administrator's dashboard. However, there are too many, and the administrator was only involved in some of them.
Additionally, the engineer could not prioritize correctly because they didn't know the purpose of specific servers or how information was distributed across storage devices.
New Replacement Procedure
The first thing we did was create a separate type 'HW-Disk' for all disk incidents and added fields for 'block device name,' 'size,' and 'disk type' so that this information would be saved in the ticket, eliminating the need for constant exchanges in chat.
We also agreed that within a single incident, we would only change one disk. This significantly simplified the subsequent automation process, statistics collection, and workflow.
Additionally, we added a field for the 'responsible administrator.' The on-duty sysadmin is automatically assigned there. This is very convenient because now the engineer always knows who is responsible, eliminating the need to check the calendar.
To ensure that all participants benefit maximally from the innovations, we created filters and dashboards and informed the team about them. When people understand the changes, they do not distance themselves from them as if they were something unnecessary. An engineer needs to know the rack number where the server is located, as well as the size and type of the disk. It is crucial for the administrator to first understand what group of servers this is and what effect may occur upon replacing the disk.
Having fields and their display is convenient, but it did not free us from the need to use chats. This required us to change our workflow.
Previously, it was like this:
Today, this is how engineers continue to work when they do not require assistance from an administrator.
The first thing we did was to introduce a new status Investigate. In this status, the ticket remains while the engineer has yet to decide whether they will need an administrator or not. Through this status, the engineer can pass the ticket to the administrator. Moreover, we mark the tickets with this status when a disk replacement is required, but the disk itself is not available on-site. This can happen in the case of CDN and remote sites.
We also added the status Ready. The ticket is transferred to this status after the disk is replaced. That is, everything is already done, but the HW/SW RAID on the server is synchronizing. This can take quite a bit of time.
If an administrator is involved in the work, the scheme becomes a little more complicated.
From the status Open the ticket can be transferred both by the system administrator and the engineer. In the status In progress , the administrator takes the disk out of rotation so that the engineer can simply remove it: they turn on the light, unmount the disk, stop applications, depending on the specific group of servers.
Then the ticket is transferred to Ready to change: this is a signal to the engineer that the disk can be pulled out. All fields in Jira are already filled in, and the engineer knows the type and size of the disk. This information is either automatically filled in during the previous status or by the administrator.
After the disk replacement, the ticket is transferred to the status Changed. It is checked whether the required disk has been inserted, marking is done, the application is launched, and some recovery tasks are performed. The ticket can also be transferred to the status Ready, in which case the administrator remains responsible because they introduced the disk into rotation. The full scheme looks like this.
The addition of new fields has significantly improved our workflow. The team started working with structured information, making it clear what is needed and at what stage. Priorities have become much more relevant since they are now set by the administrator.
There’s no longer a need for chats. Of course, the administrator can message the engineer, saying, 'this needs to be replaced faster' or 'it’s evening, will you make the replacement in time?'. But we no longer communicate daily in chats about these matters.
Disks are now replaced in batches. If an administrator arrives a bit earlier for their shift and has some spare time, they can prepare a series of servers for replacement: marking fields, removing disks from rotation, and assigning the task to the engineer. The engineer comes to the data center later, sees the task, collects the required drives from storage, and replaces them immediately. As a result, the speed of replacement has increased.
Experience gained in building the Workflow
- When creating a procedure, it is necessary to gather information from various sources.
Some of our administrators were unaware that engineers change disks independently. Some thought that the engineers monitored MD RAID synchronization, although some didn’t even have access to do so. A few senior engineers handled this, but not always, as the process wasn’t documented anywhere. - The procedure should be simple and clear.
It's difficult for a person to keep many steps in mind. The most important adjacent statuses in Jira should be displayed on the main screen. For example, we call In progress 'Ready to change'. Other statuses can be hidden in a dropdown menu to avoid clutter. But it’s best not to restrict people and allow them to make transitions.
Clarify the value of innovations. When people understand, they are more likely to accept the new procedure. It was crucial for us that people didn’t just click through the process but actually followed it. Later, we built automation on top of this. - Wait, analyze, understand.
It took us about a month to build the procedure, implement the technical aspects, hold meetings, and discussions. The actual deployment took over three months. I saw how people gradually started to use the new feature. Initially, there was a lot of negativity. However, it had nothing to do with the procedure itself or its technical implementation. For example, one administrator used a Jira plugin in Confluence instead of Jira, which made certain tasks unavailable to him. Once we showed him Jira, his productivity improved both in general tasks and in disk replacements.
Disk Replacement Automation
We attempted automation for disk replacements several times. We already had some scripts and tools, but they either worked in interactive mode or required manual initiation. Only after implementing the new procedure did we realize it was exactly what we were lacking.
Since we now break the replacement process into stages, with each stage having a designated executor and a list of actions, we can gradually incorporate automation instead of implementing it all at once. For example, the simplest stage—Ready (checking RAID/data synchronization)—can easily be delegated to a bot. Once the bot learns a bit, we can assign it more responsible tasks, such as introducing a disk into rotation, etc.
A Zoo of Setups
Before introducing the bot, let's take a brief tour of our setup zoo. This is primarily due to the massive scale of our infrastructure. Secondly, we strive to select the optimal hardware configuration for each service. We have about 20 models of hardware RAID, mostly from LSI and Adaptec, but we also see various versions from HP and DELL. Each RAID controller comes with its own management utility. The command sets and outputs may vary from version to version for each RAID controller. In areas where HW-RAID isn't used, we may have mdraid.
Almost all new installations are done without disk redundancy. We try to avoid using hardware and software RAID since we back up our systems at the data center level rather than at the server level. However, there are certainly many legacy servers that need to be supported.
Somewhere disks in RAID controllers are presented as raw devices, while in other cases JBOD is used. There are configurations with a single system disk in the server, and if it needs to be replaced, the server must be redeployed with the OS and applications installed, and those applications must be the same versions. Configuration files then need to be added, and applications need to be started. Many groups of servers also have redundancy implemented not at the disk subsystem level, but directly within the applications themselves.
In total, we have over 400 unique server groups running about 100 different applications. To cover such a vast number of configurations, we needed a multifunctional automation tool. Ideally, it should have a simple DSL so that maintenance could be handled by anyone, not just the person who wrote it.
We chose Ansible because it is agentless: there was no need to prepare the infrastructure, and it offers a quick start. Additionally, it is written in Python, which is the accepted standard in the team.
Overall Scheme
Let's consider the overall automation scheme using one incident as an example. Zabbix detects that disk sdb has failed, a trigger activates, and a ticket is created in Jira. The administrator reviews it, realizes it is neither a duplicate nor a false positive, determines that the disk needs to be replaced, and moves the ticket to In Progress.
The DiskoBot application, written in Python, periodically polls Jira for new tickets. It notices that a new ticket In Progress has appeared, the corresponding thread is activated, which launches a playbook in Ansible (this is done for each status in Jira). In this case, Prepare2change is executed.
Ansible is sent to the host, removes the disk from rotation, and reports the status back to the application via Callbacks.
As a result, the bot automatically moves the ticket to Ready to Change. The engineer receives a notification and goes to change the disk, after which they move the ticket to Changed.
Following the aforementioned scheme, the ticket goes back to the bot, which launches another playbook, accesses the host, and reintroduces the disk into rotation. The bot then closes the ticket. Hooray!
Now let's discuss some components of the system.
Diskobot
This application is written in Python. It selects tickets from Jira based on JQL. Depending on the ticket's status, it is routed to the corresponding handler, which in turn executes the Ansible playbook that corresponds to that status.
JQL and polling intervals are defined in the application's configuration file.
jira_states:
investigate:
jql: '… status = Open and "Disk Size" is EMPTY'
interval: 180
inprogress:
jql: '… and "Disk Size" is not EMPTY and "Device Name" is not EMPTY'
ready:
jql: '… and (labels not in ("dbot_ignore") or labels is EMPTY)'
interval: 7200
For example, among tickets in the In progress status, only those with the Disk size and Device name fields filled are selected. Device name is the name of the block device needed to execute the playbook. Disk size is necessary for the engineer to know what size disk is required.
Among the tickets with the Ready status, tickets labeled with dbot_ignore are filtered out. By the way, we use Jira labels for both this kind of filtering and marking duplicates, as well as for gathering statistics.
In case of a playbook failure, Jira assigns the dbot_failed label for further investigation.
Interaction with Ansible
The application interacts with Ansible through . In playbook_executor, we pass the filename and a set of variables. This allows us to maintain the Ansible project in the form of regular yml files instead of describing it in Python code.
Also, in Ansible, the name of the block device, ticket status, and callback_url, which contains the issue key, are passed through *extra_vars*—this is used for the callback in HTTP.
A temporary inventory is generated for each run, consisting of a single host and a group that includes this host, so that group_vars are applied.
Here's an example task that implements the HTTP callback.
We receive the results of playbook executions through callbacks. There are two types:
- , it provides data on the results of the playbook execution. It describes the tasks that were started, successfully completed or failed. This callback is invoked at the end of the playbook execution.
- HTTP callback to receive information during playbook execution. In the Ansible task, we perform a POST/GET request to our application.
Through HTTP callbacks, variables defined during the execution of the playbook are passed, which we want to save and use in subsequent runs. We write this data in sqlite.
Also, through HTTP callbacks, we leave comments and change the ticket status.
HTTP callback
# Make callback to Diskobot App
# Variables:
# callback_post_body: # A dict with follow keys. All keys are optional
# msg: If exist it would be posted to Jira as comment
# data: If exist it would be saved in Incident.variables
# desire_state: Set desire_state for incident
# status: If exist Proceed issue to that status
- name: Callback to Diskobot app (jira comment/status)
uri:
url: "{{ callback_url }}/{{ devname }}"
user: "{{ diskobot_user }}"
password: "{{ diskobot_pass }}"
force_basic_auth: True
method: POST
body: "{{ callback_post_body | to_json }}"
body_format: json
delegate_to: 127.0.0.1
Like many similar tasks, we moved it to a separate common file and include it when necessary, to avoid repeatedly writing it in playbooks. Here, we reference the callback_url, which contains the issue key and host name. When Ansible executes this POST request, the bot understands that it is related to a specific incident.
Here is an example from the playbook where we removed a disk from an MD device:
# Save mdadm configuration
- include: common/callback.yml
vars:
callback_post_body:
status: 'Ready to change'
msg: "Removed disk from mdraid {{ mdadm_remove_disk.msg | comment_jira }}"
data:
mdadm_data: "{{ mdadm_remove_disk.removed }}"
parted_info: "{{ parted_info | default() }}"
when:
- mdadm_remove_disk | changed
- mdadm_remove_disk.removed
This task transitions the Jira ticket to the status "Ready to change" and adds a comment. Also, the variable mdam_data contains a list of md-devices from which the disk was removed, while parted_info holds the partition dump from parted.
When the engineer inserts a new disk, we can use these variables to restore the partition dump and add the disk back to the md-devices from which it was removed.
Ansible check mode
It was scary to enable automation. Therefore, we decided to run all the playbooks in
mode, where Ansible does not perform any actions on the servers but only emulates them.
Such a run goes through a separate callback module, and the results of the playbook execution are saved in Jira as comments.
Firstly, this allowed us to validate the bot's functionality and the playbooks. Secondly, it increased the administrators' trust in the bot.
Once we validated everything and realized we could run Ansible not just in dry run mode, we created a Run Diskobot button in Jira to trigger the same playbook with the same variables on the same host, but in normal mode.
Additionally, the button is used to rerun the playbook in case it fails.
Structure of Playbooks
I already mentioned that depending on the status of the Jira ticket, the bot triggers different playbooks.
Firstly, this makes it much easier to manage incoming requests.
Secondly, in some cases, it is simply necessary.
For example, when replacing the system disk, the first step is to go to the deployment system, create a task, and after successful deployment, the server will be accessible via ssh, allowing us to deploy the application. If we were to do all this in one playbook, Ansible would not be able to execute it due to the host's unavailability.
We use Ansible roles for each group of servers. Here you can see how the playbook(s) are organized within one of them.
This is convenient because it immediately shows where the tasks are located. In main.yml, which serves as the entry point for the Ansible role, we can simply include tasks by ticket status or general tasks needed for all, such as completing identification or obtaining a token.
Investigation.yml
It is triggered for tickets in Investigation and Open status. The most important aspect of this playbook is the name of the block device. This information is not always available.
To obtain it, we analyze the Jira summary and the latest value from the Zabbix trigger. It may contain the name of the block device — lucky us. It might also contain the mount point — in that case, we need to go to the server, parse it, and determine the required disk. The trigger may also provide the SCSI address or some other information. However, sometimes there are no clues, and we have to analyze.
Having determined the name of the block device, we gather information about its type and size to fill in the fields in Jira. We also collect details about the vendor, model, firmware, ID, SMART, and insert all of this into the comment on the Jira ticket. The administrator and engineer no longer need to look for this information. 🙂
prepare2change.yml
Removing the disk from rotation, preparing for replacement. This is the most complex and responsible stage. It's at this point that you can stop the application when it cannot be stopped. Or you might pull out a disk that was missing replicas, thus impacting users and potentially losing some data. Here we have the most checks and notifications in the chat.
In the simplest case, this involves removing a disk from HW/MD RAID.
In more complex situations (in our storage systems), when redundancy is performed at the application level, we need to access the application via API, report the disk removal, deactivate it, and initiate recovery.
We are currently migrating en masse to , and if the server is cloud-based, Diskobot interacts with the cloud's API, stating it is going to work with this minion — the server running the containers — and requests 'migrate all containers from this minion'. It also activates disk highlighting so the engineer can immediately see which one needs to be pulled out.
changed.yml
After replacing the disk, we first check its availability.
Engineers do not always install new disks, so we added a check for acceptable SMART values.
What attributes are we looking atReallocated Sectors Count (5) < 100
Current Pending Sector Count (107) == 0
If the disk fails the check, the engineer is notified for a replacement. If everything is fine, the light is turned off, marking is applied, and the disk is put into rotation.
ready.yml
The simplest case: checking HW/SW RAID synchronization or finishing data synchronization in an application.
Application API
I have mentioned several times that often the bot accesses the application API. Of course, not all applications had the necessary methods, so we had to improve them. Here are the most important methods we use:
- Status. Status of the cluster or disk to understand whether it can be operated;
- Start/stop. Activating-deactivating the disk;
- Migrate/restore. Migration and restoration of data during and after replacement.
Experience gained with Ansible
I really love Ansible. However, often when I look at various open-source projects and see how people write playbooks, I get a bit scared. Complex logical intertwining of when/loop, lack of flexibility and idempotence due to frequent use of shell/command.
We decided to simplify everything as much as possible, taking advantage of Ansible's modularity. At the highest level are playbooks, which can be written by any administrator or third-party developer who has a little knowledge of Ansible.
- name: Blink disk
become: True
register: locate_action
disk_locate:
locate: '{{ locate }}'
devname: '{{ devname }}'
ids: '{{ locate_ids | default(pd_id) | default(omit) }}'
If some logic is difficult to implement in playbooks, we move it to an Ansible module or filter. Scripts can be written in Python or any other language.
They are easy and quick to write. For example, the disk blinking module, the usage example of which is given above, consists of 265 lines.
At the lowest level is the library. For this project, we wrote a separate application, a kind of abstraction over hardware and software RAID that performs the corresponding queries.
The strongest points of Ansible are its simplicity and clear playbooks. I believe we should use this and not generate scary YAML files with a huge number of conditions, shell code, and loops.
If you want to replicate our experience with the Ansible API, keep in mind two things:
- The Playbook_executor and the playbook itself cannot be assigned a timeout. There is a timeout on SSH sessions, but no timeout for the playbook. If we attempt to unmount a disk that no longer exists in the system, the playbook will run indefinitely, so we had to wrap its execution in a separate wrapper and kill it after a timeout.
- Ansible operates on a fork-process basis, so its API is not thread-safe. We run all our playbooks in a single-threaded manner.
As a result, we managed to automate the replacement of about 80% of the disks. Overall, the replacement speed has doubled. Today, the administrator simply looks at the incident and decides whether to replace the disk or not, then makes a single click.
But now we are starting to face another problem: some new administrators do not know how to replace disks. 🙂
Source: habr.com
