{"id":92073,"date":"2020-08-22T19:41:56","date_gmt":"2020-08-22T17:41:56","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io"},"modified":"2020-08-22T19:41:56","modified_gmt":"2020-08-22T17:41:56","slug":"post-mortem-po-nedostupnosti-quay-io","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/post-mortem-po-nedostupnosti-quay-io","title":{"rendered":"Post Mortem on Quay.io Downtime","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Note: translation.<\/b>: in early August, Red Hat publicly discussed the resolution of accessibility issues that users of its service had encountered in recent months Quay.io<\/a><\/noindex> (based on a registry for container images, which came to the company with the acquisition of CoreOS). Regardless of your interest in this service itself, the journey that the SRE engineers took to diagnose and resolve the causes of the outage is enlightening.<\/i><\/p>\n

\"Post
\n
\nOn May 19, early in the morning (EDT), the quay.io service went down. The outage affected both quay.io customers and open source projects using quay.io as a platform for building and distributing software. Red Hat values the trust of both groups.<\/p>\n

The SRE engineering team immediately engaged and tried to stabilize the Quay service as quickly as possible. However, while they were working on that, customers lost the ability to push new images and could only occasionally pull existing ones. For unknown reasons, the quay.io database was locking up after scaling the service to full capacity.<\/a><\/noindex><\/p>\n

\u00abWhat Changed?<\/b>\" \u2014 this is the first question typically asked in such cases. We noticed that shortly before the problem, the OpenShift Dedicated cluster (on which quay.io runs) began updating to version 4.3.19. Since quay.io operates on Red Hat OpenShift Dedicated (OSD), regular updates had been a routine operation and had never caused issues. Moreover, over the past six months, we had updated the Quay clusters several times without any service interruptions.<\/p>\n

While we were trying to restore service, other engineers started preparing a new OSD cluster with the previous version of the software, so that everything could be deployed on it if needed.<\/p>\n

Root Cause Analysis<\/h2>\n

\nThe main symptom of the failure was an avalanche of tens of thousands of connections to the database, rendering the MySQL instance virtually inoperative. This made it difficult to diagnose the problem. We set a limit on the maximum number of client connections to help the SRE team assess the issue. No unusual traffic to the database was noted: in fact, most queries were for reading, with only a few for writing.<\/p>\n

We also tried to identify a pattern in the database traffic that could have triggered this avalanche. However, no patterns could be found in the logs. While waiting for the new cluster with OSD 4.3.18 to be ready, we continued our attempts to launch the quay.io pods. Each time the cluster reached full capacity, the database would hang. This meant it was necessary to restart the RDS instance in addition to all quay.io pods.<\/p>\n

By evening, we stabilized the service in read-only mode and disabled as many non-critical functions (such as garbage collection in the namespace) to reduce the load on the database. The hangs stopped, but the root cause was never found.<\/b>. The new OSD cluster was ready, and we moved the service, connected the traffic, and continued monitoring.<\/p>\n

Quay.io was running smoothly on the new OSD cluster, so we returned to the database logs, but we still couldn\u2019t find any correlation to explain the locks. OpenShift engineers worked with us to try to determine whether changes in Red Hat OpenShift 4.3.19 could have led to issues with Quay. However, nothing was discovered, and we were unable to reproduce the problem in the lab.<\/b>.<\/p>\n

The second failure<\/h2>\n

\non May 28, just before noon EDT, quay.io crashed again with the same symptom: the database operations were locked. Once again, we diverted all resources to the investigation. First and foremost, we needed to restore the service. However, This time, restarting the RDS and the quay.io pods did not help.<\/b>Quay is written in Python, and each pod operates as a single monolithic container. Within the container runtime, multiple parallel tasks are executed simultaneously. We use the<\/p>\n

gevent library for handling web requests. When a request comes to Quay (via our own API or through the Docker API), a gevent worker is assigned to it. This worker is usually supposed to connect to the database. After the first failure, we found that the gevent workers connected to the database using the default settings.<\/code> under gunicorn<\/code> For processing web requests. When a request comes into Quay (through our own API or through Docker's API), a gevent worker is assigned. Typically, this worker should connect to the database. After the first failure, we found that the gevent workers connected to the database using default settings.<\/p>\n

Given the significant number of Quay pods and the thousands of incoming requests per second, a large number of database connections could theoretically overload the MySQL instance. Monitoring indicated that Quay was handling an average of 5,000 requests per second. Approximately the same number of connections to the database was noted. 5,000 connections comfortably fit within the capabilities of our RDS instance (unlike the tens of thousands). For some reason, there were unexpected spikes in the number of connections<\/b>, however, we did not observe any correlation with incoming requests.<\/p>\n

This time we resolutely decided to find and eliminate the source of the problem rather than just rebooting. Changes were made in the Quay codebase Changes were made to limit the number of database connections for each worker.<\/b> gevent. This number became a configurable parameter: it was possible to change it 'on the fly' without building a new container image. To determine the number of connections that could actually be handled, we conducted several tests in a staging environment, where different values were set to see how it would affect load testing scenarios. Ultimately, it turned out that Quay starts returning 502 errors when the number of connections exceeds 10,000.<\/b><\/p>\n

We immediately deployed this new version to production and began monitoring the database connection graph. Previously, the database would lock up after about 20 minutes. After 30 problem-free minutes, we felt hopeful, and after an hour \u2014 confident. We restored write traffic to the site and began the postmortem analysis.<\/p>\n

By managing to bypass the problem that led to the blockage, we did not uncover its true causes<\/b>. It was confirmed that it is not related to any changes in OpenShift 4.3.19, as the same issue occurred on version 4.3.18, which had previously worked with Quay without any problems.<\/p>\n

Clearly, something else was hidden in the cluster.<\/p>\n

A detailed examination<\/h2>\n

\nQuay.io used default settings for connecting to the database for six years without any issues. What has changed? It is clear that traffic to quay.io has been steadily increasing all this time. In our case, it seemed like some threshold value had been reached, triggering a surge in connections. We continued to analyze the database logs after the second crash but found no patterns or obvious correlations.<\/p>\n

Meanwhile, the SRE team was working on improvements in request observability in Quay and the overall health of the service. New metrics and dashboards were deployed<\/b>, showing which parts of Quay are most in demand by customers.<\/p>\n

Quay.io was functioning normally until June 9. In the morning (EDT), we again witnessed a significant increase in the number of database connections. This time, there was no downtime<\/b>, as a new parameter limited their number and prevented exceeding MySQL's capacity. However, for about half an hour, many users reported slow performance on quay.io. We quickly gathered all possible data using the added monitoring tools. Suddenly, a pattern emerged.<\/p>\n

Right before the spike in connections, a large number of requests hit the App Registry API<\/b>. The App Registry is a lesser-known feature of quay.io. It allows storing things like Helm charts and containers with rich metadata. Most quay.io users do not work with this feature, but it is actively used by Red Hat OpenShift. The OperatorHub within OpenShift hosts all the operators in the App Registry. These operators form the foundation for the OpenShift workload ecosystem and operational model (within Day 2 operations) aimed at partners.<\/p>\n

Each OpenShift 4 cluster uses operators from the built-in OperatorHub to publish the catalog of available operators for installation and provide updates for those already installed. With the growing popularity of OpenShift 4, the number of clusters worldwide has also increased. Each of these clusters loads the contents of operators to launch the built-in OperatorHub, using App Registry within quay.io as a backend. In the search for the source of the problem, we overlooked the fact that as OpenShift's popularity grew, the load on one of the rarely used features of quay.io also increased.<\/b>.<\/p>\n

We conducted some analysis of the request traffic to the App Registry and looked into the registry code. Immediately, shortcomings were revealed, causing database requests to be formed suboptimally. Under light load, they caused no issues; however, as the load increased, they became problematic. The App Registry had two problematic endpoints that poorly responded to increased load: the first returned a list of all packages in the repository, and the second returned all blobs for a package.<\/p>\n

Addressing the causes<\/h2>\n

\nFor the entire following week, we focused on optimizing the code of the App Registry itself and its environment. Clearly inefficient SQL queries were rewritten, unnecessary command calls were eliminated (which were triggered on every blob extraction), and caching was added wherever possible. Then, extensive performance testing was conducted to compare the speed of the App Registry before and after the changes. tar<\/code> (it was triggered on every blob extraction), caching was added wherever possible. Then extensive performance testing was conducted and the speed of App Registry was compared before and after the changes.<\/p>\n

. The following week, we deployed the changes to production, and since then, quay.io has been running smoothly. During this time, there were several sharp spikes in traffic on the App Registry endpoint, but the improvements made prevented database outages.<\/b>. Next week, we deployed the changes to production, and since then quay.io has been stable. During this time, there were several sharp spikes in traffic at the App Registry endpoint, but the improvements made prevented database outages.<\/p>\n

It is clear that any service strives to avoid downtime. In our case, we believe that the recent failures helped make quay.io better. We took away several key lessons that we would like to share:<\/h2>\n

\nData on who and how uses your service is never unnecessary.<\/p>\n

    \n
  1. Since Quay was 'just working', we never felt the need to spend time optimizing traffic and managing load. All of this created a false sense of security that the service could scale indefinitely.<\/b>When a service goes down,<\/li>\n
  2. restoring its operation is the top priority. restoring its functionality is the main priority<\/b>As Quay continued to suffer from a locked database during the first outage, our standard procedures did not have the intended effect, and we were unable to restore service with their help. This led to a situation where we had to spend time analyzing and collecting data in hopes of finding the root cause \u2014 instead of directing all efforts toward restoring functionality.<\/li>\n
  3. Assess the impact of each service feature.<\/b>Clients rarely used the App Registry, so it was not a priority for our team. When certain product features are hardly utilized, their bugs rarely 'surface', and developers stop monitoring the code. It\u2019s easy to fall victim to the misconception that this is how it should be \u2014 until suddenly that feature becomes the center of a major incident.<\/li>\n<\/ol>\n

    <\/p>\n

    What's next?<\/h2>\n

    \nWork on ensuring service stability never stops, and we continuously improve it. Traffic volumes on quay.io continue to grow, and we recognize that we must do everything possible to justify our clients' trust. Therefore, we are currently working on the following tasks:<\/p>\n

      \n
    1. Deploying read-only database replicas to help the service handle the corresponding traffic in case there are issues with the primary RDS instance.<\/li>\n
    2. Updating the RDS instance. The current version is not a problem in itself. Rather, we just want to eliminate the false lead (that we followed during the outage); keeping the software up to date will remove another factor in case of future outages.<\/li>\n
    3. Additional caching throughout the cluster. We continue to look for areas where caching can reduce the load on the database.<\/li>\n
    4. Adding a web application firewall (WAF) to see who connects to quay.io and why.<\/li>\n
    5. Starting with the next release, Red Hat OpenShift clusters will discontinue the App Registry in favor of Operator Catalogs, which are based on container images available on quay.io.<\/li>\n
    6. A long-term replacement for the App Registry may be support for the Open Container Initiative (OCI) artifact specifications. This is currently being implemented as native functionality in Quay and will be available to users once the specification is finalized.<\/li>\n<\/ol>\n

      \nAll of the above is part of Red Hat's ongoing investment in quay.io as we transition from a small, startup-like team to a mature, SRE-managed platform. We understand that many of our clients rely on quay.io for their daily operations (including Red Hat!) and we strive to be as transparent as possible regarding recent outages and our ongoing efforts to improve.<\/p>\n

      P.S. from the translator<\/h2>\n

      \nAlso read in our blog:<\/p>\n