ProHoster > Blog > Administration > Kubernetes YAML validation against best practices and policies

Kubernetes YAML validation against best practices and policies

Note. transl.: With the growth in the number of YAML configurations for K8s environments, the need for automated verification of them becomes more and more urgent. The author of this review not only selected existing solutions for this task, but also looked at how they work using the example of Deployment. It turned out to be very informative for those who are interested in this topic.

Kubernetes YAML validation against best practices and policies

TL; DR: This article compares six static tools for validating and evaluating Kubernetes YAML files against best practices and requirements.

Kubernetes workloads are typically defined in the form of YAML documents. One of the problems with YAML is the difficulty of setting limits or relationships between manifest files.

What if we need to make sure that all images deployed in the cluster come from a trusted registry?

How to prevent sending Deployments to the cluster that do not have PodDisruptionBudgets set?

Static testing integration allows you to detect errors and policy violations at the development stage. This increases the assurance that resource definitions are correct and secure, and increases the likelihood that production workloads will follow best practices.

The Kubernetes YAML static validation ecosystem can be divided into the following categories:

  • API Validators. The tools in this category validate the YAML manifest against the requirements of the Kubernetes API server.
  • Ready testers. Tools from this category come with ready-made tests for security, compliance with best practices, etc.
  • Custom validators. Members of this category allow you to create custom tests in various languages, such as Rego and Javascript.

In this article, we will describe and compare six different tools:

  1. kubeval;
  2. kube-score;
  3. config-lint;
  4. copper;
  5. conftest;
  6. polaris.

Well, let's get started!

Checking Deployments

Before we start comparing tools, let's create some base on which to test them.

The manifest below contains a number of errors and inconsistencies with best practices: how many can you find?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo

(base-valid.yaml)

We will use this YAML to compare different tools.

The above manifest base-valid.yaml and other manifests from this article can be found in Git repositories.

The manifest describes a web application whose main task is to respond with a "Hello World" message on port 5678. It can be deployed with the following command:

kubectl apply -f hello-world.yaml

And so - to check the work:

kubectl port-forward svc/http-echo 8080:5678

Now go to http://localhost:8080 and confirm that the application is running. But does it follow best practices? Let's check.

1. Kubeval

At the heart kubeval lies the idea that any interaction with Kubernetes occurs through its REST API. In other words, you can use an API schema to check if a given YAML conforms to it. Let's look at an example.

Installation instructions kubeval are available on the project website.

At the time of writing the original article, version 0.15.0 was available.

Once installed, let's feed it the manifest above:

$ kubeval base-valid.yaml
PASS - base-valid.yaml contains a valid Deployment (http-echo)
PASS - base-valid.yaml contains a valid Service (http-echo)

If successful, kubeval will exit with an exit code of 0. You can check it as follows:

$ echo $?
0

Let's now try kubeval with a different manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo

(kubeval-invalid.yaml)

Can you spot the problem? We launch:

$ kubeval kubeval-invalid.yaml
WARN - kubeval-invalid.yaml contains an invalid Deployment (http-echo) - selector: selector is required
PASS - kubeval-invalid.yaml contains a valid Service (http-echo)

# ΠΏΡ€ΠΎΠ²Π΅Ρ€ΠΈΠΌ ΠΊΠΎΠ΄ Π²ΠΎΠ·Π²Ρ€Π°Ρ‚Π°
$ echo $?
1

The resource is not validated.

Deployments using the API version apps/v1, should include a selector that matches the pod's label. The manifest above does not include a selector, so kubeval reported an error and exited with a non-zero code.

I wonder what happens if you run kubectl apply -f with this manifest?

Well, let's try:

$ kubectl apply -f kubeval-invalid.yaml
error: error validating "kubeval-invalid.yaml": error validating data: ValidationError(Deployment.spec):
missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors,
turn validation off with --validate=false

Exactly the error that kubeval warned about. You can fix it by adding a selector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:          # !!!
    matchLabels:     # !!!
      app: http-echo # !!!
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo

(base-valid.yaml)

The advantage of tools like kubeval is that these errors can be caught early in the deployment cycle.

In addition, these checks do not require access to the cluster: they can be performed offline.

By default, kubeval checks resources against the latest Kubernetes API schema. However, in most cases, you may need to validate against a specific Kubernetes release. You can do this with the flag --kubernetes-version:

$ kubeval --kubernetes-version 1.16.1 base-valid.yaml

Note that the version must be in the format Major.Minor.Patch.

For a list of versions for which verification is supported, refer to JSON Schema on GitHub, which kubeval uses for validation. If you want to run kubeval offline, download the schemas and specify their local location using the flag --schema-location.

In addition to individual YAML files, kubeval can also work with directories and stdin.

In addition, Kubeval easily integrates into the CI pipeline. Those who wish to run tests before pushing manifests to the cluster will be happy to know that kubeval supports three output formats:

  1. Plain text;
  2. JSON;
  3. Test Anything Protocol (TAP).

And any of the formats can be used to further parse the output to form a summary of the results of the desired kind.

One of the downsides of kubeval is that it does not currently check for Custom Resource Definitions (CRDs). However, kubeval can be configured ignore them.

Kubeval is a great resource validation and evaluation tool; however, it should be emphasized that passing the test does not guarantee that the resource complies with best practices.

For example, using the tag latest in a container does not meet best practices. However, kubeval does not consider this a bug and does not report it. That is, validation of such YAML will complete without warning.

But what if you need to evaluate YAML and detect violations like the tag latest? How to validate a YAML file against best practices?

2. Kube score

Kube score parses YAML manifests and evaluates them against built-in tests. These tests are selected based on security recommendations and best practices, such as:

  • Running a container not as root.
  • Availability of pod health checks.
  • Setting requests and limits for resources.

The test results in three results: OK, WARNING ΠΈ CRITICAL.

Kube-score can be tried online or installed locally.

At the time of writing the original article, the latest version of kube-score is 1.7.0.

Let's test it on our manifest base-valid.yaml:

$ kube-score score base-valid.yaml

apps/v1/Deployment http-echo
[CRITICAL] Container Image Tag
  Β· http-echo -> Image with latest tag
      Using a fixed tag is recommended to avoid accidental upgrades
[CRITICAL] Pod NetworkPolicy
  Β· The pod does not have a matching network policy
      Create a NetworkPolicy that targets this pod
[CRITICAL] Pod Probes
  Β· Container is missing a readinessProbe
      A readinessProbe should be used to indicate when the service is ready to receive traffic.
      Without it, the Pod is risking to receive traffic before it has booted. It is also used during
      rollouts, and can prevent downtime if a new version of the application is failing.
      More information: https://github.com/zegl/kube-score/blob/master/README_PROBES.md
[CRITICAL] Container Security Context
  Β· http-echo -> Container has no configured security context
      Set securityContext to run the container in a more secure context.
[CRITICAL] Container Resources
  Β· http-echo -> CPU limit is not set
      Resource limits are recommended to avoid resource DDOS. Set resources.limits.cpu
  Β· http-echo -> Memory limit is not set
      Resource limits are recommended to avoid resource DDOS. Set resources.limits.memory
  Β· http-echo -> CPU request is not set
      Resource requests are recommended to make sure that the application can start and run without
      crashing. Set resources.requests.cpu
  Β· http-echo -> Memory request is not set
      Resource requests are recommended to make sure that the application can start and run without crashing.
      Set resources.requests.memory
[CRITICAL] Deployment has PodDisruptionBudget
  Β· No matching PodDisruptionBudget was found
      It is recommended to define a PodDisruptionBudget to avoid unexpected downtime during Kubernetes
      maintenance operations, such as when draining a node.
[WARNING] Deployment has host PodAntiAffinity
  Β· Deployment does not have a host podAntiAffinity set
      It is recommended to set a podAntiAffinity that stops multiple pods from a deployment from
      being scheduled on the same node. This increases availability in case the node becomes unavailable.

YAML passes kubeval checks, while kube-score points to the following shortcomings:

  • Readiness checks are not configured.
  • There are no requests and limits for CPU and memory resources.
  • Pod disruption budgets not set.
  • There are no rules for separate existence (anti-affinity) to maximize availability.
  • The container runs as root.

All of these are reasonable remarks about shortcomings that should be addressed in order for Deployment to become more efficient and reliable.

Team kube-score outputs information in a human-readable form, including all violations of the type WARNING ΠΈ CRITICALwhich helps a lot during development.

Those wishing to use this tool as part of a CI pipeline can enable more compressed output using the flag --output-format ci (in this case, tests with the result are also displayed OK):

$ kube-score score base-valid.yaml --output-format ci

[OK] http-echo apps/v1/Deployment
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Image with latest tag
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: The pod does not have a matching network policy
[CRITICAL] http-echo apps/v1/Deployment: Container is missing a readinessProbe
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Container has no configured security context
[CRITICAL] http-echo apps/v1/Deployment: No matching PodDisruptionBudget was found
[WARNING] http-echo apps/v1/Deployment: Deployment does not have a host podAntiAffinity set
[OK] http-echo v1/Service
[OK] http-echo v1/Service
[OK] http-echo v1/Service
[OK] http-echo v1/Service

Similar to kubeval, kube-score returns a non-zero exit code when there is a test that fails. CRITICAL. You can also enable similar processing for WARNING.

In addition, it is possible to check resources against different API versions (similar to kubeval). However, this information is 'hardcoded' in the kube-score itself: you cannot select a different version of Kubernetes. This limitation can become a big problem if you intend to upgrade your cluster or if you have multiple clusters with different versions of K8s.

Note that already have an issue with a proposal to implement this opportunity.

More information about kube-score can be found at the official website.

Kube-score tests are a great tool for implementing best practices, but what if you need to make changes to the test or add custom rules? Alas, this cannot be done.

Kube-score is not extensible: you cannot add policies to it or tweak them.

If you want to write custom tests to check against company policies, you can use one of the following four tools: config-lint, copper, conftest, or polaris.

3.Config-lint

Config-lint is a tool for validating YAML, JSON, Terraform, CSV configuration files and Kubernetes manifests.

You can install it using instructions on the project website.

The current release as of the time of writing the original article is 1.5.0.

Config-lint does not include built-in tests for validating Kubernetes manifests.

To conduct any tests, you must create the appropriate rules. They are written in YAML files called "rule sets" (rulesets), and have the following structure:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
   # список ΠΏΡ€Π°Π²ΠΈΠ»

(rule.yaml)

Let's study it more closely:

  • Field type specifies what type of configuration config-lint will use. For K8s manifests this is always Kubernetes.
  • In the files in addition to the files themselves, you can specify a directory.
  • Field rules is intended for setting user tests.

Let's say you want to make sure that images in a Deployment are always downloaded from a trusted repository like my-company.com/myapp:1.0. A config-lint rule that does this check would look like this:

- id: MY_DEPLOYMENT_IMAGE_TAG
  severity: FAILURE
  message: Deployment must use a valid image tag
  resource: Deployment
  assertions:
    - every:
        key: spec.template.spec.containers
        expressions:
          - key: image
            op: starts-with
            value: "my-company.com/"

(rule-trusted-repo.yaml)

The following attributes must be specified for each rule:

  • id β€” unique identifier of the rule;
  • severity - may be FAILURE, WARNING ΠΈ NON_COMPLIANT;
  • message - if the rule is violated, the contents of this line are displayed;
  • resource β€” the type of resource to which this rule applies;
  • assertions - a list of conditions that will be evaluated in relation to this resource.

In the rule above assertion called every checks that all containers in Deployment'e (key: spec.templates.spec.containers) use trusted images (i.e. those starting with my-company.com/).

The complete ruleset looks like this:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:

 - id: DEPLOYMENT_IMAGE_REPOSITORY # !!!
    severity: FAILURE
    message: Deployment must use a valid image repository
    resource: Deployment
    assertions:
      - every:
          key: spec.template.spec.containers
          expressions:
            - key: image
              op: starts-with
              value: "my-company.com/"

(ruleset.yaml)

To try out the test, let's save it as check_image_repo.yaml. Let's check the file base-valid.yaml:

$ config-lint -rules check_image_repo.yaml base-valid.yaml

[
  {
  "AssertionMessage": "Every expression fails: And expression fails: image does not start with my-company.com/",
  "Category": "",
  "CreatedAt": "2020-06-04T01:29:25Z",
  "Filename": "test-data/base-valid.yaml",
  "LineNumber": 0,
  "ResourceID": "http-echo",
  "ResourceType": "Deployment",
  "RuleID": "DEPLOYMENT_IMAGE_REPOSITORY",
  "RuleMessage": "Deployment must use a valid image repository",
  "Status": "FAILURE"
  }
]

The check failed. Now let's check the following manifest with the correct image repository:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
         image: my-company.com/http-echo:1.0 # !!!
         args: ["-text", "hello-world"]
         ports:
         - containerPort: 5678

(image-valid-mycompany.yaml)

We run the same test with the above manifest. No issues found:

$ config-lint -rules check_image_repo.yaml image-valid-mycompany.yaml
[]

Config-lint is a forward looking framework that allows you to create custom tests to validate Kubernetes YAML manifests using YAML DSL.

But what if more complex logic and tests are required? Aren't YAML capabilities too small for that? What if you could create tests in a full-fledged programming language?

4.Copper

Copper V2 is a framework for validating manifests using custom tests (similar to config-lint).

However, it differs from the latter in that it does not use YAML to describe tests. Instead, tests can be written in JavaScript. Copper provides a library with several basic toolsto help read information about Kubernetes objects and report errors.

The sequence of steps to install Copper can be found in official documentation.

2.0.1 is the latest release of this utility at the time of writing the original article.

Like config-lint, Copper has no built-in tests. Let's write one. Let him check that deployments use container images exclusively from trusted repositories like my-company.com.

Create a file check_image_repo.js with the following content:

$$.forEach(function($){
    if ($.kind === 'Deployment') {
        $.spec.template.spec.containers.forEach(function(container) {
            var image = new DockerImage(container.image);
            if (image.registry.lastIndexOf('my-company.com/') != 0) {
                errors.add_error('no_company_repo',"Image " + $.metadata.name + " is not from my-company.com repo", 1)
            }
        });
    }
});

Now to test our manifest base-valid.yaml, use the command copper validate:

$ copper validate --in=base-valid.yaml --validator=check_image_tag.js

Check no_company_repo failed with severity 1 due to Image http-echo is not from my-company.com repo
Validation failed

It is clear that more complex tests can be carried out using copper - for example, checking domain names in Ingress manifests or rejecting pods running in privileged mode.

Various utility functions are built into Copper:

  • DockerImage reads the specified input file and creates an object with the following attributes:
    • name - name of the image,
    • tag - image tag,
    • registry - image registry
    • registry_url - protocol (https://) and an image registry,
    • fqin β€” the full location of the image.
  • Function findByName helps to find a resource by a given type (kind) and name (name) from the input file.
  • Function findByLabels helps to find a resource by the specified type (kind) and labels (labels).

All available utility functions can be found here.

By default, it loads the entire YAML input file into a variable $$ and makes it available to scripts (a familiar technique for those with jQuery experience).

The main advantage of Copper is obvious: you do not need to learn a specialized language and you can use various JavaScript features to create your own tests, such as string interpolation, functions, etc.

It should also be noted that the current version of Copper works with the ES5 version of the JavaScript engine, not ES6.

Details are available at official website of the project.

However, if you're not a big fan of JavaScript and prefer a language specifically designed for querying and describing policies, then you should look into conftest.

5. Contest

Conftest is a framework for checking configuration data. Also suitable for testing/verifying Kubernetes manifests. Tests are described using a specialized query language Rego.

You can install conftest with instructionslisted on the project website.

At the time of writing the original article, the latest version available is 0.18.2.

Similar to config-lint and copper, conftest comes without any built-in tests. Let's try it and write our own policy. As in the previous examples, we will check if the container images are taken from a reliable source.

Create a directory conftest-checks, and in it - a file with the name check_image_registry.rego with the following content:

package main

deny[msg] {

  input.kind == "Deployment"
  image := input.spec.template.spec.containers[_].image
  not startswith(image, "my-company.com/")
  msg := sprintf("image '%v' doesn't come from my-company.com repository", [image])
}

Now let's test base-valid.yaml via conftest:

$ conftest test --policy ./conftest-checks base-valid.yaml

FAIL - base-valid.yaml - image 'hashicorp/http-echo' doesn't come from my-company.com repository
1 tests, 1 passed, 0 warnings, 1 failure

The test failed as expected because the images are from an untrusted source.

In the Rego file, we define a block deny. Its truth is regarded as a violation. If blocks deny multiple, conftest checks them independently, and the truth of any of the blocks is treated as a violation.

In addition to the default output, conftest supports JSON, TAP, and tabular format - a very useful feature if you need to embed reports into an existing CI pipeline. You can set the desired format using the flag --output.

To facilitate policy debugging, conftest has a flag --trace. It prints a trace of how conftest parses the specified policy files.

conftest policies can be published and shared in OCI (Open Container Initiative) registries as artifacts.

commands push ΠΈ pull allow you to publish an artifact or retrieve an existing artifact from a remote registry. Let's try to publish the policy we created to the local Docker registry with conftest push.

Start local Docker registry:

$ docker run -it --rm -p 5000:5000 registry

In another terminal, go to the directory you created earlier conftest-checks and run the following command:

$ conftest push 127.0.0.1:5000/amitsaha/opa-bundle-example:latest

If the command was successful, you will see a message like this:

2020/06/10 14:25:43 pushed bundle with digest: sha256:e9765f201364c1a8a182ca637bc88201db3417bacc091e7ef8211f6c2fd2609c

Now create a temporary directory and run the command in it conftest pull. It will download the package created by the previous command into it:

$ cd $(mktemp -d)
$ conftest pull 127.0.0.1:5000/amitsaha/opa-bundle-example:latest

A subdirectory will appear in the temporary directory policy, containing our policy file:

$ tree
.
└── policy
  └── check_image_registry.rego

Tests can be run directly from the repository:

$ conftest test --update 127.0.0.1:5000/amitsaha/opa-bundle-example:latest base-valid.yaml
..
FAIL - base-valid.yaml - image 'hashicorp/http-echo' doesn't come from my-company.com repository
2 tests, 1 passed, 0 warnings, 1 failure

Unfortunately, DockerHub is not yet supported. So consider yourself lucky if you use Azure Container Registry (ACR) or your own registry.

Artifact format is the same as Open Policy Agent packages (OPA), which allows you to use conftest to run tests from existing OPA packages.

You can learn more about policy sharing and other conftest features at official website of the project.

6. Polaris

The last tool that will be discussed in this article is Polaris. (His last year's announcement we already translated β€” approx. transl.)

Polaris can be installed in a cluster or used in command line mode. As you may have guessed, it allows you to statically parse Kubernetes manifests.

When running in command line mode, built-in tests are available covering areas such as security and best practices (similar to kube-score). In addition, you can create your own tests (as in config-lint, copper, and conftest).

In other words, Polaris combines the advantages of both categories of tools: with built-in and custom tests.

To install Polaris in command line mode, use instructions on the project website.

At the time of writing the original article, version 1.0.3 is available.

After the installation is complete, you can run polaris on the manifest base-valid.yaml with the following command:

$ polaris audit --audit-path base-valid.yaml

It will output a JSON string detailing the tests that were run and their results. The output will have the following structure:

{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "test-data/base-valid.yaml",
  "DisplayName": "test-data/base-valid.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 2,
    "Namespaces": 0,
    "Controllers": 2
  },
  "Results": [
    /* Π΄Π»ΠΈΠ½Π½Ρ‹ΠΉ список */
  ]
}

Full withdrawal available here.

Like kube-score, Polaris identifies issues where the manifest doesn't follow best practices:

  • Pod health checks are missing.
  • No tags specified for container images.
  • The container runs as root.
  • Request'y and limit'y for memory and CPU are not specified.

Each test, depending on its results, is assigned a degree of criticality: warning or hazard. To learn more about the available built-in tests, refer to documentation.

If details are not needed, you can specify a flag --format score. In this case, Polaris will output a number in the range from 1 to 100 βˆ’ score (i.e. evaluation):

$ polaris audit --audit-path test-data/base-valid.yaml --format score
68

The closer the score is to 100, the higher the degree of compliance. If you check the exit code of the command polaris audit, it turns out that it is equal to 0.

Make polaris audit You can exit with a non-zero code using two flags:

  • Flag --set-exit-code-below-score takes as an argument a threshold value in the range 1-100. In this case, the command will end with an exit code of 4 if the score is below the threshold. This is very handy when you have some threshold value (say 75) and you want to be alerted if the score drops below.
  • Flag --set-exit-code-on-danger will cause the command to exit with code 3 if one of the danger tests fails.

Now let's try to create a custom test that checks if the image is taken from a trusted repository. User tests are specified in YAML format, and the test itself is described using JSON Schema.

The following YAML snippet describes a new test called checkImageRepo:

checkImageRepo:
  successMessage: Image registry is valid
  failureMessage: Image registry is not valid
  category: Images
  target: Container
  schema:
    '$schema': http://json-schema.org/draft-07/schema
    type: object
    properties:
      image:
        type: string
        pattern: ^my-company.com/.+$

Let's take a closer look at it:

  • successMessage - this line will be displayed if the test completes successfully;
  • failureMessage - this message will be shown in case of failure;
  • category - indicates one of the categories: Images, Health Checks, Security, Networking ΠΈ Resources;
  • target--- defines what type of object (spec) the test is applied. Possible values: Container, Pod or Controller;
  • The test itself is specified in the object schema using JSON schema. In this test, the keyword pattern used to compare the image source with the desired one.

To run the above test, you need to create the following Polaris configuration:

checks:
  checkImageRepo: danger
customChecks:
  checkImageRepo:
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^my-company.com/.+$

(polaris-conf.yaml)

Let's parse the file:

  • In the checks tests and their level of criticality are prescribed. Since it is desirable to be warned when an image is taken from an untrusted source, we set the level here danger.
  • The test itself checkImageRepo then assigned to the object customChecks.

Save the file as custom_check.yaml. Now you can run polaris audit with a YAML manifest requiring validation.

Testing our manifest base-valid.yaml:

$ polaris audit --config custom_check.yaml --audit-path base-valid.yaml

Team polaris audit only ran the custom test given above and it failed.

If you fix the image to my-company.com/http-echo:1.0, Polaris will complete successfully. Manifesto with changes is already in repositoriesso you can check the previous command on the manifest image-valid-mycompany.yaml.

Now the question arises: how to run built-in tests together with custom ones? Easily! You just need to add the built-in test IDs to the config file. As a result, it will take the following form:

checks:
  cpuRequestsMissing: warning
  cpuLimitsMissing: warning
  # Other inbuilt checks..
  # ..
  # custom checks
  checkImageRepo: danger # !!!
customChecks:
  checkImageRepo:        # !!!
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^my-company.com/.+$

(config_with_custom_check.yaml)

An example of a complete configuration file is available here.

Check Manifest base-valid.yaml, using built-in and custom tests, you can use the command:

$ polaris audit --config config_with_custom_check.yaml --audit-path base-valid.yaml

Polaris complements the built-in tests with custom tests, thereby combining the best of both worlds.

On the other hand, the inability to use more powerful languages ​​such as Rego or JavaScript can be a limiting factor preventing you from creating more sophisticated tests.

More information about Polaris is available at project site.

Summary

While there are many tools for validating and evaluating Kubernetes YAML files, it is important to have a clear understanding of how tests will be designed and executed.

For example, the if you take Kubernetes manifests going through a pipeline, kubeval could be the first step in such a pipeline. It would make sure that the object definitions match the Kubernetes API schema.

Once such a review is completed, more sophisticated tests could be moved on, such as compliance with standard best practices and specific policies. This is where kube-score and Polaris would come in handy.

For those who have complex requirements and need to fine-tune their tests, copper, config-lint and conftest would be great..

Conftest and config-lint use YAML to define custom tests, while copper gives you access to a full programming language, making it a pretty attractive choice.

On the other hand, should you use one of these tools and, therefore, create all the tests manually, or prefer Polaris, and add only what is needed to it? There is no single answer to this question.

The table below contains a brief description of each tool:

Tool
Purpose
Disadvantages
User Tests

kubeval
Validates YAML manifests against a specific API schema version
Doesn't know how to work with CRD
No

kube score
Parses YAML manifests against best practices
You can't select your version of the Kubernetes API for resource validation
No

copper
Generic framework for creating custom JavaScript tests for YAML manifests
There are no built-in tests. Scant Documentation
Yes

config-lint
General framework for creating tests in a domain-specific language embedded in YAML. Supports various configuration formats (e.g. Terraform)
There are no ready-made tests. Built-in assertions and functions may not be enough
Yes

conftest
A framework for creating your own tests in Rego (a specialized query language). Allows policy sharing via OCI bundles
There are no built-in tests. You have to learn Rego. Docker Hub is not supported when publishing policies
Yes

Polaris
Parses YAML manifests against standard best practices. Allows you to create your own tests using JSON Schema
JSON Schema-based test capabilities may not be enough
Yes

Since these tools do not depend on access to the Kubernetes cluster, they are easy to install. They allow you to filter source files and provide quick feedback to pull request authors on projects.

PS from translator

Read also on our blog:

Source: habr.com

Add a comment