🥇Liveness probes in Kubernetes can be dangerous

Note: translation.Henning Jacobs, lead engineer at Zalando, has often noticed that users of Kubernetes struggle to understand the purpose of liveness (and readiness) probes and their correct application. Therefore, he compiled his thoughts into this concise note, which will eventually become part of the K8s documentation.

Health checks, known in Kubernetes as liveness probes (i.e., literally, "viability tests" — ed.), can be quite dangerous. I recommend avoiding them wherever possible; exceptions are only when they are truly necessary and you fully understand the specifics and consequences of their use. This publication will discuss both liveness and readiness checks, as well as explain when is located and when not to apply them.

My colleague Sandor recently shared on Twitter the most common mistakes he encounters, including those related to the use of readiness/liveness probes:

A misconfigured livenessProbe can exacerbate high-load situations (cascading failures + potentially long startup of the container/application) and lead to other negative consequences, such as dependency crashes (see also my recent article on limiting the number of requests in conjunction with K3s+ACME). Even worse is when a liveness probe is combined with a dependency health check, with an external database acting as the dependency: a single database failure will restart all your containers.!

The general message "Do not use liveness probes" is of little help in this case, so let’s examine the intended purposes of readiness and liveness checks.

Note: Most of the test provided below was originally included in the internal documentation for Zalando developers.

Readiness and Liveness Checks

Kubernetes provides two important mechanisms, called liveness probes and readiness probes. They periodically perform some action — such as sending an HTTP request, opening a TCP connection, or executing a command in the container — to confirm that the application is functioning properly.

Kubernetes uses readiness probes, to understand when a container is ready to accept traffic. A pod is considered ready when all its containers are ready. One application of this mechanism is to control which pods are used as backends for Kubernetes services (especially Ingress).

Liveness probes help Kubernetes understand when it's time to restart a container. For example, such a check can catch a deadlock when an application gets 'stuck' in one place. Restarting the container in such a state helps move the application off the dead point, despite errors, but it can also lead to cascading failures (see below).

If you try to deploy an application update that fails liveness/readiness checks, its rollout will be stalled as Kubernetes waits for the status Ready of all pods.

Example

Here’s an example of a readiness probe that checks the path /health via HTTP with default settings (interval: 10 seconds, timeout: 1 second, success threshold: 1, failure threshold: 3):

# часть общего описания deployment'а/стека
podTemplate:
  spec:
    containers:
    - name: my-container
      # ...
      readinessProbe:
        httpGet:
          path: /health
          port: 8080

Recommendations

For microservices with an HTTP endpoint (REST, etc.) always define a readiness probe, which checks if the application (pod) is ready to accept traffic.
Ensure that the readiness probe covers the readiness of the actual web server port:
- using ports for administrative needs called 'admin' or 'management' (e.g., 9090), for readinessProbe, ensure that the endpoint returns OK only if the main HTTP port (like 8080) is ready to accept traffic*;
  * I know of at least one case in Zalando where this did not happen, that is, readinessProbe it checked the 'management' port, but the server itself did not start due to cache loading issues.
- hanging the readiness probe on a separate port may result in the load on the main port not being reflected in the health check (i.e., the thread pool on the server is full, yet the health check still shows that everything is OK).
Ensure that the readiness probe includes database initialization/migration;
- the simplest way to achieve this is to call the HTTP server only after initialization is complete (e.g., database migration with Flyway etc.); that is, instead of changing the health check status, simply do not start the web server until the database migration is finished*.
  * You can also run database migrations from init containers outside the pod. I still favor self-contained applications, meaning those where the application container knows how to prepare the database without external coordination.
Use httpGet for readiness checks through typical health check endpoints (for example, /health).
Familiarize yourself with the default check parameters. (interval: 10s, timeout: 1s, successThreshold: 1, failureThreshold: 3):
- the default parameters mean that the pod will become not-ready in about 30 seconds (3 failed health checks).
Use a separate port for ‘admin’ or ‘management’ if the tech stack (for example, Java/Spring) allows it, to separate ‘health’ management and metrics from regular traffic:
- but don’t forget about point 2.
If necessary, the readiness probe can be used for warming/loading the cache and return a 503 status code until the container is ‘warmed up’:
- I also recommend checking out the new probe startupProbe, introduced in version 1.16, (we wrote about it in Russian. here - translator's note).

Warnings

Do not rely on external dependencies (such as data stores) when conducting readiness/liveness tests — this can lead to cascading failures:
- for example, consider a stateful REST service with 10 pods depending on a single Postgres database: when the check relies on a functioning connection to the database, all 10 pods can crash if there’s a delay in the network/database — usually, it all ends worse than it could;
- note that Spring Data checks the database connection by default*;
  * This is the default behavior of Spring Data Redis (at least it was the last time I checked), which led to a ‘catastrophic’ failure: when Redis was briefly unavailable, all pods went down.
- ‘external’ in this sense can also mean other pods of the same application, so ideally, the check should not rely on the state of other pods in the same cluster to prevent cascading failures:
  - results can vary for applications with distributed state (for example, in-memory caching in pods).
Do not use the liveness probe for pods (exceptions are cases when they are genuinely needed and you fully understand their specifics and implications):
- A liveness probe can help recover 'stuck' containers, but since you have full control over your application, issues like 'stuck' processes and deadlocks ideally should not occur: the better alternative is to intentionally crash the application and return it to a previous stable state;
- A failed liveness probe will lead to the container being restarted, potentially exacerbating the consequences of loading errors: restarting the container will result in downtime (at least for the time it takes to launch the application, say, for over 30 seconds), causing new errors, increasing the load on other containers, and raising the likelihood of their failure, etc.;
- Liveness checks combined with an external dependency are the worst possible combination, threatening cascading failures: a minor delay on the database side will result in the restart of all your containers!
Liveness and readiness probe parameters should be different:
- you can use a liveness probe with the same health check but a higher trigger threshold (failureThreshold), for example, assign a status not-ready after 3 attempts and consider the liveness probe failed after 10 attempts;
Do not use exec checks, as they have known issues that lead to zombie processes:
- details: see the presentation by Datadog experts.

Summary

Use readiness probes to determine when a pod is ready to accept traffic.
Use liveness probes only when absolutely necessary.
Incorrect use of readiness/liveness probes can lead to reduced availability and cascading failures.

Additional resources on the topic

Kubernetes docs: Configure Liveness and Readiness Probes;
Kubernetes Liveness and Readiness Probes Revisited: How to Avoid Shooting Yourself in the Other Foot;
NRE Labs Outage Post-Mortem (also discusses livenessProbe).

Update #1 from 2019-09-29

About init-containers for database migration: a footnote has been added.

EJ reminded me about PDB: one of the issues with liveness checks is the lack of coordination between pods. In Kubernetes, there are Pod Disruption Budgets (PDB) to limit the number of concurrent failures that an application can experience; however, checks do not account for PDB. Ideally, we could instruct K8s: 'Restart one pod if its check fails, but do not restart them all to avoid making it worse.'

Bryan put it very well: 'Use liveness probing when you are certain that the best course of action is to 'kill' the application' (again, don't get carried away).

Update No. 2 from 2019-09-29

Regarding reading the documentation before use: I created a corresponding request (feature request) to supplement the documentation about liveness probes.