{"id":38965,"date":"2019-10-31T22:27:03","date_gmt":"2019-10-31T19:27:03","guid":{"rendered":"https:\/\/prohoster.info\/blog\/liveness-probes-v-kubernetes-mogut-byt-opasny\/"},"modified":"2019-10-31T22:27:03","modified_gmt":"2019-10-31T19:27:03","slug":"liveness-probes-v-kubernetes-mogut-byt-opasny","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/liveness-probes-v-kubernetes-mogut-byt-opasny","title":{"rendered":"Liveness probes in Kubernetes can be dangerous","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Note: translation.<\/b>Henning Jacobs, lead engineer at Zalando, has often noticed that users of Kubernetes struggle to understand the purpose of liveness (and readiness) probes and their correct application. Therefore, he compiled his thoughts into this concise note, which will eventually become part of the K8s documentation.<\/i><\/p>\n

\"Liveness
\n
\nHealth checks, known in Kubernetes as liveness probes<\/i> (i.e., literally, \"viability tests\" \u2014 ed.)<\/i>, can be quite dangerous. I recommend avoiding them wherever possible; exceptions are only when they are truly necessary and you fully understand the specifics and consequences of their use. This publication will discuss both liveness and readiness checks, as well as explain when is located<\/i> and when not to apply them.<\/a><\/noindex><\/p>\n

My colleague Sandor recently shared on Twitter the most common mistakes he encounters, including those related to the use of readiness\/liveness probes:<\/p>\n

\"Liveness<\/a><\/noindex><\/p>\n

A misconfigured livenessProbe<\/code> can exacerbate high-load situations (cascading failures + potentially long startup of the container\/application) and lead to other negative consequences, such as dependency crashes (see also my recent article<\/a><\/noindex> on limiting the number of requests in conjunction with K3s+ACME)<\/i>. Even worse is when the liveness probe is combined with a health check that relies on an external database: a single database failure will restart all your containers.<\/b>!<\/p>\n

The general message \"Do not use liveness probes\"<\/i> is of little help in this case, so let\u2019s examine the intended purposes of readiness and liveness checks. <\/p>\n

Note: Most of the test provided below was originally included in the internal documentation for Zalando developers.<\/i><\/p>\n

Readiness and Liveness Checks<\/h2>\n

\nKubernetes provides two important mechanisms, called liveness probes and readiness probes<\/a><\/noindex>. They periodically perform some action \u2014 such as sending an HTTP request, opening a TCP connection, or executing a command in the container \u2014 to confirm that the application is functioning properly.<\/p>\n

Kubernetes uses readiness probes<\/b>, to understand when the container is ready to accept traffic. A pod is considered ready when all of its containers are ready. One application of this mechanism is to monitor which pods are used as backends for Kubernetes services (particularly Ingress).<\/p>\n

Liveness probes<\/b> help Kubernetes understand when it's time to restart a container. For example, such a check can catch a deadlock when an application gets 'stuck' in one place. Restarting the container in such a state helps move the application off the dead point, despite errors, but it can also lead to cascading failures (see below).<\/p>\n

If you try to deploy an application update that fails liveness\/readiness checks, its rollout will be stalled as Kubernetes waits for the status Ready<\/code> from all pods.<\/p>\n

Example<\/h2>\n

\nHere\u2019s an example of a readiness probe that checks the path \/health<\/code> via HTTP with default settings (interval<\/i>: 10 seconds, timeout<\/i>: 1 second, success threshold<\/i>: 1, failure threshold<\/i>: 3):<\/p>\n

# \u0447\u0430\u0441\u0442\u044c \u043e\u0431\u0449\u0435\u0433\u043e \u043e\u043f\u0438\u0441\u0430\u043d\u0438\u044f deployment'\u0430\/\u0441\u0442\u0435\u043a\u0430\npodTemplate:\n  spec:\n    containers:\n    - name: my-container\n      # ...\n      readinessProbe:\n        httpGet:\n          path: \/health\n          port: 8080<\/code><\/pre>\n
<\/p>\n
Recommendations<\/h2>\n
<\/p>\n
    \n
  1.  For microservices with HTTP endpoints (REST, etc.), always define a readiness probe<\/b>, which checks if the application (pod) is ready to accept traffic.<\/li>\n
  2.  Ensure that the readiness probe covers the readiness of the actual web server port<\/b>:\n