Istio and Kubernetes in production. Part 2. Tracing

Previously article We reviewed the basic components of the Service Mesh Istio, familiarized ourselves with the system, and answered the main questions that usually arise when starting to work with Istio. In this part, we will look at how to organize the collection of tracing information across the network.

The first thing that comes to mind for many developers and system administrators when they hear the words Service Mesh is tracing. Indeed, we add a special proxy server to each node in the network through which all TCP traffic passes. It seems that we can now easily send information about all network interactions. Unfortunately, in reality, there are many nuances that need to be considered. Let’s take a look at them.

Myth number one: we can get network traversal data for free.

In reality, for a relatively low cost, we can only obtain connected nodes of our system represented by arrows and the rate of data that flows between services (essentially just the number of bytes per unit of time). However, in most cases, our services communicate using some application-level protocol, such as HTTP, gRPC, Redis, and so on. And of course, we want to see tracing information specifically for these protocols; we want to see the rate of requests rather than the rate of data. We want to understand the latency of requests according to our protocol. Finally, we want to see the full path that a request takes from entry into our system to the response received by the user. This task is not as straightforward as it may seem.

To begin with, let's look at how sending tracing spans works from an architectural perspective in Istio. As we recall from the first part, Istio has a separate component for collecting telemetry called Mixer. However, in the current version 1.0.*, sending is done directly from the proxy servers, specifically from the envoy proxy. Envoy proxy supports sending tracing spans using the zipkin protocol out of the box. Other protocols can be integrated, but only through a plugin. With Istio, we immediately get a pre-configured envoy proxy that only supports the zipkin protocol. If we want to use, for example, the Jaeger protocol and send tracing spans over UDP, we will need to build our own istio-proxy image. There is support for custom plugins for istio-proxy, but it is still in alpha. Therefore, if we want to avoid a lot of custom settings, the range of technologies available for storing and receiving tracing spans decreases. Currently, the main systems we can use are either Zipkin itself or Jaeger, but sending everything there using the zipkin-compatible protocol (which is significantly less efficient). The zipkin protocol requires sending all tracing information to collectors over HTTP, which is fairly resource-intensive.

As I mentioned, we want to trace application-level protocols. This means that the proxy servers next to each service must understand the specific interactions happening at the moment. By default, Istio configures all ports as plain TCP, which means no traces will be sent. To enable trace sending, we must first enable this option in the main mesh config and, crucially, name all ports in Kubernetes service entities according to the protocol used in the service. For example, like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 80
    targetPort: 80
    name: http
  selector:
    app: nginx

Composite names can also be used, for example, http-magic (Istio will detect http and recognize this port as an http endpoint). The format is proto-extra.

To avoid patching a large number of configurations to define the protocol, a dirty workaround can be employed: patch the Pilot component at the moment it is executing the protocol determination logic. to perform the logic of protocol determination.In the end, of course, this logic will need to be changed to standard and we will need to switch to a naming convention for all ports.

To truly understand if the protocol is defined correctly, you need to access any of the sidecar containers with the envoy proxy and make a request to the envoy admin interface on the /config_dump port. In the resulting configuration, you need to look for the operation field for the required service. It is used in Istio as an identifier for the destination of the request. To customize the value of this parameter in Istio (which we will later see in our tracing system), it is necessary to specify the serviceCluster flag at the startup stage of the sidecar container. For example, it can be calculated from variables obtained from the downward API of Kubernetes:

--serviceCluster ${POD_NAMESPACE}.$(echo ${POD_NAME} | sed -e 's/-[a-z0-9]*-[a-z0-9]*$//g')

A good example to understand how tracing works in envoy is here.

The endpoint for sending tracing spans must also be specified in the startup flags of the envoy proxy, for example: --zipkinAddress tracing-collector.tracing:9411

Myth number two: we can get complete traces of requests passing through the system out of the box for a low cost.

Unfortunately, this is not the case. The complexity of implementation depends on how you have already implemented service interaction. Why is that?

The fact is that for istio-proxy to understand the correspondence of incoming requests to the service with outgoing requests from that same service, it is not enough just to intercept all traffic. You need to have some identifier of the connection. In HTTP, envoy proxy uses special headers, which allow envoy to understand which specific request to the service generates specific requests to other services. The list of such headers includes:

• x-request-id,
• x-b3-traceid,
• x-b3-spanid,
• x-b3-parentspanid,
• x-b3-sampled,
• x-b3-flags,
• x-ot-span-context.

If you have a unified point, for example, a basic client in which you can add such logic, then everything is great; you just need to wait for this library to be updated in all clients. But if you have a very heterogeneous system without unification in the approach from services to services over the network, then this will likely be a big problem. Without adding such logic, all tracing information will be just 'single-level.' This means we will get all inter-service interactions, but they will not be linked into single chains of passage through the network.

Conclusion

Istio provides a convenient tool for collecting tracing information across the network, but it is important to understand that implementing it will require adapting your system and considering the specifics of Istio's implementation. Ultimately, two main issues need to be addressed: defining the application-level protocol (which must be supported by the envoy proxy) and configuring the transmission of information regarding the correlation of requests from service to service (using headers, in the case of the HTTP protocol). Once these questions are resolved, we gain a powerful tool that allows for seamless information gathering from the network, even in highly heterogeneous systems built with many different languages and frameworks.

In the next article about Service Mesh, we will look at one of the biggest issues with Istio – the high memory consumption of each sidecar proxy container and discuss how to tackle it.

Source: habr.com