What's happened ? This is the so-called Service mesh, a technology that adds a layer of abstraction over the network. We intercept all or part of the traffic in the cluster and perform a certain set of operations with it. Which one? For example, we do smart routing, or we implement the circuit breaker approach, we can organize “canary deployment”, partially switching traffic to a new version of the service, or we can limit external interactions and control all trips from the cluster to the external network. It is possible to set policy rules to control trips between different microservices. Finally, we can get the entire network interaction map and make the unified collection of metrics completely transparent to applications.
You can read about the mechanism of work in . Istio is a really powerful tool that allows you to solve a lot of tasks and problems. In this article, I would like to answer the main questions that usually arise when getting started with Istio. This will help you deal with it faster.
Principle of operation
Istio consists of two main areas - the control plane and the data plane. The control plane contains the main components that ensure the correct operation of the rest. In the current version (1.0) the control plane has three main components: Pilot, Mixer, Citadel. We will not consider Citadel, it is needed to generate certificates to ensure mutual TLS between services. Let's take a closer look at the device and purpose of Pilot and Mixer.
Pilot is the main control component that distributes all the information about what we have in the cluster - services, their endpoints and routing rules (for example, rules for Canary deployment or circuit breaker rules).
Mixer is an optional control plane component that provides the ability to collect metrics, logs, and any information about network interaction. He also monitors compliance with Policy rules and compliance with rate limits.
The data plane is implemented using sidecar proxy containers. Powerful is used by default. . It can be replaced by another implementation, such as nginx (nginmesh).
In order for Istio to work completely transparent to applications, there is an automatic injection system. The latest implementation is suitable for Kubernetes 1.9+ versions (mutational admission webhook). For Kubernetes versions 1.7, 1.8 it is possible to use the Initializer.
Sidecar containers are connected to Pilot using the GRPC protocol, which allows you to optimize the push model for changes occurring in the cluster. GRPC has been used in Envoy since version 1.6, in Istio it has been used since version 0.8 and is a pilot-agent - a golang wrapper over envoy that configures launch options.
Pilot and Mixer are completely stateless components, all state is kept in memory. The configuration for them is set in the form of Kubernetes Custom Resources, which are stored in etcd.
Istio-agent gets the Pilot's address and opens a GRPC stream to it.
As I said, Istio implements all functionality completely transparent to applications. Let's see how. The algorithm is this:
- Deploying a new version of the service.
- Depending on the sidecar container injecting approach, the istio-init container and the istio-agent container (envoy) are added at the stage of applying the configuration, or they can already be manually inserted into the description of the Kubernetes Pod entity.
- The istio-init container is a script that applies the iptables rules to the pod. There are two options for configuring traffic to be wrapped in an istio-agent container: use iptables redirect rules, or . At the time of writing, the default approach is with redirect rules. In istio-init, it is possible to configure which traffic should be intercepted and sent to istio-agent. For example, in order to intercept all incoming and all outgoing traffic, you need to set the parameters
-iи-bin value*. You can specify specific ports to intercept. In order not to intercept a specific subnet, you can specify it using the flag-x. - After the init containers are executed, the main ones are launched, including the pilot-agent (envoy). It connects to the already deployed Pilot via GRPC and receives information about all existing services and routing policies in the cluster. According to the data received, he configures the clusters and assigns them directly to the endpoints of our applications in the Kubernetes cluster. It is also necessary to note an important point: envoy dynamically configures listeners (IP, port pairs) that it starts listening to. Therefore, when requests enter the pod, are redirected using the redirect iptables rules in the sidecar, envoy can already successfully process these connections and understand where to further proxy the traffic. Also at this stage, information is sent to the Mixer, which we will look at later, and tracing spans are sent.
As a result, we get a whole network of envoy proxy servers that we can configure from one point (Pilot). All inbound and outbound requests go through envoy. Moreover, only TCP traffic is intercepted. This means that Kubernetes service IP is resolved using kube-dns over UDP without changing. Then, after the resolve, the outgoing request is intercepted and processed by envoy, which already decides which endpoint the request should be sent to (or not sent, in the case of access policies or the circuit breaker of the algorithm).
We figured out Pilot, now we need to understand how Mixer works and why it is needed. You can read the official documentation for it .
Mixer in its current form consists of two components: istio-telemetry, istio-policy (before version 0.8 it was one istio-mixer component). Both of them are mixers, each of which is responsible for its own task. Istio telemetry receives information about who goes where and with what parameters from sidecar Report containers via GRPC. Istio-policy accepts Check requests to verify that Policy rules are satisfied. Poilicy checks are, of course, not carried out for every request, but are cached on the client (in the sidecar) for a certain time. Report checks are sent as batch requests. Let's see how to configure and what parameters should be sent a little later.
The Mixer is supposed to be a highly available component that ensures uninterrupted work on the assembly and processing of telemetry data. The system is obtained as a result as a multi-level buffer. Initially, data is buffered on the sidecar side of containers, then on the mixer side, and then sent to the so-called mixer backends. As a result, if any of the system components fails, the buffer grows and is flushed after the system is restored. Mixer backends are endpoints for sending telemetry data: statsd, newrelic, etc. You can write your own backend, it's quite simple, and we'll see how to do it.
To summarize, the scheme for working with istio-telemetry is as follows.
- Service 1 sends a request to service 2.
- When leaving service 1, the request is wrapped in its own sidecar.
- Sidecar envoy monitors how the request goes to service 2 and prepares the necessary information.
- Then sends it to istio-telemetry using a Report request.
- Istio-telemetry determines whether this Report should be sent to the backends, to which and what data should be sent.
- Istio-telemetry sends Report data to the backend if needed.
Now let's see how to deploy Istio in the system, consisting only of the main components (Pilot and sidecar envoy).
First, let's look at the main configuration (mesh) that Pilot reads:
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
namespace: istio-system
labels:
app: istio
service: istio
data:
mesh: |-
# пока что не включаем отправку tracing информации (pilot настроит envoy’и таким образом, что отправка не будет происходить)
enableTracing: false
# пока что не указываем mixer endpoint’ы, чтобы sidecar контейнеры не отправляли информацию туда
#mixerCheckServer: istio-policy.istio-system:15004
#mixerReportServer: istio-telemetry.istio-system:15004
# ставим временной промежуток, с которым будет envoy переспрашивать Pilot (это для старой версии envoy proxy)
rdsRefreshDelay: 5s
# default конфигурация для envoy sidecar
defaultConfig:
# аналогично как rdsRefreshDelay
discoveryRefreshDelay: 5s
# оставляем по умолчанию (путь к конфигурации и бинарю envoy)
configPath: "/etc/istio/proxy"
binaryPath: "/usr/local/bin/envoy"
# дефолтное имя запущенного sidecar контейнера (используется, например, в именах сервиса при отправке tracing span’ов)
serviceCluster: istio-proxy
# время, которое будет ждать envoy до того, как он принудительно завершит все установленные соединения
drainDuration: 45s
parentShutdownDuration: 1m0s
# по умолчанию используются REDIRECT правила iptables. Можно изменить на TPROXY.
#interceptionMode: REDIRECT
# Порт, на котором будет запущена admin панель каждого sidecar контейнера (envoy)
proxyAdminPort: 15000
# адрес, по которому будут отправляться trace’ы по zipkin протоколу (в начале мы отключили саму отправку, поэтому это поле сейчас не будет использоваться)
zipkinAddress: tracing-collector.tracing:9411
# statsd адрес для отправки метрик envoy контейнеров (отключаем)
# statsdUdpAddress: aggregator:8126
# выключаем поддержку опции Mutual TLS
controlPlaneAuthPolicy: NONE
# адрес, на котором будет слушать istio-pilot для того, чтобы сообщать информацию о service discovery всем sidecar контейнерам
discoveryAddress: istio-pilot.istio-system:15007
All the main control components (control plane) will be located in the namespace istio-system in Kubernetes.
At a minimum, we only need to deploy Pilot. For this we use
And we will manually configure the injecting sidecar of the container.
Init container:
initContainers:
- name: istio-init
args:
- -p
- "15001"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -b
- '*'
- -d
- ""
image: istio/proxy_init:1.0.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 128Mi
securityContext:
capabilities:
add:
- NET_ADMIN
And sidecar:
name: istio-proxy
args:
- "bash"
- "-c"
- |
exec /usr/local/bin/pilot-agent proxy sidecar
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
service-name
--drainDuration
45s
--parentShutdownDuration
1m0s
--discoveryAddress
istio-pilot.istio-system:15007
--discoveryRefreshDelay
1s
--connectTimeout
10s
--proxyAdminPort
"15000"
--controlPlaneAuthPolicy
NONE
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
image: istio/proxyv2:1.0.0
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 2048Mi
securityContext:
privileged: false
readOnlyRootFilesystem: true
runAsUser: 1337
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
In order for everything to start successfully, you need to create a ServiceAccount, ClusterRole, ClusterRoleBinding, CRD for Pilot, the descriptions of which can be found .
As a result, the service into which we inject sidecar with envoy should start successfully, receive all discovery from the pilot and process requests.
It is important to understand that all control plane components are stateless applications and can be horizontally scaled without problems. All data is stored in etcd in the form of custom descriptions of Kubernetes resources.
Also, Istio (still experimental) has the ability to run outside the cluster and the ability to watch and fumble service discovery between several Kubernetes clusters. You can read more about this .
For a multi-cluster installation, be aware of the following limitations:
- Pod CIDR and Service CIDR must be unique across all clusters and must not overlap.
- All CIDR Pods must be accessible from any CIDR Pods between clusters.
- All Kubernetes API servers must be accessible to each other.
This is the initial information to help you get started with Istio. However, there are still many pitfalls. For example, features of routing external traffic (outside the cluster), approaches to debugging sidecars, profiling, setting up a mixer and writing a custom mixer backend, setting up a tracing mechanism and its operation using envoy.
All this we will consider in the following publications. Ask your questions, I will try to cover them.
Source: habr.com