Loki - log collection using the Prometheus approach.

Hello, Habr users! In anticipation of the new enrollment for the course, DevOps Practices and Tools we have prepared a translation of some interesting material for you.

This article is a brief introduction to Loki. The Loki project is supported by Grafana and is aimed at the centralized collection of logs (from servers or containers).

The main source of inspiration for Loki was Prometheus the idea of applying its approaches to log management:

• using labels to store data,
• consuming minimal resources.

We will return to the principles of Prometheus and provide some examples of its use in the context of Kubernetes.

A few words about Prometheus

To fully understand how Loki works, it’s important to take a step back and recall a bit about Prometheus.

One of the distinguishing features of Prometheus is the extraction of metrics from scrape endpoints (via exporters) and saving them in a TSDB (Time Series Database) with additional metadata in the form of labels.

Why is this needed?

Recently, Prometheus has become the de facto standard in the world of containers and Kubernetes: it’s very easy to install, and there’s initially an endpoint for Prometheus present in a Kubernetes cluster. Prometheus can also extract metrics from applications deployed in containers while retaining specific labels. Therefore, monitoring applications is very easy to implement.

Unfortunately, there is still no off-the-shelf solution for log management, and you have to find a solution that works for you:

• a managed cloud service for log centralization (AWS, Azure, or Google),
• monitoring service 'monitoring as a service' (e.g., Datadog),
• building your own log collection service.

For the third option, I traditionally used Elasticsearch, even though I haven’t always been satisfied with it (especially its heaviness and complexity of setup).

Loki was designed to simplify implementation according to the following principles:

• be simple to start,
• consume low resources,
• operate independently without any special maintenance,
• serve as an addition to Prometheus to assist in bug investigation.

However, this simplicity comes at the expense of certain trade-offs. One of these is not indexing content. This makes text search less effective or rich and does not allow for tracking statistics on the text content. But since Loki aims to be the equivalent of grep and a complement to Prometheus, this is not a drawback.

Incident Investigation

To better understand why Loki does not require indexing, let’s revisit the incident investigation method used by the Loki developers:

1 Alert → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed Tracing → 6 Fix!
(1 Alert → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed Tracing → 6 Fix!)

The idea is that we receive some kind of alert (Slack Notification, SMS, etc.), and then:

• we check Grafana dashboards
• we look at service metrics (for example, in Prometheus)
• we examine log entries (for instance, in Elasticsearch)
• we may look at distributed traces (Jaeger, Zipkin, etc.)
• and finally, we fix the original problem.

Here, in the case of the Grafana + Prometheus + Elasticsearch + Zipkin stack, we would need to use four different tools. To reduce time, it would be beneficial to have the ability to perform all these steps with a single tool: Grafana. It is worth noting that this approach to investigation has been implemented in Grafana since version 6. This makes it possible to access Prometheus data directly from Grafana.

The Explorer screen is split between Prometheus and Loki

On this screen, you can view logs in Loki that are related to Prometheus metrics, using the concept of screen splitting. Starting from version 6.5, Grafana allows processing the trace ID in Loki log entries for linking to your favorite distributed tracing tools (Jaeger).

Local Testing of Loki

The simplest way to test Loki locally is to use docker-compose. The docker-compose file is located in the Loki repository. You can get the repository using the following command git:

$ git clone https://github.com/grafana/loki.git

Then you need to navigate to the production directory:

$ cd production

After that, you can get the latest version of the Docker images:

$ docker-compose pull

Finally, the Loki stack can be started with the following command:

$ docker-compose up

Loki Architecture

Here is a small diagram with the architecture of Loki:

Principles of Loki Architecture

The web client launches applications on the server, Promtail collects logs and sends them to Loki, the web client also sends metadata to Loki. Loki aggregates everything and passes it to Grafana.
Loki is running. To view the available components, execute the following command:

$ docker ps

In the case of a freshly installed Docker, the command should return the following result:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

We see the following components:

• Promtail: an agent responsible for log centralization
• Grafana: a well-known tool for dashboards
• Loki: a data centralization daemon

In a traditional infrastructure (for example, based on virtual machines), an agent Promtail must be deployed on each machine. Grafana and Loki can be installed on the same machine.

Deployment in Kubernetes

Installing Loki components in Kubernetes will consist of the following:

• a daemonSet for deploying the Promtail agent on each of the machines in the server cluster
• a deployment (Deployment) of Loki
• and finally — deploying Grafana.

Fortunately, Loki is available as a Helm package, which simplifies its deployment.

Installation via Helm

Helm should already be installed on your system. It can be downloaded from the project GitHub repository. It installs by unpacking the archive corresponding to your architecture and adding helm to $PATH.

Note: version 3.0.0 of Helm was recently released. Since there were many changes, it is recommended for the reader to wait a bit before starting to use it.

Adding a source for Helm

The first step will be adding the ‘loki’ repository using the following command:

$ helm add loki https://grafana.github.io/loki/charts

After that, you can search for packages named ‘loki’:

$ helm search loki

Result:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

These packages have the following functions:

• the package loki/loki corresponds only to the Loki server
• the package loki/fluent-bit allows you to deploy a DaemonSet using fluent-bit for log collection instead of Promtail
• the package loki/promtail contains the log file collection agent
• the package loki/loki-stackallows you to deploy Loki together with Promtail at once.

Installing Loki

To deploy Loki in Kubernetes, execute the following command in the ‘monitoring’ namespace:

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

To save to disk, add the parameter --set loki.persistence.enabled=true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

Note: If you want to deploy Grafana simultaneously, add the parameter --set grafana.enabled=true

When you run this command, you should see the following output:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been successfully deployed to your cluster. You can now add Loki as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

By checking the status of the pods in the 'monitoring' namespace, we will see that everything is deployed:

$ kubectl -n monitoring get pods -l release=loki

Result:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

All pods are running. Now it's time to make some tests!

Connecting to Grafana

To connect to Grafana under Kubernetes, you need to open a tunnel to its pod. Below is the command to open port 3000 for the Grafana pod:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

Another important point is the need to retrieve the Grafana admin password. The password is stored in the secret loki-grafana in the field .data.admin-user in base64 format.

To retrieve it, you need to run the following command:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

Use this password along with the default admin account (admin).

Defining the Loki Data Source in Grafana

First, make sure that the Loki data source is created (Configuration / Datasource).
Here is an example:

Example configuration for the Loki data source

By clicking 'Test', you can check the connection to Loki.

Making requests to Loki

Now go to Grafana's 'Explore' section. When receiving logs from containers, Loki adds metadata from Kubernetes. This allows viewing logs from a specific container.

For example, to select logs from the promtail container, you can use the following query: {container_name="promtail"}.
Also, don't forget to select the Loki data source.

This request will return container activity in the following form:

Query result in Grafana

Adding to the dashboard

Starting with Grafana 6.4, you can place log information directly on the dashboard. After that, users can quickly switch between the number of requests on their site and application traces.

Below is an example dashboard that implements this interaction:

Sample dashboard with Prometheus metrics and Loki logs

The Future of Loki

I started using Loki back in May/June with version 0.1. Today, version 1 has already been released, along with 1.1 and 1.2.

It must be acknowledged that version 0.1 was not stable enough. However, 0.3 already showed real signs of maturity, and the subsequent versions (0.4, and then 1.0) only reinforced that impression.

After 1.0.0, there can no longer be excuses for not using this wonderful tool.

Further improvements should focus not on Loki itself, but rather on its integration with the excellent Grafana. In fact, Grafana 6.4 already introduced good integration with dashboards.

Grafana 6.5, which was recently released, enhances this integration even further by automatically recognizing the contents of logs in JSON format.

Below in the video is a brief example of this mechanism:

Using Loki strings displayed in Grafana

It becomes possible to use one of the JSON fields, for example, for:

• linking to external tools
• filtering log contents

For example, you can click on traceId to go to Zipkin or Jaeger.

Traditionally, we invite your comments and welcome you to an open webinar, where we'll discuss how the DevOps industry evolved throughout 2019 and explore possible paths for development in 2020.

Source: habr.com