Monitoring has become a crucial component of growing cloud solutions with the increasing complexity of distributed systems. It is essential for understanding their behavior. Scalable tools are needed to collect data from all services and provide specialists with a unified interface for performance analysis, error demonstration, availability, and logging.
These tools must also be efficient and high-performing. In this article, we will review two popular technology stacks: EFK (Elasticsearch) and PLG (Loki), and explore their architectures and differences.
EFK Stack
You may have heard of the widely used ELK or EFK. The stack consists of several separate components: Elasticsearch (object storage), Logstash or FluentD (log collection and aggregation), and Kibana for visualization.
A typical workflow looks like this:
Elasticsearch β a distributed object store with real-time search and analytics. An excellent solution for semi-structured data, such as logs. Information is stored as JSON documents, indexed in real-time, and distributed across the cluster nodes. An inverted index is used, containing all unique words and the associated documents for full-text search, which is based on the Apache Lucene search engine.
FluentD β is a data collector that performs data unification during collection and consumption. It strives to organize data in JSON as much as possible. Its architecture is extensible, with more than , available for various purposes.
Kibana β a visualization tool for Elasticsearch with various additional features such as time series analysis, graphs, machine learning, and more.
Elasticsearch Architecture
Data in the Elasticsearch cluster is stored spread across all its nodes. The cluster consists of several nodes to enhance availability and resilience. Any node can perform all the cluster roles, but in large scalable deployments, nodes are typically assigned separate tasks.
Types of cluster nodes:
- master node β manages the cluster, requiring a minimum of three, with one always active;
- data node β stores indexed data and performs various tasks with it;
- ingest node β organizes pipelines for data transformation before indexing;
- coordinating node β routes requests, reduces search processing phase, coordinates bulk indexing;
- alerting node β initiates alerting tasks;
- machine learning node β processes machine learning tasks.
The diagram below shows how data is stored and replicated across nodes to achieve higher data availability.
The data of each replica is stored in an inverted index; the diagram below illustrates how this works:
Installation
You can see the details , I will use helm chart:
$ helm install efk-stack stable/elastic-stack --set logstash.enabled=false --set fluentd.enabled=true --set fluentd-elasticsPLG stack
Don't be surprised if you can't find this acronym, as it is better known as Grafana Loki. In any case, this stack is gaining popularity as it applies refined technical solutions. You may already have heard of Grafana, a popular visualization tool. Its creators, inspired by Prometheus, developed Loki, a horizontally scalable high-performance log aggregation system. Loki indexes only metadata, not the logs themselves, which has made it simple to operate and cost-effective.
Promtail β an agent for sending logs from the operating system to the Loki cluster. Grafana β a visualization tool based on data from Loki.
Loki is built on the same principles as Prometheus, so it is well-suited for storing and analyzing Kubernetes logs.
Loki Architecture
Loki can run both as a single process and as multiple processes, providing horizontal scalability.
It can also operate as a monolithic application or as a microservice. Running as a single process may be useful for local development or small-scale monitoring. For industrial deployment and scalable workloads, the microservice version is recommended. The paths for writing and reading data are separate, allowing for fine-tuning and scaling as needed.
Let's look at the architecture of the log collection system without going into detail:
And here is a description (microservices architecture):
Components:
Promtail β an agent that is installed on nodes (as a set of services), it collects logs from tasks and queries the Kubernetes API for metadata to label the logs. Then it sends the logs to the main Loki service. The same labeling rules are supported for matching the metadata as in Prometheus.
Distributor β a distribution service that acts as a buffer. To handle millions of records, it packages incoming data, compressing them in blocks as they arrive. Several data receivers operate simultaneously, but the logs belonging to a single stream of incoming data must reside in only one of them for all its blocks. This is organized as a ring of receivers and sequential hashing. For fault tolerance and redundancy, it is done n times (3 if not configured).
Ingester β a receiver service. Data blocks arrive compressed with added logs. Once a block reaches a sufficient size, it is flushed to the database. Metadata goes into the index, while the log block data ends up in Chunks (usually in object storage). After flushing, the receiver creates a new block to which new records will be added.
Index β databases such as DynamoDB, Cassandra, Google BigTable, and others.
Chunks β compressed log blocks, usually stored in object storage, such as S3.
Querier β the read path that does all the heavy lifting. It looks at the time range and labels, then checks the index for matches. Next, it reads data blocks and filters them to get the results.
Now, let's see everything in action.
Installation
To install on Kubernetes, it's easiest to use helm. We assume you've already installed and configured it ( translator's note)
Add the repository and install the stack.
$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo update
$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=falseBelow is an example dashboard that shows data from Prometheus for Etcd metrics and Loki for Etcd pod logs.
Now, let's discuss the architecture of both systems and compare their capabilities to each other.
Comparison
Query language
Elasticsearch utilizes Query DSL and the Lucene query language, providing capabilities for full-text search. It is a well-established powerful search engine with extensive support for operators. It allows searching within context and sorting by relevance.
On the other side of the ring is LogQL, used in Loki, the successor to PromQL (Prometheus query language). It employs log labels for filtering and extracting log data. It also supports some operators and arithmetic, as described , but in terms of capabilities, it falls short of Elastic language.
Since queries in Loki are tied to labels, they can easily be related to metrics, making it simpler to organize operational monitoring.
Scalability
Both stacks are horizontally scalable, but with Loki, it's easier, as it has separated read and write paths for data and a microservices architecture. Loki can be configured to meet your specific needs and can handle very large volumes of log data.
Multi-tenancy
Cluster multi-tenancy is a common theme for reducing OPEX; both stacks provide multi-tenancy. For Elasticsearch, there are several for client separation: a separate index for each client, client-based routing, unique client fields, and search filters. In Loki, it is implemented through the HTTP header X-Scope-OrgID.
Cost
Loki is very cost-effective because it does not index data, only metadata. This results in and memory (cache), as object storage is cheaper than block storage, which is used in Elasticsearch clusters.
Conclusion
The EFK stack can be used for various purposes, providing maximum flexibility and a multifunctional Kibana interface for analytics, visualization, and queries. It can be further enhanced with machine learning capabilities.
The Loki stack is beneficial in the Kubernetes ecosystem due to its metadata discovery mechanism. It is easy to correlate data for time-series monitoring in Grafana and logs.
When it comes to cost and long-term log storage, Loki is an excellent choice for entering cloud solutions.
There are more alternatives on the market β some may be better for you. For example, GKE has Stackdriver integration, which provides an excellent monitoring solution. We did not include them in our analysis in this article.
Links:
The article has been translated and prepared for Π₯Π°Π±Ρ by the staff of β intensives, video courses, and corporate training from practicing specialists (Kubernetes, DevOps, Docker, Ansible, Ceph, SRE, Agile)
Source: habr.com
