1. Elastic stack: analyzing security logs. Introduction

With the cessation of sales of the logging and analytics system Splunk in Russia, the question arose about what solution can replace it. After spending time familiarizing myself with various solutions, I settled on a solution for the real man — the ELK stack. This system requires time for setup, but as a result, one can achieve a very powerful system for analyzing the state and responding to information security incidents in the organization. In this series of articles, we will explore the basic (and perhaps not) capabilities of the ELK stack, examining how to parse logs, how to build charts and dashboards, and what interesting features can be implemented using logs from Check Point firewalls or the OpenVas security scanner. To begin, let's take a closer look at what the ELK stack is and what components it consists of.

the ELK stack — is an abbreviation for three open-source projects: Elasticsearch, Logstash and Kibana. Developed by Elastic along with all related projects. Elasticsearch is the core of the entire system, which combines the functions of a database, search, and analytics system. Logstash is a data processing pipeline on the server side that collects data from multiple sources simultaneously, parses the logs, and then sends them to the Elasticsearch database. Kibana allows users to visualize data using charts and graphs within Elasticsearch. Database administration can also be done through Kibana. Next, we will take a closer look at each system individually.

Logstash

Logstash is a utility for processing log events from various sources, enabling the extraction of fields and their values from messages, as well as configuration of data filtering and editing. After all manipulations, Logstash redirects events to the final data storage. The utility is configured exclusively through configuration files.
A typical Logstash configuration consists of one or more files containing several incoming information streams (input), several filters for this information (filter), and several outgoing streams (output). It looks like one or several configuration files, which in the simplest case (that does nothing at all) appears like this:

input {
}

filter {
}

output {
}

In INPUT, we configure which port the logs will be sent to and which protocol to use, as well as the folder from which to read new or continuously appending files. In FILTER, we set up the log parser: field analysis, value editing, adding new parameters, or deletion. FILTER is a field for managing the messages that come to Logstash with numerous editing options. In OUTPUT, we specify where to send the already parsed log. If it's to Elasticsearch, a JSON request is sent containing fields with values, or during debugging, it can be outputted to stdout or written to a file.

ElasticSearch

Initially, Elasticsearch is a solution for full-text search, offering additional conveniences such as easy scalability, replication, and more, making it a very user-friendly and effective solution for high-load projects with large data volumes. Elasticsearch is a non-relational document store (NoSQL) in JSON format and a search engine based on full-text search using Lucene. The hardware platform is the Java Virtual Machine, which requires a significant amount of CPU and RAM resources to operate.
Each incoming message, whether from Logstash or through an API request, is indexed as a "document" – similar to a table in relational SQL. All documents are stored in an index – akin to a database in SQL.

Example of a document in the database:

{
  "_index": "checkpoint-2019.10.10",
  "_type": "_doc",
  "_id": "yvNZcWwBygXz5W1aycBy",
  "_version": 1,
  "_score": null,
  "_source": {
	"layer_uuid": [
      "dae7f01c-4c98-4c3a-a643-bfbb8fcf40f0",
      "dbee3718-cf2f-4de0-8681-529cb75be9a6"
	],
	"outzone": "External",
	"layer_name": [
  	"TSS-Standard Security",
  	"TSS-Standard Application"
	],
	"time": "1565269565",
	"dst": "103.5.198.210",
	"parent_rule": "0",
	"host": "10.10.10.250",
	"ifname": "eth6",
    ]
}

All interactions with the database are built on JSON requests via the REST API, which either return documents by index or provide some statistics in a question-and-answer format. To visualize all responses to the queries, Kibana was developed, which is a web service.

Kibana

Kibana allows you to search and query data from an Elasticsearch database, building many beautiful graphs and dashboards based on the responses. The system also features administration functionality for the Elasticsearch database; in subsequent articles, we will explore this service in more detail. For now, we'll show examples of dashboards for the Check Point firewall and the OpenVas vulnerability scanner that can be created.

Example dashboard for Check Point, the image is clickable:

Example dashboard for OpenVas, the image is clickable:

Conclusion

We have examined what consists of ELK stack, got a little acquainted with the main products, and in the course ahead, we will separately consider writing a Logstash configuration file, setting up dashboards in Kibana, getting familiar with API requests, automation, and much more!

So stay tuned for updates (Telegram, Facebook, VK, TS Solution Blog), Yandex.Zen.

Source: habr.com