In previous articles, we briefly familiarized ourselves with the ELK stack and the configuration file setup for Logstash as a log parser. In this article, we will move on to the most critical aspect from an analytics perspectiveāthe data visualization you want from the system and the reason it was all created: graphs and tables combined into dashboards. Today, we will take a closer look at the visualization system Kibana, explore how to create graphs and tables, and ultimately build a simple dashboard based on logs from the Check Point firewall.
The first step in working with Kibana is creating an index pattern, which, logically, is a base of indices unified by a specific principle. Of course, this is solely a configuration to help Kibana search for information across all indices more conveniently. It is defined by matching a string, such as ācheckpoint-*ā with the index name. For example, ācheckpoint-2019.12.05ā will fit this pattern, but simply ācheckpointā will not. It is worth mentioning that searching for information across different index patterns simultaneously is not possible; later in subsequent articles, we will see that API requests are made either by the index name or by one pattern string. The image is clickable:
After this, we check in the Discover menu that all logs are indexed and the parser is set up correctly. If any discrepancies are found, such as changing the data type from string to integer, it is necessary to edit the Logstash configuration file so that the new logs are recorded correctly. To ensure that old logs take the required format after the change, only reindexing can help; we will discuss this operation in more detail in subsequent articles. Let's confirm that everything is in order; the image is clickable:
The logs are in place, so we can start building dashboards. Based on the analytics from security product dashboards, we can understand the state of information security in the organization, visually identify vulnerable spots in the current policy, and subsequently develop ways to address them. We'll build a small dashboard using several visualization tools. The dashboard will consist of 5 components:
- a table to count the total number of logs by blades
- a table for critical IPS signatures
- a pie chart for Threat Prevention events
- chart of the most popular visited websites
- chart of the usage of the most dangerous applications
To create visualization figures, go to the menu Visualize, and select the desired shape that we want to build! Let's go step by step.
Table for counting the total number of logs by blades
For this, let's choose a shape Data Table, diving into the toolkit for creating graphs, with settings for the shape on the left and how it will look with the current settings on the right. First, I will demonstrate what the finished table will look like, and after that, we will go through the settings; the image is clickable:
More detailed settings for the shape, the image is clickable:
Let's break down the settings.
Initially, configure metric, which is the value by which all fields will be aggregated. Metrics are calculated based on values extracted in one way or another from documents. Values are usually extracted from fields of the document, but can also be generated using scripts. In this case, we set it to Aggregation: Count (total number of logs).
After that, we divide the table by segments (fields) on which the metric will be calculated. This function is performed by the Buckets setting, which consists of 2 configuration options:
- split rows ā adding columns and subsequently dividing the table into rows
- split table ā dividing into several tables by the values of a specific field.
In buckets. You can add several divisions to create multiple columns or tables; the limitations are mostly logical. In aggregation, you can choose the method of segmentation: ipv4 range, date range, Terms, etc. The most interesting choice is Terms and Significant Terms, segmentation is performed based on the values of a specific index field, the difference between them lies in the number of returned values and their display. Since we want to divide the table by the names of the blades, we choose the field ā product.keyword and set the size to return 25 values.
Instead of rows, Elasticsearch uses 2 types of data ā text and keywordIf you want to perform a full-text search, you need to use the text type, which is very convenient when developing your own search service; for example, you can search for mentions of a word within a specific field value (text). If you only want an exact match, you should use the keyword type. The keyword data type should also be used for fields that require sorting or aggregation.
As a result, Elasticsearch counts the number of logs over a specific time period, aggregating by the value in the product field. In Custom Label, we set the name of the column that will be displayed in the table, set the time period for which we collect logs, and start the visualization ā Kibana sends a request to Elasticsearch, waits for a response, and then visualizes the obtained data. The table is ready!
Pie chart for Threat Prevention events
The information about the percentage of responses is of particular interest. detect and prevent for security incidents in the current security policy. In this case, a pie chart is suitable. Select in Visualize ā Pie chart. Also, in the metric, we specify aggregation by the number of logs. In buckets, set Terms => action.
Everything seems correct, but the results show values for all blades; we need to filter only for those blades that operate within Threat Prevention. Therefore, it's essential to set up a filter to search for information only on the blades responsible for security incidents ā product: ("Anti-Bot" OR "New Anti-Virus" OR "DDoS Protector" OR "SmartDefense" OR "Threat Emulation"). The image is clickable:
And more detailed settings; the image is clickable:
Table of IPS events
Next, from a security perspective, it is crucial to review and check events by the blade IPS and Threat Emulation, which are not blocked by the current policy, so that later either the signature can be switched to prevent or, if the traffic is valid, the signature does not need to be checked. The table is created just like in the first example, with the difference that we create several columns: protections.keyword, severity.keyword, product.keyword, originsicname.keyword. Be sure to set a filter to search for information only on the blades responsible for security incidents ā product: ("SmartDefense" OR "Threat Emulation"). The image is clickable:
More detailed settings; the image is clickable:
Diagrams of the most popular visited sites
To do this, we create a figure ā Vertical Bar. We also use the count metric (Y-axis), and on the X-axis, we will use the names of the visited sites ā āappi_nameā. There is a small trick here: if you run the settings in the current version, all sites will be marked on the graph in one color. To make them colorful, we use an additional setting ā āsplit seriesā, which allows dividing an already prepared column into several values, depending on the selected field of course! This division can either be used as one colorful column by values in stacked mode or in normal mode to create several columns by specific values on the X-axis. In this case, we use the same value as on the X-axis, which allows all columns to be colorful; they will be marked by colors at the top right. In the filter, we set ā product: Ā«URL FilteringĀ» to see information only for the visited sites; the image is clickable:
Settings:
Chart for the usage of the most dangerous applications
To do this, we create a figure ā Vertical Bar. We also use the count metric (Y-axis), and on the X-axis, we will use the names of the used applications ā āappi_nameā. The most important part is setting the filter ā product: Ā«Application ControlĀ» AND app_risk: (4 OR 5 OR 3) AND action: Ā«acceptĀ». We filter the logs by the Application control blade, taking only those sites that are categorized as sites with Critical, High, and Medium risk only if access to these sites is allowed. The image is clickable:
Settings, clickable:
Dashboard
Viewing and creating dashboards is found in a separate menu item ā Dashboard. Here everything is simple: a new dashboard is created, visualizations are added, arranged in place, and thatās it!
We create a dashboard to understand the basic state of information security in the organization, of course, only at the Check Point level; the image is clickable:
Based on these graphs, we can understand which critical signatures are not blocked by the firewall, where users are going, and which are the most dangerous applications they are using.
Conclusion
We explored the basic visualization capabilities in Kibana and created a dashboard, but that is just a small part. In the next section of the course, we will separately address map configuration, working with the Elasticsearch system, get acquainted with API requests, automation, and much more!
So stay tuned for updates (, , , ), .
Source: habr.com
