In previous articles, we got a little acquainted with the elk stack and setting up the Logstash configuration file for the log parser, in this article we will move on to the most important from the point of view of analytics, what you want to see from the system and for what everything was created - these are graphs and tables combined in dashboards. Today we will take a closer look at the visualization system kibana, we will consider how to create graphs, tables, and as a result we will build a simple dashboard based on logs from the Check Point firewall.
The first step in working with kibana is to create index pattern, logically, this is the base of single indexes according to a certain principle. Of course, this is only a setting for Kibana to more conveniently search for information on all indexes at the same time. It is set by comparing the string, say “checkpoint-*” and the name of the index. For example, “checkpoint-2019.12.05 ” will fit the pattern, but simply “checkpoint” is no longer there. Separately, it is worth mentioning that it is impossible to search for information on different index patterns at the same time in the search, a little later in subsequent articles we will see that API requests are made either by the name of the index, or just by one line of the pattern, the picture is clickable:
After that, we check in the Discover menu that all logs are indexed, and the correct parser is configured. If any inconsistencies are found, for example, changing the data type from a string to an integer, you need to edit the Logstash configuration file, as a result, new logs will be written correctly. In order for the old logs to take the desired form before the change, only the reindexing process helps, in subsequent articles this operation will be considered in more detail. Make sure everything is in order, the picture is clickable:
The logs are in place, so you can start building dashboards. Based on dashboard analytics from security products, you can understand the state of information security in the organization, visually see the vulnerabilities in the current policy, and develop ways to eliminate them in the future. Let's build a small dashboard using several visualization tools. The dashboard will consist of 5 components:
- table for calculating the total number of logs by blades
- IPS critical signature table
- pie chart by Threat Prevention events
- chart by most visited sites
- chart on the use of the most dangerous applications
To create visualization shapes, you need to go to the menu Visualize, and select the shape you want to build! Let's go in order.
Table for calculating the total number of logs by blades
To do this, we choose a figure Data Table, we fall into the snap-in for creating charts, the figure settings are put down on the left, how it will look in the current settings on the right. First, I will demonstrate how the finished table will look like, after that we will go through the settings, the picture is clickable:
More detailed settings of the figure, the picture is clickable:
Let's take a look at the settings.
Initially configured metrics, this is the value by which all fields will be aggregated. Metrics are calculated based on values extracted in one way or another from documents. Values are usually taken from fields document, but can also be generated using scripts. In this case, we put in Aggregation: Count (total number of logs).
After that, we divide the table into segments (fields) by which the metric will be calculated. This function is performed by the Buckets setting, which in turn consists of 2 setting options:
- split rows - adding columns and then dividing the table into rows
- split table - division into several tables by the values of a certain field.
В buckets you can add several divisions to create several columns or tables, the restrictions here are rather logical. In aggregation, you can choose how the division into segments will take place: ipv4 range, date range, Terms, etc. The most intriguing choice is precisely Terms и Significant Terms, segmentation is performed by the values of a certain index field, the difference between them is the number of returned values, and their display. Since we want to divide the table by the name of the blades, select the field - product.keyword and set the size to 25 return values.
Instead of strings, elasticsearch uses 2 data types − text и keyword. If you want to do a full text search, you should use the text type, a very handy thing to do when writing your search service, like looking for the occurrence of a word in a particular field value (text). If you only want an exact match, you should use the keyword type. Also, the keyword data type should be used for fields that require sorting or aggregation, that is, in our case.
As a result, Elasticsearch counts the number of logs for a certain time, aggregated by the value in the product field. In the Custom Label, we set the name of the column that will be displayed in the table, set the time for which we collect logs, start drawing - Kibana sends a request to elasticsearch, waits for a response, and then visualizes the received data. The table is ready!
Threat Prevention Pie Chart
Of particular interest is the information, and how much in general in the percentage of reactions detect и Prevent for information security incidents in the current security policy. A pie chart works well for this case. Select in Visualize - pie chart. Also in the metric we set the aggregation by the number of logs. In buckets we put Terms => action.
Everything seems to be correct, but as a result, values are shown for all blades, you need to filter only for those blades that work as part of Threat Prevention. Therefore, we must set filter in order to search for information only on blades responsible for information security incidents - product: ("Anti-Bot" OR "New Anti-Virus" OR "DDoS Protector" OR "SmartDefense" OR "Threat Emulation"). The picture is clickable:
And more detailed settings, the picture is clickable:
Table by IPS events
Further, it is very important from the point of view of information security to view and check events by blade IPS и Threat EmulationThat are not blocked current policy in order to subsequently either transfer the signature to prevent, or if the traffic is valid, do not check the signature. We create the table in the same way as for the first example, only with the difference that we create several columns: protections.keyword, severity.keyword, product.keyword, originsicname.keyword. Be sure to set up a filter in order to search for information only on the blades responsible for information security incidents - product: ("SmartDefense" OR "Threat Emulation"). The picture is clickable:
More detailed settings, the picture is clickable:
Charts for the most popular visited sites
To do this, we create a figure - vertical bar. We also use the count metric (Y axis), and on the X axis, we will use the name of the visited sites — “appi_name” as values. There is a little trick here, if you run the settings in the current version, then all sites will be marked on the chart with one color, in order to make them multi-colored, we use an additional setting - “split series”, which allows you to divide an already prepared column into several more values, depending on from the selected field of course! This very division can either be used as one multi-colored column by values in stacked mode, or in normal mode in order to create several columns by a certain value from the x-axis. In this case, here we use the same value as for the x-axis, this makes it possible to make all columns multi-colored, on the top right they will be indicated by colors. In the filter, set - product: "URL Filtering" in order to see information only on visited sites, the picture is clickable:
Settings:
Chart on the use of the most dangerous applications
To do this, create a shape - Vertical Bar. We also use the metric count (Y axis), and on the X axis we will use the name of the applications used - “appi_name” as values. The most important is the filter setting - product: "Application Control" AND app_risk: (4 OR 5 OR 3 ) AND action: "accept". We filter the logs by the Application control blade, take only those sites that are categorized as sites with the risk of Critical, High, Medium, and only if access to these sites is allowed. The picture is clickable:
Settings, clickable:
Dashboard
Viewing and creating dashboards is in a separate menu item - Dashboard. Everything is simple here, a new dashboard is created, visualization is added to it, put in place and that's it!
We create a dashboard by which it will be possible to understand the basic situation of the state of information security in the organization, of course, only at the Check Point level, the picture is clickable:
Based on these graphs, we can understand what critical signatures are not blocked on the firewall, where users go, what are the most dangerous applications they use.
Conclusion
We looked at the basic visualization capabilities in Kibana and built a dashboard, but this is only a small part. Further in the course, we will separately consider setting up maps, working with the elasticsearch system, get acquainted with API requests, automation, and much more!
So stay tuned, , , ), .
Source: habr.com