Hello, my name is Yevgeny, and I am the B2B team lead at Citymobil. One of my team's tasks is to support integrations for taxi orders from partners, and to ensure stable service, we always need to understand what is happening in our microservices. Therefore, we must constantly monitor the logs.
At Citymobil, we use the ELK stack (ElasticSearch, Logstash, Kibana) to work with logs, and the volume of incoming data is enormous. Finding issues in this sea of requests that may arise after deploying new code is quite challenging. To visualize these issues, Kibana has a Dashboard section.
There are quite a few articles on Habr with examples of how to set up the ELK stack for receiving and storing data, but there are no relevant materials on creating a Dashboard. Therefore, I want to show how to create a visual representation of data based on incoming logs in Kibana.
Settings
To make it clearer, I created a Docker image with ELK and Filebeat. I placed a small Go application that will generate test logs for our example. I won’t go into detail about configuring ELK as there is enough written about it on Habr.
We clone the repository with the configuration docker-compose and the settings of ELK and run it using the command to start the containers. This command will bring up 3 containers:. I intentionally do not add the key -d, to observe the operation of the ELK stack.
git clone https://github.com/et-soft/habr-elk
cd habr-elk
docker-compose up
If everything is set up correctly, we will see a log entry (possibly not immediately, starting the container with the entire stack may take a few minutes):
{"type":"log","@timestamp":"2020-09-20T05:55:14Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at http://0:5601"}
At the address localhost:5061 Kibana should open.
The only thing we need to configure is to create an Index Pattern for Kibana with information on which data to display. For this, we will execute a curl request or perform a series of actions in the graphical interface.
$ curl -XPOST -D- 'http://localhost:5601/api/saved_objects/index-pattern'
-H 'Content-Type: application/json'
-H 'kbn-xsrf: true'
-d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'
Creating an Index Pattern through the graphical interface
To configure, select the Discover section in the left menu, and we arrive at the Index pattern creation page.
By clicking on the 'Create index pattern' button, we reach the index creation page. In the 'Index pattern name' field, we enter 'logstash-*'. If everything is set up correctly, Kibana will show the indices that match the rule below.
On the next page, we choose the timestamp field, which in our case is @timestamp.
As a result, a settings index page will appear, but no further actions are required from us at this time.
We can now return to the Discover section, where we will see the log entries.
Dashboard
In the left menu, click on the Dashboard creation section to access the corresponding page.
Click on "Create new dashboard" to go to the page for adding objects to the Dashboard.
Click on the "Create new" button, and the system will prompt you to choose a data display type. Kibana offers a wide variety, but we will look at creating a graphical representation of a "Vertical Bar" and a tabular representation of a "Data Table". Other display types are configured similarly.
Some available objects are labeled with B and E — this means the format is experimental or in beta testing. Over time, the format may change or disappear entirely from Kibana.
Vertical Bar
For the example "Vertical Bar", we will create a histogram showing the ratio of successful to unsuccessful response statuses from our service. Upon completing the settings, we will get the following chart:
We will consider successful all requests with a response status = 400.
To create the "Vertical Bar" chart, we need to select the data source. Choose the Index Pattern that you created earlier.
By default, after selecting the data source, one continuous chart will appear. Let's configure it.
In the "Buckets" section, click the "Add" button, select "X-axis", and configure the X-axis. We will plot the timestamps of log entries. For the "Aggregation" field, choose "Date Histogram", and in the "Field" field, select "@timestamp", which points to the time field. Leave the "Minimum interval" as "Auto" so that it automatically adjusts to our display.
After clicking the "Update" button, we will see a chart showing the number of requests every 30 seconds.
Now, let's configure the columns on the Y-axis. Currently, we are displaying the total number of requests in the selected time interval.
Change the value of "Aggregation" to "Sum Bucket", which will allow us to combine data for successful and unsuccessful requests. In the Bucket -> Aggregation section, choose aggregation by "Filters" and set the filter for "statusCode >= 400". In the "Custom label" field, we specify our metric name for clearer display in the legend on the chart and in the overall list.
Clicking the "Update" button under the settings block will yield a chart with problematic requests.
If you click on the circle near the legend, a window will appear where you can change the color of the columns.
Now let's add data about successful requests to the chart. In the 'Metrics' section, click the 'Add' button and select 'Y-axis'.
For the created metric, make the same settings as for failed requests. Just specify 'statusCode < 400' in the filter.
By changing the color of the new column, we will get a display of the ratio of problematic to successful requests.
By clicking the 'Save' button at the top of the screen and specifying a name, we will see the first chart on the Dashboard.
Data Table
Now let's look at the 'Data Table' representation. We will create a table with a list of all URLs that were requested and the number of those requests. Like in the Vertical Bar example, we first choose the data source.
After that, a table with one column will be displayed on the screen, showing the total number of requests for the selected time interval.
We will only modify the 'Buckets' block. Click the 'Add' button and select 'Split rows'.
In the 'Aggregation' field, select 'Terms'. In the newly appeared 'Field' field, choose 'url.keyword'.
By specifying 'Url' in the 'Custom label' field and clicking 'Update', we will get the desired table with the number of requests for each URL over the selected time period.
At the top of the screen, we will click the 'Save' button again and name the table, for example, Urls. We will go back to the Dashboard and see both created displays.
Working with the Dashboard
When creating a Dashboard, we only set the basic view parameters in the display object settings. There's no point in specifying data for filters in objects, such as 'date range', 'filter by user agent', 'filter by request country', etc. It's much more convenient to specify the desired time interval or set the necessary filtering in the query panel located above the objects.
The filters added to this panel will apply to the entire Dashboard, and all display objects will be rebuilt according to the current filtered data.
Conclusion
Kibana is a powerful tool that allows you to visualize any data in a convenient way. I tried to show the setup of two main types of displays. But other types can be configured similarly. The multitude of settings that I left 'off-screen' will allow you to flexibly customize charts according to your needs.
Source: habr.com
