🥇Integrating ELK and Exchange. Part 1

I'm starting a series of articles where I want to share my experience connecting Exchange and ELK. This stack will help process large volumes of logs without questioning at what size the usual logging tools will stop being helpful. Let's get acquainted with the new log fighter.

Exchange has a fairly extensive logging system. The most sought-after logs are tracking logs that monitor the step-by-step passage of a specific email within the email organization; web server logs that track each new user session in the system, and logs of specific web applications with varying degrees of session detail. Exchange can also store raw logs of smtp, imap, and pop3 protocols.

What tools can we use to work with logs:

The built-in cmdlet Get-MessageTrackingLog: conveniently processes tracking logs;
The logparser utility: uses a pseudo-SQL language for logging and works quite fast;
External SQL server: for more specific cases (for example, analyzing data over long time intervals).

All this works well when we have a couple of servers and the volume of processed logs is measured in tens or hundreds of gigabytes. But what if the number of servers goes into the dozens, and the size of the logs exceeds a terabyte? This setup will likely start falling apart.

And here’s what happens: Get-MessageTrackingLog starts timing out, logparser hits the ceiling of a 32-bit architecture, and the export to the SQL server breaks down at the worst possible moment, unable to handle a multi-line exception from the service.

Here comes a new player on the scene – the ELK stack, which is specifically designed for juggling huge volumes of logs in reasonable timeframes and with acceptable resource consumption.

In the first part, I will detail how to connect filebeat, which is part of the ELK stack – responsible for reading and sending plain text files where different applications write their logs. In upcoming articles, we will delve into the components of Logstash and Kibana. So, the filebeat agent archive

Installation

can be downloaded from this site We will perform the installation by simply unpacking the contents of the zip file. For example, in.

c:Program Filesfilebeat . Then we need to run the PowerShell scriptinstall-service-filebeat.ps1 , which comes bundled, to install the filebeat service., который идёт в комплекте, для установки сервиса filebeat.

We are now ready to start configuring the configuration file.

Fault tolerance

Filebeat ensures the delivery of logs to the log collection system. This is achieved by maintaining a registry of records in log files. The registry holds information about the records that have been read from the log files and marks specific records that have been successfully delivered to their destination.

If a record cannot be delivered, Filebeat will attempt to resend it until it receives confirmation of delivery from the receiving system or the original log file is deleted during rotation.

Upon restarting the Filebeat service, it will read from the registry information about the last read and delivered records and will read log file entries based on information in the registry.

This minimizes the risk of losing information about logs that need to be sent to the elastic logstash servers during unforeseen failures and while performing server maintenance operations.

For more details, you can read in the documentation in the sections: How does Filebeat keep the state of files and How does Filebeat ensure at-least-once delivery?

Settings

All configuration is done in a configuration file of the format yml, which is divided into several sections. Let's consider some of them that are involved in the process of collecting logs from Exchange servers.

Log processing block

The log processing block starts with the field:

filebeat.inputs:

We will use a common log collection tool:

- type: log

Next, we specify the status (enabled) and the paths to the folder with the logs. For example, in the case of IIS logs, the settings may be as follows:

    enabled: true
    paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log

Another important setting: how Filebeat should read multiline entries. By default, Filebeat considers one line of the log file as one entry. This works well until exceptions related to the service's improper operation begin to enter the log. In this case, exceptions may consist of several lines. Therefore, Filebeat should consider a multiline entry as one if the next line starts with a date. The log entry format in Exchange is such that each new entry in the log file starts with a date. In the configuration, this condition looks like this:

multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after

It makes sense to add tags to the log entries sent, for example:

  tags: ['IIS', 'ex-srv1']

And don't forget to exclude lines that start with a hash symbol:

  exclude_lines: ['^#']

So, the log reading block will look like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log
  multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']

Log sending block

Each entry in the filebeat log file is sent as a JSON object, where the specific log entry is contained in a single message field. If we want to work with this information, we need to parse this field into separate fields. This can be done, for example, in logstash. It will be the recipient of entries from filebeat. Here’s how it might look in the filebeat configuration file:

output.logstash:
  hosts: ["logstash1.domain.com:5044"]

If there are multiple servers, load balancing can be enabled: then filebeat will distribute the logs among several servers instead of sending them to just the first available server in the list:

hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true

During log processing, filebeat adds certain metadata to the sent JSON, in addition to the log entry contained in the message field, which affects the document size that goes into elastic. This metadata can be selectively removed from the send. This is done in the processor block using the processor drop_fields. You can exclude, for example, the following fields:

processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

Care should be taken when choosing fields to exclude, as some of them may be used on the elastic side for index creation.

So, the log sending block will look like this:

output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

Filebeat logging settings

It makes sense to set the following logging settings:

Logging level info;
Log files are written to the default location (logs directory in the filebeat installation directory);
Log file name is filebeat;
Keep the last 10 log files;
Rotate at a size of 1MB.

The final logging configuration block will look like this:

logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

The final configuration will look as follows:

We have compiled the configuration, and it now looks as follows:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:inetpublogsLogFilesW3SVC1*.log
    - C:inetpublogsLogFilesW3SVC2*.log
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']
 
output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
    fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]
 
logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

It is important to understand that the configuration file format is yml. Therefore, it is crucial to set spaces and minus signs correctly.

Filebeat can check the configuration file, and if there are syntax errors, it will indicate which line and where in the line the syntax is incorrect. This check is performed as follows: