Hello everyone.
I am a Linux sysadmin who moved from Russia to Australia on an independent skilled visa in 2015, but this article will not be about how a pig should get a tractor. There are already enough articles like that (if there's interest, I can write about that too), so I would like to talk about how I initiated the migration from one monitoring system to another in my job in Australia as a Linux Ops engineer. Specifically, from Nagios to Icinga2.
The article is partially technical and partially about communication with people and the issues related to cultural differences and work methods.
Unfortunately, the 'code' tag does not highlight Puppet and YAML code, so I had to use 'plaintext'.
Nothing foreshadowed trouble on the morning of December 21, 2016. I was, as usual, reading Habr as an anonymous user in the first half hour of the workday, sipping coffee and stumbled upon .
Since my company was using Nagios, I quickly created a ticket in Redmine and dropped the link in the group chat, as I considered it important. Initiative can be punished even in Australia, so the lead engineer assigned this issue to me since I discovered it.
Screenshot from Redmine
In our department, it's customary to suggest at least one alternative when expressing an opinion, even if the choice is obvious, so I started googling which monitoring systems are currently relevant, as I had my own homemade system at my last job in Russia, which was very primitive but nonetheless quite functional and performed all the tasks assigned to it. Python, St. Petersburg Polytechnic, and the metro rule. No, the metro is awful. This is personal (11 years of work) and deserves a separate article, but not now.
A little about the rules for making changes to the infrastructure configuration at my current workplace. We use Puppet, GitLab, and the principle of Infrastructure as Code, so:
- No manual changes via SSH by altering any files on the virtual machines. Over three years of work, I've faced criticism for this many times, the last time just a week ago, and I don't think it will be the last. Honestly, fixing a single line in the config, restarting the service, and checking if the problem is resolved takes 10 seconds. Creating a new branch in GitLab, pushing the changes, waiting for r10k to work on Puppetmaster, running Puppet --environment=mybranch, and then waiting a few more minutes for everything to process—at least 5 minutes.
- Any changes are made by creating a Merge Request in GitLab and require approval from at least one team member. Serious changes as determined by the team lead require two or three approvals.
- All changes are, in some way, textual (since Puppet manifests, scripts, and Hiera data are text); binary files are strongly discouraged and require compelling reasons for approval.
So, the options I considered:
- Munin — when there are more than 10 servers in the infrastructure, administration turns into a nightmare (from . I didn't have a strong desire to verify this, so I took it at face value).
- Zabbix — I had been eyeing it for a long time, even back in Russia, but at that time it was excessive for my needs. Here, I had to dismiss it due to using Puppet as the configuration manager and GitLab as the version control system. As far as I understood at that time, Zabbix stores all its configuration in a database, which made it unclear how to manage the configuration under current conditions and how to track changes.
- Prometheus — something we will eventually settle on, going by the mood in the department, but at that time I couldn't handle it and couldn't demonstrate a functioning proof of concept, so I had to pass.
- There were also several other options that either required a complete overhaul of the system, or were in their infancy or abandoned, and were thus rejected for the same reasons.
In the end, I settled on Icinga2 for three reasons:
1 — compatibility with Nrpe (the client service that executes checks based on commands from Nagios). This was very important because at that time we had 135 virtual machines (now in 2019 there are 165) with a lot of custom services/checks, and rewriting all of this would have been a huge hassle.
2 — all configuration files are in text format, which allows for easy modifications, creating merge requests to see what has been added or removed.
3 — this is a live and evolving OpenSource project. We really love OpenSource and make significant contributions by creating Pull Requests and Issues to resolve problems.
So, let's get started with Icinga2.
The first challenge was the inertia of my colleagues. Everyone was used to Nagios/Nadji (although we couldn't even agree on how to pronounce it) and the CheckMK interface. The Icinga interface looks quite different (which was a downside), but it offers the ability to flexibly configure what you want to see using filters on virtually any parameter (which was a plus, but I had a hard time fighting for it).
Filters
Evaluate the ratio of the scrollbar size to the scroll area size.
Secondly, everyone was accustomed to viewing the entire infrastructure on one monitor, since CheckMK allows working with multiple Nagios hosts; however, the Icinga interface did not support this (though it actually did, but more on that later). An alternative was something called Thruk, but its design made everyone on the team want to throw up, except for one person — the one who proposed it (not me).
Down with Thruk — the team's unanimous decision.
After a couple of days of brainstorming, I proposed the idea of cluster monitoring, where there is one master host in the production zone and two subordinates — one in dev/test and one external host located with another provider to monitor our services from the perspective of a client or outside observer. This configuration allowed us to see all issues in a single web interface and worked quite well, but Puppet... The problem with Puppet was that the master host now had to know about all hosts and services/checks in the system and had to distribute them among the zones (dev-test, staging-prod, ext), but sending changes via the Icinga API takes a couple of seconds, while compiling the Puppet catalog of all services for all hosts takes a couple of minutes. This is still held against me, although I have explained how everything works and why it takes so long several times.
Thirdly — a bunch of SnowFlakes — things that deviate from the common system because they have something special, so the general rules do not apply to them. This was resolved through a blunt approach — if there are alerts, but everything is actually fine, then I need to dig deeper and understand why it alerts me when it shouldn't. Or conversely — why Nagios panics, but Icinga does not.
Fourthly — Nagios had been running here for three years before me and there was initially more trust in it than in my trendy hipster system, so each time Icinga raised a panic — no one did anything until Nagios got worked up about the same issue. But very rarely did Icinga issue real alerts before Nagios, and I consider this a serious flaw, which I will discuss in the 'Conclusions' section.
As a result, the deployment was delayed for more than 5 months (originally scheduled for June 28, 2018, but actually on December 3, 2018), mainly due to the 'parity check' — that annoying situation where there are several services in Nagios that no one has heard about for the last couple of years, but RIGHT NOW they, damn it, issued a crit without any reason, and I had to explain why they are not on my panel and had to add them to Icinga to ensure 'parity check is complete' (All services/checks in Nagios correspond to services/checks in Icinga).
Implementation:
First, it's the war of Code vs Data, Puppet Style. All data, absolutely everything, must be in Hiera and no other way. All code resides in .pp files. Variables, abstractions, functions—all go into .pp.
As a result, we have a bunch of virtual machines (165 at the time of writing this article) and 68 web applications that need to be monitored for performance and validity of SSL certificates. However, due to historical issues, the information for monitoring applications comes from a separate GitLab repository, and the data format hasn't changed since Puppet 3, creating additional configuration challenges.
Puppet code for applications, take care of your eyes.
define profiles::services::monitoring::docker_apps(
Hash $app_list,
Hash $apps_accessible_from,
Hash $apps_access_list,
Hash $webhost_defaults,
Hash $webcheck_defaults,
Hash $service_overrides,
Hash $targets,
Hash $app_checks,
)
{
#### APPS ####
$zone = $name
$app_list.each | String $app_name, Hash $app_data |
{
$notify_group = { 'notify_group' => ($webcheck_defaults[$zone]['notify_group'] + pick($app_data['notify_group'], {} )) } # adds notifications for default group (systems) + any group defined in int/pm_docker_apps.eyaml
$data = merge($webhost_defaults, $apps_accessible_from, $app_data)
$site_domain = $app_data['site_domain']
$regexp = pick($app_data['check_regex'], 'html') # Pick a regex to check
$check_url = $app_data['check_url'] ? {
undef => { 'http_uri' => '/ },
default => { 'http_uri' => $app_data['check_url'] }
}
$check_regex = $regexp ?{
'absent' => {},
default => {'http_expect_body_regex' => $regexp}
}
$site_domain.each | String $vhost, Hash $vdata | { # Split an app by domains if there are two or more
$vhost_name = {'http_vhost' => $vhost}
$vars = $data['vars'] + $vhost_name + $check_regex + $check_url
$web_ipaddress = is_array($vdata['web_ipaddress']) ? { # Make IP-address an array if it's not, because askizzy has 2 ips and it's an array
true => $vdata['web_ipaddress'],
false => [$vdata['web_ipaddress']],
}
$access_from_zones = [$zone] + $apps_access_list[$data['accessible_from']] # Merge default zone (where the app is defined) and extra zones if they exist
$web_ipaddress.each | String $ip_address | { # For each IP (if we have multiple)
$suffix = length($web_ipaddress) ? { # If we have more than one - add IP as a suffix to this hostname to avoid duplicating resources
1 => '',
default => "_${ip_address}"
}
$octets = split($ip_address, '.')
$ip_tag = "${octets[2]}.${octets[3]}" # Using last octet only causes a collision between nginx-vip 203.15.70.94 and ext. ip 49.255.194.94
$access_from_zones.each | $zone_prefix |{
$zone_target = $targets[$zone_prefix]
$nginx_vip_name = "${zone_prefix}_nginx-vip-${ip_tag}" # If it's a host for ext - prefix becomes 'ext_' (ext_nginx-vip...)
$nginx_host_vip = {
$nginx_vip_name => {
ensure => present,
target => $zone_target,
address => $ip_address,
check_command => 'hostalive',
groups => ['nginx_vip',],
}
}
$ssl_vars = $app_checks['ssl']
$regex_vars = $app_checks['http'] + $vars + $webcheck_defaults[$zone] + $notify_group
if !defined( Profiles::Services::Monitoring::Host[$nginx_vip_name] ) {
ensure_resources('profiles::services::monitoring::host', $nginx_host_vip)
}
if !defined( Icinga2::Object::Service["${nginx_vip_name}_ssl"] ) {
icinga2::object::service {"${nginx_vip_name}_ssl":
ensure => $data['ensure'],
assign => ["host.name == $nginx_vip_name",],
groups => ['webchecks',],
check_command => 'ssl',
check_interval => $service_overrides['ssl']['check_interval'],
target => $targets['services'],
apply => true,
vars => $ssl_vars
}
}
if $regexp != 'absent'{
if !defined(Icinga2::Object::Service["${vhost}${$suffix} regex"]){
icinga2::object::service {"${vhost}${$suffix} regex":
ensure => $data['ensure'],
assign => ["match(*_nginx-vip-${ip_tag}, host.name)",],
groups => ['webchecks',],
check_command => 'http',
check_interval => $service_overrides['regex']['check_interval'],
target => $targets['services'],
enable_flapping => true,
apply => true,
vars => $regex_vars
}
}
}
}
}
}
}
}The configuration code for hosts and services also looks terrible:
monitoring/config.pp
class profiles::services::monitoring::config(
Array $default_config,
Array $hostgroups,
Hash $hosts = {},
Hash $host_defaults,
Hash $services,
Hash $service_defaults,
Hash $service_overrides,
Hash $webcheck_defaults,
Hash $servicegroups,
String $servicegroup_target,
Hash $user_defaults,
Hash $users,
Hash $oncall,
Hash $usergroup_defaults,
Hash $usergroups,
Hash $notifications,
Hash $notification_defaults,
Hash $notification_commands,
Hash $timeperiods,
Hash $webhost_defaults,
Hash $apps_access_list,
Hash $check_commands,
Hash $hosts_api = {},
Hash $targets = {},
Hash $host_api_defaults = {},
)
{
# Profiles::Services::Monitoring::Hostgroup <> # will be enabled when we move to icinga completely
#### APPS ####
case $location {
'int', 'ext': {
$apps_by_zone = {}
}
'pm': {
$int_apps = hiera('int_docker_apps')
$int_app_defaults = hiera('int_docker_app_common')
$st_apps = hiera('staging_docker_apps')
$srs_apps = hiera('pm_docker_apps_srs')
$pm_apps = hiera('pm_docker_apps') + $st_apps + $srs_apps
$pm_app_defaults = hiera('pm_docker_app_common')
$apps_by_zone = {
'int' => $int_apps,
'pm' => $pm_apps,
}
$app_access_by_zone = {
'int' => {'accessible_from' => $int_app_defaults['accessible_from']},
'pm' => {'accessible_from' => $pm_app_defaults['accessible_from']},
}
}
default: {
fail('Please ensure the node has $location fact set (int, pm, ext)')
}
}
file { '/etc/icinga2/conf.d/':
ensure => directory,
recurse => true,
purge => true,
owner => 'icinga',
group => 'icinga',
mode => '0750',
notify => Service['icinga2'],
}
$default_config.each | String $file_name |{
file {"/etc/icinga2/conf.d/${file_name}":
ensure => present,
source => "puppet:////modules/profiles/services/monitoring/default_config/${file_name}",
owner => 'icinga',
group => 'icinga',
mode => '0640',
}
}
$app_checks = {
'ssl' => $services['webchecks']['checks']['ssl']['vars'],
'http' => $services['webchecks']['checks']['http_regexp']['vars']
}
$apps_by_zone.each | String $zone, Hash $app_list | {
profiles::services::monitoring::docker_apps{$zone:
app_list => $app_list,
apps_accessible_from => $app_access_by_zone[$zone],
apps_access_list => $apps_access_list,
webhost_defaults => $webhost_defaults,
webcheck_defaults => $webcheck_defaults,
service_overrides => $service_overrides,
targets => $targets,
app_checks => $app_checks,
}
}
#### HOSTS ####
# Profiles::Services::Monitoring::Host <> # This is for spaceship invasion when it's ready.
$hosts_has_large_disks = query_nodes('mountpoints.*.size_bytes >= 1099511627776')
$hosts.each | String $hostgroup, Hash $list_of_hosts_with_settings | { # Splitting site lists by hostgroups - docker_host/gluster_host/etc
$list_of_hosts_in_group = $list_of_hosts_with_settings['hosts']
$hostgroup_settings = $list_of_hosts_with_settings['settings']
$merged_hostgroup_settings = deep_merge($host_defaults, $list_of_hosts_with_settings['settings'])
$list_of_hosts_in_group.each | String $host_name, Hash $host_settings |{ # Splitting grouplists by hosts
# Is this host in the array $hosts_has_large_disks ? If so set host.vars.has_large_disks
if ( $hosts_has_large_disks.reduce(false) | $found, $value| { ( $value =~ "^${host_name}" ) or $found } ) {
$vars_has_large_disks = { 'has_large_disks' => true }
} else {
$vars_has_large_disks = {}
}
$host_data = deep_merge($merged_hostgroup_settings, $host_settings)
$hostgroup_settings_vars = pick($hostgroup_settings['vars'], {})
$host_settings_vars = pick($host_settings['vars'], {})
$host_notify_group = delete_undef_values($host_defaults['vars']['notify_group'] + $hostgroup_settings_vars['notify_group'] + $host_settings_vars['notify_group'])
$host_data_vars = delete_undef_values(deep_merge($host_data['vars'] , {'notify_group' => $host_notify_group}, $vars_has_large_disks)) # Merging vars separately
$hostgroups = delete_undef_values([$hostgroup] + $host_data['groups'])
profiles::services::monitoring::host{$host_name:
ensure => $host_data['ensure'],
display_name => $host_data['display_name'],
address => $host_data['address'],
groups => $hostgroups,
target => $host_data['target'],
check_command => $host_data['check_command'],
check_interval => $host_data['check_interval'],
max_check_attempts => $host_data['max_check_attempts'],
vars => $host_data_vars,
template => $host_data['template'],
}
}
}
if !empty($hosts_api){ # All hosts managed by API
$hosts_api.each | String $zone, Hash $hosts_api_zone | { # Split api hosts by zones
$hosts_api_zone.each | String $hostgroup, Hash $list_of_hosts_with_settings | { # Splitting site lists by hostgroups - docker_host/gluster_host/etc
$list_of_hosts_in_group = $list_of_hosts_with_settings['hosts']
$hostgroup_settings = $list_of_hosts_with_settings['settings']
$merged_hostgroup_settings = deep_merge($host_api_defaults, $list_of_hosts_with_settings['settings'])
$list_of_hosts_in_group.each | String $host_name, Hash $host_settings |{ # Splitting grouplists by hosts
# Is this host in the array $hosts_has_large_disks ? If so set host.vars.has_large_disks
if ( $hosts_has_large_disks.reduce(false) | $found, $value| { ( $value =~ "^${host_name}" ) or $found } ) {
$vars_has_large_disks = { 'has_large_disks' => true }
} else {
$vars_has_large_disks = {}
}
$host_data = deep_merge($merged_hostgroup_settings, $host_settings)
$hostgroup_settings_vars = pick($hostgroup_settings['vars'], {})
$host_settings_vars = pick($host_settings['vars'], {})
$host_api_notify_group = delete_undef_values($host_defaults['vars']['notify_group'] + $hostgroup_settings_vars['notify_group'] + $host_settings_vars['notify_group'])
$host_data_vars = delete_undef_values(deep_merge($host_data['vars'] , {'notify_group' => $host_api_notify_group}, $vars_has_large_disks))
$hostgroups = delete_undef_values([$hostgroup] + $host_data['groups'])
if defined(Profiles::Services::Monitoring::Host[$host_name]){
$hostname = "${host_name}_from_${zone}"
}
else
{
$hostname = $host_name
}
profiles::services::monitoring::host{$hostname:
ensure => $host_data['ensure'],
display_name => $host_data['display_name'],
address => $host_data['address'],
groups => $hostgroups,
target => "${host_data['target_base']}/${zone}/hosts.conf",
check_command => $host_data['check_command'],
check_interval => $host_data['check_interval'],
max_check_attempts => $host_data['max_check_attempts'],
vars => $host_data_vars,
template => $host_data['template'],
}
}
}
}
}
#### END OF HOSTS ####
#### SERVICES ####
$services.each | String $service_group, Hash $s_list |{ # Service_group and list of services in that group
$service_list = $s_list['checks'] # List of actual checks, separately from SG settings
$service_list.each | String $service_name, Hash $data |{
$merged_defaults = merge($service_defaults, $s_list['settings']) # global service defaults + service group defaults
$merged_data = merge($merged_defaults, $data)
$settings_vars = pick($s_list['settings']['vars'], {})
$this_service_vars = pick($data['vars'], {})
$all_service_vars = delete_undef_values($service_defaults['vars'] + $settings_vars + $this_service_vars)
# If we override default check_timeout, but not nrpe_timeout, make nrpe_timeout the same as check_timeout
if ( $merged_data['check_timeout'] and ! $this_service_vars['nrpe_timeout'] ) {
# NB: Icinga will convert 1m to 60 automatically!
$nrpe = { 'nrpe_timeout' => $merged_data['check_timeout'] }
} else {
$nrpe = {}
}
# By default we use nrpe and all commands are run via nrpe. So vars.nrpe_command = $service_name is a default value
# If it's server-side Icinga command - we don't need 'nrpe_command'
# but there is no harm to have that var and the code is shorter
if $merged_data['check_command'] == 'nrpe'{
$check_command = $merged_data['vars']['nrpe_command'] ? {
undef => { 'nrpe_command' => $service_name },
default => { 'nrpe_command' => $merged_data['vars']['nrpe_command'] }
}
}else{
$check_command = {}
}
# Assembling $vars from Global Default service settings, servicegroup settings, this particular check settings and let's not forget nrpe settings.
if $all_service_vars['graphite_template'] {
$graphite_template = {'check_command' => $all_service_vars['graphite_template']}
}else{
$graphite_template = {'check_command' => $service_name}
}
$service_notify = [] + pick($settings_vars['notify_group'], []) + pick($this_service_vars['notify_group'], []) # pick is required everywhere, otherwise becomes "The value '' cannot be converted to Numeric"
$service_notify_group = $service_notify ? {
[] => $service_defaults['vars']['notify_group'],
default => $service_notify
} # Assing default group (systems) if no other groups are defined
$vars = $all_service_vars + $nrpe + $check_command + $graphite_template + {'notify_group' => $service_notify_group}
# This needs to be merged separately, because merging it as part of MERGED_DATA overwrites arrays instead of merging them, so we lose some "assign" and "ignore" values
$assign = delete_undef_values($service_defaults['assign'] + $s_list['settings']['assign'] + $data['assign'])
$ignore = delete_undef_values($service_defaults['ignore'] + $s_list['settings']['ignore'] + $data['ignore'])
icinga2::object::service {$service_name:
ensure => $merged_data['ensure'],
apply => $merged_data['apply'],
enable_flapping => $merged_data['enable_flapping'],
assign => $assign,
ignore => $ignore,
groups => [$service_group],
check_command => $merged_data['check_command'],
check_interval => $merged_data['check_interval'],
check_timeout => $merged_data['check_timeout'],
check_period => $merged_data['check_period'],
display_name => $merged_data['display_name'],
event_command => $merged_data['event_command'],
retry_interval => $merged_data['retry_interval'],
max_check_attempts => $merged_data['max_check_attempts'],
target => $merged_data['target'],
vars => $vars,
template => $merged_data['template'],
}
}
}
#### END OF SERVICES ####
#### OTHER BORING STUFF ####
$servicegroups.each | $servicegroup, $description |{
icinga2::object::servicegroup{ $servicegroup:
target => $servicegroup_target,
display_name => $description
}
}
$hostgroups.each| String $hostgroup |{
profiles::services::monitoring::hostgroup { $hostgroup:}
}
$notifications.each | String $name, Hash $settings |{
$assign = pick($notification_defaults['assign'], []) + $settings['assign']
$ignore = pick($notification_defaults['ignore'], []) + $settings['ignore']
$merged_settings = $settings + $notification_defaults
icinga2::object::notification{$name:
target => $merged_settings['target'],
apply => $merged_settings['apply'],
apply_target => $merged_settings['apply_target'],
command => $merged_settings['command'],
interval => $merged_settings['interval'],
states => $merged_settings['states'],
types => $merged_settings['types'],
assign => delete_undef_values($assign),
ignore => delete_undef_values($ignore),
user_groups => $merged_settings['user_groups'],
period => $merged_settings['period'],
vars => $merged_settings['vars'],
}
}
# Merging notification settings for users with other settings
$users_oncall = deep_merge($users, $oncall)
# Magic. Do not touch.
create_resources('icinga2::object::user', $users_oncall, $user_defaults)
create_resources('icinga2::object::usergroup', $usergroups, $usergroup_defaults)
create_resources('icinga2::object::timeperiod',$timeperiods)
create_resources('icinga2::object::checkcommand', $check_commands)
create_resources('icinga2::object::notificationcommand', $notification_commands)
profiles::services::sudoers { 'icinga_runs_ping_l2':
ensure => present,
sudoersd_template => 'profiles/os/redhat/centos7/sudoers/icinga.erb',
}
}I am still working on this code and improving it as much as possible. However, it is this kind of code that allowed for a simple and understandable syntax in Hiera:
Data
profiles::services::monitoring::config::services:
perf_checks:
settings:
check_interval: '2m'
assign:
- 'host.vars.type == linux'
checks:
procs: {}
load: {}
memory: {}
disk:
check_interval: '5m'
vars:
notification_period: '24x7'
disk_iops:
vars:
notifications:
- 'silent'
cpu:
vars:
notifications:
- 'silent'
dns_fqdn:
check_interval: '15m'
ignore:
- 'xenserver in host.groups'
vars:
notifications:
- 'silent'
iftraffic_nrpe:
vars:
notifications:
- 'silent'
logging:
settings:
assign:
- 'logserver in host.groups'
checks:
rsyslog: {}
nginx_limit_req_other: {}
nginx_limit_req_s2s: {}
nginx_limit_req_s2x: {}
nginx_limit_req_srs: {}
logstash: {}
logstash_api:
vars:
notifications:
- 'silent'All checks are divided into groups, each group has default settings regarding where and how often to run these checks, what notifications to send and to whom.
In each check, any option can be overridden, and all this eventually combines with the default settings of all checks in general. Therefore, this is the reason for such complexity in config.pp — it merges all default settings with group settings and then with each individual check.
Another significant change was the ability to use functions in the settings, for example, a function to substitute the port, address, and URL for the http_regex check.
http_regexp:
assign:
- 'host.vars.http_regex'
- 'static_sites in host.groups'
check_command: 'http'
check_interval: '1m'
retry_interval: '20s'
max_check_attempts: 6
http_port: '{{ if(host.vars.http_port) { return host.vars.http_port } else { return 443 } }}'
vars:
notification_period: 'host.vars.notification_period'
http_vhost: '{{ if(host.vars.http_vhost) { return host.vars.http_vhost } else { return host.name } }}'
http_ssl: '{{ if(host.vars.http_ssl) { return false } else { return true } }}'
http_expect_body_regex: 'host.vars.http_regex'
http_uri: '{{ if(host.vars.http_uri) { return host.vars.http_uri } else { return "\/" } }}'
http_onredirect: 'follow'
http_warn_time: 8
http_critical_time: 15
http_timeout: 30
http_sni: trueThis means — if there is a variable in the host definition http_port — use it, otherwise default to 443. For example, the Jabber web interface runs on 9090, while Unifi runs on 7443.
http_vhost means to ignore DNS and take this address.
If the host specifies a URI — follow it, otherwise take "\/".
There was a funny story with http_ssl — this thing just wouldn't turn off on demand. I spent a long time puzzled over this line until I finally realized that the variable in the host definition:
http_ssl: falseInserted into the expression
if(host.vars.http_ssl) { return false } else { return true }as false and in the end, it results in
if(false) { return false } else { return true }that is, the SSL check is always active. This was resolved by changing the syntax:
http_ssl: noConclusions:
Pros:
- We now have one monitoring system instead of two, as we had for the last 7-8 months, or one outdated and vulnerable system.
- The data structure of hosts/services (checks) is now (in my opinion) much more readable and understandable. For others, it turned out to be not so obvious, so I had to create a couple of pages in the local wiki to clarify how everything works and where to make changes.
- There is the possibility of flexible configuration of checks using variables and functions; for example, to check http_regexp, the required pattern, return code, URL, and port can be set in the host settings.
- There are several dashboards, each of which can define its own list of displayed alerts and manage them all through Puppet and merge requests.
Cons:
- The inertia of team members — Nagios worked, worked, and worked, while your Icinga constantly glitches and slows down. How can you see the history? Oh, wait, it doesn't update... (The real problem — the alert history doesn't update automatically, only on F5)
- The system's inertia — when I click on 'refresh' (check now) in the web interface, the execution result depends on the weather on Mars, especially on complex services that take tens of seconds to complete. Such a result is normal.
- Overall, according to the six-month statistics of the two systems working side by side, Nagios always performed faster than Icinga, and this really annoyed me. It seems like something was messed up with the timers, and the check every five minutes actually occurs every 5:30 or something like that.
- If you restart the service at any moment (systemctl restart icinga2) — all checks that were in progress at that time will throw a critical alert on the screen, and it appears as though everything has crashed ().
But overall — it works.
Source: habr.com
