Hammaga salom.
Men Linux tizimining ma'muriman, men 2015 yilda mustaqil professional vizada Rossiyadan Avstraliyaga ko'chib o'tdim, ammo maqola cho'chqa uchun traktorni qanday ishga tushirish haqida bo'lmaydi. Bunday maqolalar allaqachon yetarli (shunga qaramay, agar qiziqish bo'lsa, men ham bu haqda yozaman), shuning uchun men Avstraliyada Linux-ops muhandisi sifatida ishlaganimda qanday qilib bitta tizim monitoringidan migratsiyani boshlaganim haqida gapirmoqchiman. boshqasiga. Xususan - Nagios => Icinga2.
Maqola qisman texnik va qisman odamlar bilan muloqot va madaniyat va ish uslublaridagi farqlar bilan bog'liq muammolar haqida.
Afsuski, "kod" yorlig'i Qo'g'irchoq va yaml kodini ta'kidlamaydi, shuning uchun men "to'g'ri matn" dan foydalanishga majbur bo'ldim.
21 yil 2016 dekabr kuni ertalab hech qanday muammo belgilari yo'q edi. Odatdagidek, men ish kunining birinchi yarim soatida ro'yxatdan o'tmagan anonim foydalanuvchi tomonidan Habrni o'qiyotgan edim, qahva ichib, ko'rib qoldim. .
Mening kompaniyam Nagiosdan foydalanganligi sababli, ikki marta o'ylamasdan, men Redmine-da chipta yaratdim va havolani umumiy chatga yubordim, chunki men buni muhim deb bildim. Tashabbus hatto Avstraliyada ham jazolanadi, shuning uchun men uni kashf qilganimdan beri bosh muhandis muammoni menga bog'lab qo'ydi.
Redmine-dan skrinshot
Bizning bo'limda fikringizni bildirishdan oldin, tanlov aniq bo'lsa ham, hech bo'lmaganda bitta muqobil taklif qilish odatiy holdir, shuning uchun men hozirda qanday monitoring tizimlari dolzarbligini googlingdan boshladim, chunki Rossiyada oxirgi ish joyimda men o'zimning yozma tizimim bor edi, juda ibtidoiy, lekin shunga qaramay, juda yaxshi ishlaydi va unga yuklangan barcha vazifalarni bajaradi. Python, Sankt-Peterburg Politexnika va metro qoidasi. Yo'q, metro yomon. Bu shaxsiy (11 yillik ish) va alohida maqolaga loyiq, ammo hozir emas.
Mening hozirgi joyimda infratuzilma konfiguratsiyasiga o'zgartirish kiritish qoidalari haqida bir oz. Kod printsipi sifatida biz Puppet, Gitlab va Infratuzilmadan foydalanamiz, shuning uchun:
- Virtual mashinalarda biron bir faylni qo'lda o'zgartirish orqali SSH orqali qo'lda o'zgarishlar amalga oshirilmaydi. Uch yillik ish davomida men buning uchun ko'p marta shlyapa bilan urishganman, oxirgi marta bir hafta oldin bo'lgan va bu oxirgi marta emas deb o'ylayman. Xo'sh, haqiqatan ham - konfiguratsiyadagi bitta qatorni to'g'rilang, xizmatni qayta ishga tushiring va muammo hal qilinganligini tekshiring - 10 soniya. Gitlab-da yangi filial yarating, o'zgarishlarni suring, r10k Puppetmaster-da ishlashini kuting, Puppet -environment=mybranch-ni ishga tushiring va hammasi ishlaguncha yana bir necha daqiqa kuting - kamida 5 daqiqa.
- Har qanday o'zgarishlar Gitlab-da birlashtirish so'rovini yaratish orqali amalga oshiriladi va kamida bitta jamoa a'zosi tomonidan tasdiqlanishi kerak. Jamoa rahbari tomonidan qaror qabul qilingan asosiy o'zgarishlar ikki yoki uchta tasdiqlashni talab qiladi.
- Barcha o'zgarishlar u yoki bu tarzda matndir (chunki qo'g'irchoq manifestlari, skriptlar va Hiera ma'lumotlari matndir), ikkilik fayllar juda tavsiya etilmaydi va bunday fayllarni tasdiqlash uchun jiddiy sabab bo'lishi kerak.
Shunday qilib, men ko'rib chiqqan variantlar:
- Munin - agar infratuzilmada 10 dan ortiq server bo'lsa, ma'muriyat do'zaxga aylanadi (dan . Men buni tekshirishni xohlamadim, shuning uchun men uning so'zini qabul qildim).
- Zabbix - Men unga Rossiyada uzoq vaqt qarardim, lekin keyin bu mening vazifalarim uchun ortiqcha edi. Bu erda - qo'g'irchoqni konfiguratsiya menejeri sifatida va Gitlab-dan versiyani boshqarish tizimi sifatida foydalanish tufayli bekor qilinishi kerak edi. O'sha paytda, men tushunganimdek, Zabbix butun konfiguratsiyani ma'lumotlar bazasida saqlaydi va shuning uchun hozirgi sharoitda konfiguratsiyani qanday boshqarish va o'zgarishlarni qanday kuzatish mumkinligi aniq emas edi.
- Kafedradagi kayfiyatga qarab, biz oxir-oqibatda Prometeyga erishamiz, lekin o'sha paytda men uni o'zlashtirmaganman va haqiqatan ham ishlaydigan namunani (Tushuncha isboti) namoyish eta olmadim, shuning uchun men rad etishga majbur bo'ldim.
- Tizimni to'liq qayta ishlashni talab qiladigan yoki boshlang'ich davridagi / tashlab qo'yilgan va xuddi shu sababga ko'ra rad etilgan bir nechta boshqa variantlar ham mavjud edi.
Oxir-oqibat, men uchta sababga ko'ra Icinga2-ga joylashdim:
1 - Nrpe bilan moslik (Nagios-dan buyruq tekshiruvlarini bajaradigan mijoz xizmati). Bu juda muhim edi, chunki o'sha paytda bizda 135 ta (hozir 2019 yilda 165 ta) virtual mashinalar bo'lib, bir nechta maxsus yozma xizmatlar/cheklar mavjud edi va ularning barchasini qayta tiklash haqiqiy og'riq bo'lar edi.
2 - barcha konfiguratsiya fayllari matn bo'lib, bu masalani tahrirlashni osonlashtiradi, nima qo'shilgan yoki o'chirilganligini ko'rish imkoniyati bilan birlashma so'rovlarini yaratadi.
3 - bu tirik va rivojlanayotgan OpenSource loyihasi. Biz OpenSource-ni juda yaxshi ko'ramiz va muammolarni hal qilish uchun Pull so'rovlari va muammolarini yaratish orqali unga qo'shimcha hissa qo'shamiz.
Shunday qilib, ketaylik, Icinga2.
Men duch kelishim kerak bo'lgan birinchi narsa hamkasblarimning inertsiyasi edi. Hamma Nagios/Naggiosga (garchi bu erda ham uni qanday talaffuz qilish bo'yicha murosaga kela olmagan bo'lsa ham) va CheckMK interfeysiga o'rganib qolgan. Icinga interfeysi butunlay boshqacha ko'rinadi (bu minus edi), lekin tom ma'noda har qanday parametr uchun filtrlar yordamida ko'rish kerak bo'lgan narsalarni moslashuvchan tarzda sozlash mumkin (bu ortiqcha edi, lekin men buning uchun juda ko'p kurashdim).
Filtrlar
O'tkazish paneli o'lchamini aylantirish maydonining o'lchamiga nisbatini hisoblang.
Ikkinchidan, hamma butun infratuzilmani bitta monitorda ko'rishga odatlangan, chunki CheckMk bir nechta Nagios xostlari bilan ishlashga imkon beradi, lekin Icinga interfeysi buni qila olmadi (aslida, bu mumkin, lekin quyida ko'proq). Muqobil variant Thruk deb nomlangan narsa edi, lekin uning dizayni jamoadagi barchani gangib qoldirdi, faqat bitta taklif qilgan (men emas).
Thruk pechiga - jamoaning bir ovozdan qarori
Bir necha kunlik aqliy hujumdan so'ng, men ishlab chiqarish zonasida bitta asosiy xost va ikkita qul - biri ishlab/sinovda va bitta tashqi xost boshqa provayderda joylashgan bo'lsa, klaster monitoringi g'oyasini taklif qildim. mijoz yoki begona kuzatuvchi nuqtai nazaridan xizmatlar. Ushbu konfiguratsiya barcha muammolarni bitta veb-interfeysda ko'rish imkonini berdi va juda yaxshi ishladi, lekin Qo'g'irchoq... Qo'g'irchoq bilan bog'liq muammo shundaki, asosiy xost endi tizimdagi barcha xostlar va xizmatlar/cheklar haqida bilishi kerak edi va ularni zonalar o'rtasida taqsimlash (dev-test, staging-prod, ext), lekin o'zgarishlarni Icinga API orqali yuborish bir necha soniya davom etadi, lekin barcha xostlar uchun barcha xizmatlarning Qo'g'irchoq katalogini tuzish bir necha daqiqa vaqt oladi. Bu hali ham meni ayblashmoqda, garchi men hamma narsa qanday ishlashini va nima uchun bunchalik uzoq davom etishini bir necha bor tushuntirib berganman.
Uchinchidan, SnowFlakes to'plami bor - umumiy tizimdan ajralib turadigan narsalar, chunki ularda maxsus narsa bor, shuning uchun umumiy qoidalar ularga taalluqli emas. Bu frontal hujum bilan hal qilindi - agar signallar mavjud bo'lsa, lekin aslida hamma narsa tartibda bo'lsa, men chuqurroq qazishim va nima uchun bu meni ogohlantirayotganini tushunishim kerak, garchi bunday bo'lmasa ham. Yoki aksincha - nima uchun Nagios vahima qiladi, lekin Icinga emas.
To'rtinchidan, Nagios mendan oldin bu erda uch yil ishladi va dastlab mening yangi hipster tizimimdan ko'ra unga ko'proq ishonch bor edi, shuning uchun Icinga har safar vahima qo'zg'atganida, Nagios xuddi shu masaladan hayajonlanmaguncha hech kim hech narsa qilmadi. Ammo juda kamdan-kam hollarda Icinga Nagiosdan oldin haqiqiy signallarni yaratdi va men buni "Xulosalar" bo'limida gaplashadigan jiddiy muammo deb bilaman.
Natijada, foydalanishga topshirish 5 oydan ko'proq vaqtga kechiktirildi (28 yil 2018 iyunga rejalashtirilgan, aslida - 3 yil 2018 dekabr), asosan "paritet tekshiruvi" tufayli - Nagiosda hech kim bilmaydigan bir nechta xizmatlar mavjud bo'lgan axlat. Men so'nggi ikki yil davomida hech narsa eshitmadim, lekin HOZIR ular hech qanday sababsiz tanqid qilishdi va men nima uchun ular mening panelimda yo'qligini tushuntirishim kerak edi va ularni Icinga-ga qo'shishim kerak edi, shunda "paritet tekshiruvi" tugallangan" (Nagiosdagi barcha xizmatlar/cheklar Icinga-dagi xizmatlar/cheklarga mos keladi)
Amalga oshirish:
Birinchisi, Qo'g'irchoq uslubi kabi Code vs Data urushi. Barcha ma'lumotlar, mutlaqo hamma narsa Hiera'da bo'lishi kerak va boshqa hech narsa yo'q. Barcha kodlar .pp fayllarida. O'zgaruvchilar, abstraktsiyalar, funktsiyalar - hamma narsa pp.da ketadi.
Natijada, bizda bir qator virtual mashinalar (yozish vaqtida 165 ta) va SSL sertifikatlarining ishlashi va amal qilish muddati kuzatilishi kerak bo'lgan 68 ta veb-ilovalar mavjud. Ammo tarixiy gemorroy tufayli ilovalarni kuzatish uchun ma'lumotlar alohida gitlab omboridan olinadi va ma'lumotlar formati Qo'g'irchoq 3 dan beri o'zgarmadi, bu esa konfiguratsiyada qo'shimcha murakkablikni keltirib chiqaradi.
Ilovalar uchun qo'g'irchoq-kod, ko'zlaringizni himoya qiling
define profiles::services::monitoring::docker_apps(
Hash $app_list,
Hash $apps_accessible_from,
Hash $apps_access_list,
Hash $webhost_defaults,
Hash $webcheck_defaults,
Hash $service_overrides,
Hash $targets,
Hash $app_checks,
)
{
#### APPS ####
$zone = $name
$app_list.each | String $app_name, Hash $app_data |
{
$notify_group = { 'notify_group' => ($webcheck_defaults[$zone]['notify_group'] + pick($app_data['notify_group'], {} )) } # adds notifications for default group (systems) + any group defined in int/pm_docker_apps.eyaml
$data = merge($webhost_defaults, $apps_accessible_from, $app_data)
$site_domain = $app_data['site_domain']
$regexp = pick($app_data['check_regex'], 'html') # Pick a regex to check
$check_url = $app_data['check_url'] ? {
undef => { 'http_uri' => '/' },
default => { 'http_uri' => $app_data['check_url'] }
}
$check_regex = $regexp ?{
'absent' => {},
default => {'http_expect_body_regex' => $regexp}
}
$site_domain.each | String $vhost, Hash $vdata | { # Split an app by domains if there are two or more
$vhost_name = {'http_vhost' => $vhost}
$vars = $data['vars'] + $vhost_name + $check_regex + $check_url
$web_ipaddress = is_array($vdata['web_ipaddress']) ? { # Make IP-address an array if it's not, because askizzy has 2 ips and it's an array
true => $vdata['web_ipaddress'],
false => [$vdata['web_ipaddress']],
}
$access_from_zones = [$zone] + $apps_access_list[$data['accessible_from']] # Merge default zone (where the app is defined) and extra zones if they exist
$web_ipaddress.each | String $ip_address | { # For each IP (if we have multiple)
$suffix = length($web_ipaddress) ? { # If we have more than one - add IP as a suffix to this hostname to avoid duplicating resources
1 => '',
default => "_${ip_address}"
}
$octets = split($ip_address, '.')
$ip_tag = "${octets[2]}.${octets[3]}" # Using last octet only causes a collision between nginx-vip 203.15.70.94 and ext. ip 49.255.194.94
$access_from_zones.each | $zone_prefix |{
$zone_target = $targets[$zone_prefix]
$nginx_vip_name = "${zone_prefix}_nginx-vip-${ip_tag}" # If it's a host for ext - prefix becomes 'ext_' (ext_nginx-vip...)
$nginx_host_vip = {
$nginx_vip_name => {
ensure => present,
target => $zone_target,
address => $ip_address,
check_command => 'hostalive',
groups => ['nginx_vip',],
}
}
$ssl_vars = $app_checks['ssl']
$regex_vars = $app_checks['http'] + $vars + $webcheck_defaults[$zone] + $notify_group
if !defined( Profiles::Services::Monitoring::Host[$nginx_vip_name] ) {
ensure_resources('profiles::services::monitoring::host', $nginx_host_vip)
}
if !defined( Icinga2::Object::Service["${nginx_vip_name}_ssl"] ) {
icinga2::object::service {"${nginx_vip_name}_ssl":
ensure => $data['ensure'],
assign => ["host.name == $nginx_vip_name",],
groups => ['webchecks',],
check_command => 'ssl',
check_interval => $service_overrides['ssl']['check_interval'],
target => $targets['services'],
apply => true,
vars => $ssl_vars
}
}
if $regexp != 'absent'{
if !defined(Icinga2::Object::Service["${vhost}${$suffix} regex"]){
icinga2::object::service {"${vhost}${$suffix} regex":
ensure => $data['ensure'],
assign => ["match(*_nginx-vip-${ip_tag}, host.name)",],
groups => ['webchecks',],
check_command => 'http',
check_interval => $service_overrides['regex']['check_interval'],
target => $targets['services'],
enable_flapping => true,
apply => true,
vars => $regex_vars
}
}
}
}
}
}
}
}Xostlar va xizmatlar uchun konfiguratsiya kodi ham dahshatli ko'rinadi:
monitoring/config.pp
class profiles::services::monitoring::config(
Array $default_config,
Array $hostgroups,
Hash $hosts = {},
Hash $host_defaults,
Hash $services,
Hash $service_defaults,
Hash $service_overrides,
Hash $webcheck_defaults,
Hash $servicegroups,
String $servicegroup_target,
Hash $user_defaults,
Hash $users,
Hash $oncall,
Hash $usergroup_defaults,
Hash $usergroups,
Hash $notifications,
Hash $notification_defaults,
Hash $notification_commands,
Hash $timeperiods,
Hash $webhost_defaults,
Hash $apps_access_list,
Hash $check_commands,
Hash $hosts_api = {},
Hash $targets = {},
Hash $host_api_defaults = {},
)
{
# Profiles::Services::Monitoring::Hostgroup <<| |>> # will be enabled when we move to icinga completely
#### APPS ####
case $location {
'int', 'ext': {
$apps_by_zone = {}
}
'pm': {
$int_apps = hiera('int_docker_apps')
$int_app_defaults = hiera('int_docker_app_common')
$st_apps = hiera('staging_docker_apps')
$srs_apps = hiera('pm_docker_apps_srs')
$pm_apps = hiera('pm_docker_apps') + $st_apps + $srs_apps
$pm_app_defaults = hiera('pm_docker_app_common')
$apps_by_zone = {
'int' => $int_apps,
'pm' => $pm_apps,
}
$app_access_by_zone = {
'int' => {'accessible_from' => $int_app_defaults['accessible_from']},
'pm' => {'accessible_from' => $pm_app_defaults['accessible_from']},
}
}
default: {
fail('Please ensure the node has $location fact set (int, pm, ext)')
}
}
file { '/etc/icinga2/conf.d/':
ensure => directory,
recurse => true,
purge => true,
owner => 'icinga',
group => 'icinga',
mode => '0750',
notify => Service['icinga2'],
}
$default_config.each | String $file_name |{
file {"/etc/icinga2/conf.d/${file_name}":
ensure => present,
source => "puppet:///modules/profiles/services/monitoring/default_config/${file_name}",
owner => 'icinga',
group => 'icinga',
mode => '0640',
}
}
$app_checks = {
'ssl' => $services['webchecks']['checks']['ssl']['vars'],
'http' => $services['webchecks']['checks']['http_regexp']['vars']
}
$apps_by_zone.each | String $zone, Hash $app_list | {
profiles::services::monitoring::docker_apps{$zone:
app_list => $app_list,
apps_accessible_from => $app_access_by_zone[$zone],
apps_access_list => $apps_access_list,
webhost_defaults => $webhost_defaults,
webcheck_defaults => $webcheck_defaults,
service_overrides => $service_overrides,
targets => $targets,
app_checks => $app_checks,
}
}
#### HOSTS ####
# Profiles::Services::Monitoring::Host <<| |>> # This is for spaceship invasion when it's ready.
$hosts_has_large_disks = query_nodes('mountpoints.*.size_bytes >= 1099511627776')
$hosts.each | String $hostgroup, Hash $list_of_hosts_with_settings | { # Splitting site lists by hostgroups - docker_host/gluster_host/etc
$list_of_hosts_in_group = $list_of_hosts_with_settings['hosts']
$hostgroup_settings = $list_of_hosts_with_settings['settings']
$merged_hostgroup_settings = deep_merge($host_defaults, $list_of_hosts_with_settings['settings'])
$list_of_hosts_in_group.each | String $host_name, Hash $host_settings |{ # Splitting grouplists by hosts
# Is this host in the array $hosts_has_large_disks ? If so set host.vars.has_large_disks
if ( $hosts_has_large_disks.reduce(false) | $found, $value| { ( $value =~ "^${host_name}" ) or $found } ) {
$vars_has_large_disks = { 'has_large_disks' => true }
} else {
$vars_has_large_disks = {}
}
$host_data = deep_merge($merged_hostgroup_settings, $host_settings)
$hostgroup_settings_vars = pick($hostgroup_settings['vars'], {})
$host_settings_vars = pick($host_settings['vars'], {})
$host_notify_group = delete_undef_values($host_defaults['vars']['notify_group'] + $hostgroup_settings_vars['notify_group'] + $host_settings_vars['notify_group'])
$host_data_vars = delete_undef_values(deep_merge($host_data['vars'] , {'notify_group' => $host_notify_group}, $vars_has_large_disks)) # Merging vars separately
$hostgroups = delete_undef_values([$hostgroup] + $host_data['groups'])
profiles::services::monitoring::host{$host_name:
ensure => $host_data['ensure'],
display_name => $host_data['display_name'],
address => $host_data['address'],
groups => $hostgroups,
target => $host_data['target'],
check_command => $host_data['check_command'],
check_interval => $host_data['check_interval'],
max_check_attempts => $host_data['max_check_attempts'],
vars => $host_data_vars,
template => $host_data['template'],
}
}
}
if !empty($hosts_api){ # All hosts managed by API
$hosts_api.each | String $zone, Hash $hosts_api_zone | { # Split api hosts by zones
$hosts_api_zone.each | String $hostgroup, Hash $list_of_hosts_with_settings | { # Splitting site lists by hostgroups - docker_host/gluster_host/etc
$list_of_hosts_in_group = $list_of_hosts_with_settings['hosts']
$hostgroup_settings = $list_of_hosts_with_settings['settings']
$merged_hostgroup_settings = deep_merge($host_api_defaults, $list_of_hosts_with_settings['settings'])
$list_of_hosts_in_group.each | String $host_name, Hash $host_settings |{ # Splitting grouplists by hosts
# Is this host in the array $hosts_has_large_disks ? If so set host.vars.has_large_disks
if ( $hosts_has_large_disks.reduce(false) | $found, $value| { ( $value =~ "^${host_name}" ) or $found } ) {
$vars_has_large_disks = { 'has_large_disks' => true }
} else {
$vars_has_large_disks = {}
}
$host_data = deep_merge($merged_hostgroup_settings, $host_settings)
$hostgroup_settings_vars = pick($hostgroup_settings['vars'], {})
$host_settings_vars = pick($host_settings['vars'], {})
$host_api_notify_group = delete_undef_values($host_defaults['vars']['notify_group'] + $hostgroup_settings_vars['notify_group'] + $host_settings_vars['notify_group'])
$host_data_vars = delete_undef_values(deep_merge($host_data['vars'] , {'notify_group' => $host_api_notify_group}, $vars_has_large_disks))
$hostgroups = delete_undef_values([$hostgroup] + $host_data['groups'])
if defined(Profiles::Services::Monitoring::Host[$host_name]){
$hostname = "${host_name}_from_${zone}"
}
else
{
$hostname = $host_name
}
profiles::services::monitoring::host{$hostname:
ensure => $host_data['ensure'],
display_name => $host_data['display_name'],
address => $host_data['address'],
groups => $hostgroups,
target => "${host_data['target_base']}/${zone}/hosts.conf",
check_command => $host_data['check_command'],
check_interval => $host_data['check_interval'],
max_check_attempts => $host_data['max_check_attempts'],
vars => $host_data_vars,
template => $host_data['template'],
}
}
}
}
}
#### END OF HOSTS ####
#### SERVICES ####
$services.each | String $service_group, Hash $s_list |{ # Service_group and list of services in that group
$service_list = $s_list['checks'] # List of actual checks, separately from SG settings
$service_list.each | String $service_name, Hash $data |{
$merged_defaults = merge($service_defaults, $s_list['settings']) # global service defaults + service group defaults
$merged_data = merge($merged_defaults, $data)
$settings_vars = pick($s_list['settings']['vars'], {})
$this_service_vars = pick($data['vars'], {})
$all_service_vars = delete_undef_values($service_defaults['vars'] + $settings_vars + $this_service_vars)
# If we override default check_timeout, but not nrpe_timeout, make nrpe_timeout the same as check_timeout
if ( $merged_data['check_timeout'] and ! $this_service_vars['nrpe_timeout'] ) {
# NB: Icinga will convert 1m to 60 automatically!
$nrpe = { 'nrpe_timeout' => $merged_data['check_timeout'] }
} else {
$nrpe = {}
}
# By default we use nrpe and all commands are run via nrpe. So vars.nrpe_command = $service_name is a default value
# If it's server-side Icinga command - we don't need 'nrpe_command'
# but there is no harm to have that var and the code is shorter
if $merged_data['check_command'] == 'nrpe'{
$check_command = $merged_data['vars']['nrpe_command'] ? {
undef => { 'nrpe_command' => $service_name },
default => { 'nrpe_command' => $merged_data['vars']['nrpe_command'] }
}
}else{
$check_command = {}
}
# Assembling $vars from Global Default service settings, servicegroup settings, this particular check settings and let's not forget nrpe settings.
if $all_service_vars['graphite_template'] {
$graphite_template = {'check_command' => $all_service_vars['graphite_template']}
}else{
$graphite_template = {'check_command' => $service_name}
}
$service_notify = [] + pick($settings_vars['notify_group'], []) + pick($this_service_vars['notify_group'], []) # pick is required everywhere, otherwise becomes "The value '' cannot be converted to Numeric"
$service_notify_group = $service_notify ? {
[] => $service_defaults['vars']['notify_group'],
default => $service_notify
} # Assing default group (systems) if no other groups are defined
$vars = $all_service_vars + $nrpe + $check_command + $graphite_template + {'notify_group' => $service_notify_group}
# This needs to be merged separately, because merging it as part of MERGED_DATA overwrites arrays instead of merging them, so we lose some "assign" and "ignore" values
$assign = delete_undef_values($service_defaults['assign'] + $s_list['settings']['assign'] + $data['assign'])
$ignore = delete_undef_values($service_defaults['ignore'] + $s_list['settings']['ignore'] + $data['ignore'])
icinga2::object::service {$service_name:
ensure => $merged_data['ensure'],
apply => $merged_data['apply'],
enable_flapping => $merged_data['enable_flapping'],
assign => $assign,
ignore => $ignore,
groups => [$service_group],
check_command => $merged_data['check_command'],
check_interval => $merged_data['check_interval'],
check_timeout => $merged_data['check_timeout'],
check_period => $merged_data['check_period'],
display_name => $merged_data['display_name'],
event_command => $merged_data['event_command'],
retry_interval => $merged_data['retry_interval'],
max_check_attempts => $merged_data['max_check_attempts'],
target => $merged_data['target'],
vars => $vars,
template => $merged_data['template'],
}
}
}
#### END OF SERVICES ####
#### OTHER BORING STUFF ####
$servicegroups.each | $servicegroup, $description |{
icinga2::object::servicegroup{ $servicegroup:
target => $servicegroup_target,
display_name => $description
}
}
$hostgroups.each| String $hostgroup |{
profiles::services::monitoring::hostgroup { $hostgroup:}
}
$notifications.each | String $name, Hash $settings |{
$assign = pick($notification_defaults['assign'], []) + $settings['assign']
$ignore = pick($notification_defaults['ignore'], []) + $settings['ignore']
$merged_settings = $settings + $notification_defaults
icinga2::object::notification{$name:
target => $merged_settings['target'],
apply => $merged_settings['apply'],
apply_target => $merged_settings['apply_target'],
command => $merged_settings['command'],
interval => $merged_settings['interval'],
states => $merged_settings['states'],
types => $merged_settings['types'],
assign => delete_undef_values($assign),
ignore => delete_undef_values($ignore),
user_groups => $merged_settings['user_groups'],
period => $merged_settings['period'],
vars => $merged_settings['vars'],
}
}
# Merging notification settings for users with other settings
$users_oncall = deep_merge($users, $oncall)
# Magic. Do not touch.
create_resources('icinga2::object::user', $users_oncall, $user_defaults)
create_resources('icinga2::object::usergroup', $usergroups, $usergroup_defaults)
create_resources('icinga2::object::timeperiod',$timeperiods)
create_resources('icinga2::object::checkcommand', $check_commands)
create_resources('icinga2::object::notificationcommand', $notification_commands)
profiles::services::sudoers { 'icinga_runs_ping_l2':
ensure => present,
sudoersd_template => 'profiles/os/redhat/centos7/sudoers/icinga.erb',
}
}Men hali ham bu noodle ustida ishlayapman va ularni imkon qadar yaxshilashga harakat qilaman. Biroq, bu kod bizga Hiera-da oddiy va tushunarli sintaksisdan foydalanishga imkon berdi:
ma'lumotlar
profiles::services::monitoring::config::services:
perf_checks:
settings:
check_interval: '2m'
assign:
- 'host.vars.type == linux'
checks:
procs: {}
load: {}
memory: {}
disk:
check_interval: '5m'
vars:
notification_period: '24x7'
disk_iops:
vars:
notifications:
- 'silent'
cpu:
vars:
notifications:
- 'silent'
dns_fqdn:
check_interval: '15m'
ignore:
- 'xenserver in host.groups'
vars:
notifications:
- 'silent'
iftraffic_nrpe:
vars:
notifications:
- 'silent'
logging:
settings:
assign:
- 'logserver in host.groups'
checks:
rsyslog: {}
nginx_limit_req_other: {}
nginx_limit_req_s2s: {}
nginx_limit_req_s2x: {}
nginx_limit_req_srs: {}
logstash: {}
logstash_api:
vars:
notifications:
- 'silent'Barcha tekshiruvlar guruhlarga bo'lingan, har bir guruhda bu tekshiruvlarni qayerda va qanchalik tez-tez o'tkazish, qanday bildirishnomalarni va kimga yuborish kabi standart sozlamalar mavjud.
Har bir tekshiruvda siz istalgan variantni bekor qilishingiz mumkin va bularning barchasi oxir-oqibatda barcha tekshiruvlarning standart sozlamalariga qo'shiladi. Shuning uchun bunday bema'nilik config.pp da yozilgan - u barcha standart sozlamalarni guruh sozlamalari bilan, keyin esa har bir alohida tekshirish bilan birlashtiradi.
Yana bir juda muhim o'zgarish sozlamalardagi funksiyalardan foydalanish imkoniyati bo'ldi, masalan, http_regexni tekshirish uchun port, manzil va urlni almashtirish funksiyasi.
http_regexp:
assign:
- 'host.vars.http_regex'
- 'static_sites in host.groups'
check_command: 'http'
check_interval: '1m'
retry_interval: '20s'
max_check_attempts: 6
http_port: '{{ if(host.vars.http_port) { return host.vars.http_port } else { return 443 } }}'
vars:
notification_period: 'host.vars.notification_period'
http_vhost: '{{ if(host.vars.http_vhost) { return host.vars.http_vhost } else { return host.name } }}'
http_ssl: '{{ if(host.vars.http_ssl) { return false } else { return true } }}'
http_expect_body_regex: 'host.vars.http_regex'
http_uri: '{{ if(host.vars.http_uri) { return host.vars.http_uri } else { return "/" } }}'
http_onredirect: 'follow'
http_warn_time: 8
http_critical_time: 15
http_timeout: 30
http_sni: trueBu degani - agar xost ta'rifida o'zgaruvchi mavjud bo'lsa http_port — undan foydalaning, aks holda 443. Masalan, jabber veb-interfeysi 9090 da, Unifi esa 7443 da osilgan.
http_vhost DNS-ga e'tibor bermaslik va ushbu manzilni olishni anglatadi.
Agar uri xostda ko'rsatilgan bo'lsa, unga amal qiling, aks holda "/" ni oling.
http_ssl bilan kulgili hikoya bor edi - bu infektsiya talab bo'yicha o'chirishni xohlamadi. Xost ta'rifidagi o'zgaruvchi ekanligini tushunmagunimcha, men bu qator haqida uzoq vaqt chalkashdim:
http_ssl: falseIfodaga almashtirilgan
if(host.vars.http_ssl) { return false } else { return true }qanday yolg'on va oxirida bu chiqadi
if(false) { return false } else { return true }ya'ni ssl tekshiruvi doimo faol bo'ladi. Men buni sintaksisni almashtirish orqali hal qildim:
http_ssl: notopilmalar:
Taroziga soling:
- Hozir bizda oxirgi 7-8 oy davomida bo'lgani kabi ikkita emas, bitta monitoring tizimi mavjud yoki biri eskirgan va zaif.
- Xostlar/xizmatlar (tekshiradi) ma'lumotlar tuzilmasi endi (mening fikrimcha) ancha o'qilishi va tushunarli. Boshqalar uchun bu unchalik aniq emas edi, shuning uchun hammasi qanday ishlashini va nimani tahrirlash kerakligini tushuntirish uchun mahalliy wiki-da bir nechta sahifalarni joylashtirishim kerak edi.
- O'zgaruvchilar va funktsiyalar yordamida cheklarni moslashuvchan tarzda sozlash mumkin, masalan, http_regexp ni tekshirish uchun xost sozlamalarida kerakli naqsh, qaytish kodi, url va portni o'rnatish mumkin.
- Bir nechta asboblar paneli mavjud, ularning har biri uchun siz o'zingizning ko'rsatilgan signallar ro'yxatini belgilashingiz va bularning barchasini Qo'g'irchoq va birlashtirish so'rovlari orqali boshqarishingiz mumkin.
Kamchiliklari:
- Jamoa a'zolarining inertsiyasi - Nagios ishladi, ishladi va ishladi va sizning bu Isinga doimo nosozlik va sekin. Bu erda tarixni qanday ko'rish mumkin? Oh, jin ursin, u yangilanmagan... (Haqiqiy muammo shundaki, signal tarixi avtomatik ravishda yangilanmaydi, faqat F5 tomonidan)
- Tizimning inertsiyasi - veb-interfeysda "yangilash" tugmasini bosganimda (hozir tekshiring) - bajarish natijasi Marsdagi ob-havoga, ayniqsa, o'nlab soniyalarni talab qiladigan murakkab xizmatlarga bog'liq. Bunday natija normaldir.
- Umuman olganda, ikki tizimning yonma-yon ishlashining olti oylik statistik ma'lumotlariga ko'ra, Nagios har doim Icinga'dan tezroq ishlagan va bu meni juda g'azablantirdi. Menimcha, ular taymerlar bilan nimanidir chalkashtirib yuborishgan va tekshirish har besh daqiqada amalga oshiriladi, aslida bu har 5:30 yoki shunga o'xshash narsada sodir bo'ladi.
- Agar istalgan vaqtda xizmatni qayta ishga tushirsangiz (systemctl restart icinga2) - o'sha paytda amalga oshirilgan barcha tekshiruvlar muhim signalni keltirib chiqaradi. ekranda va tashqaridan hamma narsa tushib ketgandek ko'rinadi ().
Ammo umuman olganda, u ishlaydi.
Manba: www.habr.com
