How I wrote my monitoring

I decided to share my story. It may even help someone with a budget-friendly solution to a well-known problem.

When I was young and full of energy, I decided to do some freelancing. I quickly managed to build up my rating and found a couple of regular clients who asked me to maintain their servers on an ongoing basis.

The first thing that came to my mind was the need for monitoring. I decided to be smart and not reinvent the wheel, but to look at ready-made options like Munin or Zabbix. However, it quickly became clear that the web version requires a good internet connection, especially if it’s being accessed for the first time from a phone. If you're out in nature away from the city, getting a stable connection can be quite difficult. Therefore, I opted for a console-based monitoring solution.

For console monitoring, atop and the log reading tool atopsar worked well for me. They were previously mentioned on habr, atop was even dissected, but almost nothing was said about atopsar.

Installation

Very simple installation, just three commands.

#Centos

yum install atop

#Debian/Ubuntu

apt-get install atop

Next, you can configure the monitoring to your liking or use the default settings.

#Debian/Ubuntu/Centos

/etc/default/atop 

Standard file:

 #cat /etc/default/atop
INTERVAL=60                    #Время, через которое создаётся снимок нагрузки в секундах, по умолчанию каждые 10 минут
LOGPATH="/var/log/atop"        #Путь до папки хранения логов
OUTFILE="$LOGPATH/daily.log"   #Название файла логов за сегодняшний день

Add to autostart
#Debian/Ubuntu/Centos

systemctl enable atop 

Start atop as a daemon
#Debian/Ubuntu/Centos

systemctl start atop  

For the lazy, I compiled it into one command
#Centos

yum install atop && systemctl enable atop && systemctl start atop

#Debian/Ubuntu

apt-get install atop && systemctl enable atop && systemctl start atop

Atopsar

Along with atop, atopsar is installed, which is a convenient console log analyzer for the binary logs maintained by the atop daemon. Of course, you can read the logs directly with atop, but it isn’t as convenient if you need to capture a long time interval.

A brief guide to working with atopsar.

When starting atopsar without any keys, the log for today is opened and the load on each core is displayed separately, along with the idl line for all cores.

The keys I use:

-A = to display all the information from the log
-c = to display information about CPU core loads, default key
-m = memory and swap load
-d = disk activity
-O = top 3 processes by CPU load
-G = top 3 processes by RAM load
-D = top 3 processes by disk load
-N = top 3 processes by network load
-r = specify the path to the log you want to read, if you want to check the load from previous days
-b = the time from which to start the output
-e = the time by which the output must be completed
-M = creates an additional column at the end that marks the criticality of the line (+ there is a load, * — critical load)

Thanks to monitoring, we can understand the cause of incorrect server behavior at any time.

Notifications

So, we have load monitoring, but it still doesn’t allow us to promptly find and solve problems. We need notifications about any issues that arise.

I’m the only one monitoring the servers, so notifications need to go to where I can always see them and react somehow.

At first, there were SMS — quick, reliable, free. But then mobile operators shut down free SMS delivery through their gateways.
Email — slow, there can be delivery issues.
Messengers — need to be installed on the phone, you have to create bots.

As a result of the search, the messenger Telegram was chosen for its simplicity and convenient application on mobile and desktop.

I created my bot with the help of botfather.
After that, I placed several scripts on the server to monitor the server load (IDL, smartct, etc.), the presence of errors like 'oom killer', errors during backup creation, and other operations that need to be controlled.

The scripts are quite simple, written in bash; for example, checking LA and notifying about exceeding the Load Average compared to the number of cores on server.

if [ ${LA[0]} -gt 2000 ] || [ ${LA[1]} -gt 3000 ] || [ ${LA[2]} -gt 4000 ]
    then
        wget -O /dev/null "https://api.telegram.org/$bot_id:$bot_key/sendMessage?chat_id=$chat_id&text=On server $ip LA $LAd"
        wget -O /dev/null "https://api.telegram.org/$bot_id:$bot_key/sendMessage?chat_id=$chat_id&text=`top -b -n 1 | grep Cpu`"
        wget -O /dev/null "https://api.telegram.org/$bot_id:$bot_key/sendMessage?chat_id=$chat_id&text=Top 5 processes `top -b -n 1 | grep -A 5 'PID USER' | tail -5`"
    fi

The simplicity of the syntax offers many usage options (and anyone with a bit of programming knowledge can write or modify it).

One caveat — if the server is located in Russia (and you don’t have IPv6 on the server), you need to use a proxy. For this, at the beginning of the script, you need to specify the proxy connection string:

export https_proxy=http://login:password@IP.address:port

This is not the end

You're hiking peacefully through the mountains with a backpack, enjoying a break from civilization, when your phone, unexpectedly catching a signal, sends a notification about a problem on your server. What to do? That serene mood is blown away in an instant. Call your wife and dictate commands? Ha-ha!

I urgently needed to come up with a way to resolve the issues quickly without a good internet connection. This is where a messenger (#telegramlives) saved me again. I taught my bot to communicate only with me, ignoring everyone else. Now, along with the problem notification, I receive a bit more data to identify the source of the issue and can try to resolve it remotely. It's as simple as sending a message to the bot, lifting the phone a bit higher to get that message through, and voilà—the bot starts doing your job. This way, I can, for example, kill an unwanted process, restart a daemon, block an IP, and more.

I've also transferred future necessary client requests here, such as urgent password resets for users (because "Aaaah, we can't access the server, we are losing millions!"), finding a user with access to a specific folder, enabling and disabling the site, and others. Of course, I continuously improve the bot's functionality, as clients sometimes come up with unexpected and unanticipated requests. But the basics are satisfied.

There is also a version for VK, but it hasn't caught on somehow.

Now I can travel peacefully and explore this world, without fearing that something might break while I'm away and I won't be able to find out or fix it.

Source: habr.com