ProHoster > Blog > Administration > Bot for monitoring web services in half an hour: telegram + bash + cron

Bot for monitoring web services in half an hour: telegram + bash + cron

Bot for monitoring web services in half an hour: telegram + bash + cron

Sometimes you need to quickly make monitoring for a new service, but there is no ready-made infrastructure / expertise at hand. In this guide, we will implement a tool for monitoring any web services in half an hour, using only the built-in ubuntu tools: bash, cron and curl. We will use telegram to deliver notifications.

The icing on the cake will be the emotional engagement of users. Tested on people - it works.

When we created a chatbot in the telemedicine service Doctor Nearby to determine the level of stress of users, we needed monitoring. In a couple of hours, a mini-project was made, which not only works great, but also adds positivity with its messages.

First, let's get a repository with scripts:

git clone https://github.com/rshekhovtsov/msms.git

Go to the msms folder and then work in it.

If telegram is blocked, use a proxy. The simplest and most reliable option is torsocks:

sudo apt install tor
sudo apt install torsocks

As an example, let's set up monitoring of the google.com start page in three steps

STEP 1. Create a bot in telegram and get the user id

  • In the search bar for contacts in telegram, we are looking for @botfather:

    Bot for monitoring web services in half an hour: telegram + bash + cron

  • We launch it with the Start button, enter the /newbot command and answer the questions. Keep in mind that name is the name of the bot that will be displayed to users, while username is unique and must end with "bot":

    Bot for monitoring web services in half an hour: telegram + bash + cron

    Among other things, the bot will issue a secret token for the HTTP API, which you need to copy and save to the telegram-api-key.txt file in the msms folder.

  • We type the name of our bot in the telegram search bar, and run it.
  • As a finishing touch, let's add ourselves to the list of monitoring notification recipients:
    sudo chmod +x ./recipients-setup.sh
torsocks ./recipients-setup.sh

    The script will display a list of recent calls to the bot, there should be one line with our id and name in telegram. We take this id and save it in the services/google-recipients.txt file. File format: each line is one id. Example:

    123456789
987654321

To add a new recipient, you need to ask him to start the bot in telegram, run recipients-setup.sh and add the id to the file.

STEP 2. Set up monitoring

The service is described by creating an ini-file in the services folder. You need to set five parameters:

  1. MSMS_SERVICE_NAME: service name - will be used in alerts and monitoring log.
  2. MSMS_SERVICE_ENDPOINT: endpoint of the service that we will call with curl.
  3. MSMS_CURL_PARAMS: additional options for curl, see example below.
  4. MSMS_EXPECTED: expected service response. Used when the answer is short.
  5. MSMS_EXPECTED_FILE: filename with expected service response. If specified, overwrites MSMS_EXPECTED.
  6. MSMS_RECIPIENTS: a file with a list of notification recipients.

A request to google.com returns a fixed html with a redirect, we will use it as the expected server response:

curl google.com > services/google-response.html

Let's create a services/google.ini file:

MSMS_SERVICE_NAME='google front page'

# service endpoint
MSMS_SERVICE_ENDPOINT='google.com'

# curl parameters
MSMS_CURL_PARAMS='-s --connect-timeout 3 -m 7'

# expected service response
MSMS_EXPECTED_FILE='google-response.html'

# recipients list file
MSMS_RECIPIENTS='google-recipients.txt'

В MSMS_CURL_PARAMS you can set everything that curl can do, including:

  1. Disable curl messages so as not to litter the console and log: -s
  2. Set connection timeout with the checked service (in seconds): --connect-timeout 3
  3. Set response timeout: -m 7
  4. Disable certificate verification for SSL (for example, if a self-signed certificate is used): --insecure
  5. Specify http request type: -X POST
  6. Specify headers: -H "Content-Type: application/json"
  7. Specify the request body as a string or file. Example for a file: -d @request.json

We disabled alerts and set timeouts to 3 seconds. per connection and 7 sec. to receive a response from the service.

Attention: specify parameter values ​​in single quotes, as in the example. Unfortunately, bash is rather fragile in this sense, and an accidentally flown butterfly in the wrong quote can lead to the death of the universe with hard-to-diagnose errors.

We set up monitoring. Let's check that everything is OK:

sudo chmod +x ./monitoring.sh
torsocks ./monitoring.sh

The script should display a message like:

2020-01-10 12:14:31
health-check "google front page": OK

STEP 3. Set up the schedule

Set up a monitoring schedule in cron:

sudo crontab -e

Let's add a line to check google.com every minute:

*/1 * * * * torsocks <ПУТЬ К ПАПКЕ РЕПОЗИТОРИЯ>/monitoring.sh >> <ПУТЬ К ПАПКЕ РЕПОЗИТОРИЯ>/monitoring.log 2>&1

Let's add a notification every day at 11.00, confirming the performance of the monitoring itself. To do this, pass the DAILY parameter to the script:

0 11 * * * torsocks <ПУТЬ К ПАПКЕ РЕПОЗИТОРИЯ>/monitoring.sh DAILY >> <ПУТЬ К ПАПКЕ РЕПОЗИТОРИЯ>/monitoring.log 2>&1

2>&1 is a standard trick that redirects errors to the main output stream. As a result, they will also be included in the monitoring log.

Save the changes and pick them up with the command:

 sudo service cron reload

You can read more about setting up cron, for example, here.

Thus, every minute a monitoring script will be launched, which will contact google.com via curl. If the received response differs from the expected one, the script will send a notification to the list of recipients in telegram. The check log is kept in the monitoring.log file

If we need to add another service, we simply create a new ini file for it in the services folder and, if necessary, form a separate list of recipients. Everything else will work automatically.

If the service being checked becomes unavailable, a notification will be sent every minute. If you cannot quickly restore the service, you can temporarily disable notifications in the properties of the bot in telegram.

Now let's take a closer look at additional features and script implementation.

Message Templates and Emotional Engagement

To make communication with the bot more lively, we named it Manechka, added an appropriate avatar picture, and engaged professional PR people to create message texts. You can use our developments or change to your taste.

For example:

Bot for monitoring web services in half an hour: telegram + bash + cron
or even like this:

Bot for monitoring web services in half an hour: telegram + bash + cron
Why not?

The bot name and avatar are set via @botfather.
The message templates are in the folder customizable templates:

  • curl-fail.txt: message sent when curl returns a non-zero error code. Usually indicates the impossibility of reaching the service.
  • daily.txt: Daily message confirming that the service monitoring is working.
  • service-fail.txt: The message sent when the service's response is different than expected.

Let's analyze the possibilities of customization using the built-in message templates as an example.
The templates use emoji. Unfortunately, habr does not display them.
To select emoji, it is convenient to use the search on emojipedia.org:

Bot for monitoring web services in half an hour: telegram + bash + cron

You simply copy and paste the appropriate character into the template text (this is regular unicode).

  1. curl-fail.txt:
    Котёнок,  помоги мне... 
Не могу достучаться до сервиса "$MSMS_SERVICE_NAME" 
`CURL EXIT CODE: $EXIT_CODE`

    We used the service name we specified (variable MSMS_SERVICE_NAME) and an internal script variable with curl exit code (EXIT_CODE). We also formatted the message using the markup telegram markdown: "`" characters surround fixed-width text. Since quotation marks and apostrophes are bash special characters, we escape them with "". Variable names are preceded by the "$" sign.

    Result:

    Bot for monitoring web services in half an hour: telegram + bash + cron

  2. service-fail.txt:
    Котёнок, помоги мне... 
Сервис "$MSMS_SERVICE_NAME" меня расстроил
Он работает неправильно, вот что он мне отвечает:
`$RESPONSE`

    Result:

    Bot for monitoring web services in half an hour: telegram + bash + cron

    Here we use another script variable: RESPONSE. It contains the response of the service.

  3. daily.txt:
    Малыш, привет!
У меня всё хорошо, cлежу за сервисом:
"$MSMS_SERVICE_NAME" каждую минутку...
А как у тебя дела?

    Result:

    Bot for monitoring web services in half an hour: telegram + bash + cron

Let's move on to the implementation of scripts.

Monitoring script

monitoring.sh makes a simple auto-discovery - takes all ini files from the services folder and executes the main script for each with the logic for checking and sending alerts:

#!/bin/bash
cd $(dirname "$0")/services

for service_ini  in $(ls *.ini); do
    bash ../msms.sh "$1" "$service_ini"
done

To generate a daily message about the monitoring status, the DAILY parameter can be passed to the script.

Please note that when the script starts, the current folder changes to services. This allows ini files to specify file paths relative to services.

Script for checking and sending alerts

msms.sh contains the main logic for checking the service and sending alerts.

Working with telegram:

# telegram endpoint
TG_API_URL="https://api.telegram.org/bot$(cat ../telegram-api-key.txt)/sendMessage"

#################################################################
# send message to telegram
# parameter: message text
#################################################################
function send_message {
    for chat_id  in $(cat ../$MSMS_RECIPIENTS); do
	curl -s -X POST --connect-timeout 10 $TG_API_URL -d chat_id=$chat_id -d parse_mode="Markdown" -d text="$1"
	echo
    done
}

We generate a URL to access the telegram REST API using the secret key stored in the file.

The send_message function uses curl to send messages to this REST API, taking the id of the recipients from the file we specified in the ini. In the sent data, we indicate that we use message markup: parse_mode="Markdown".

Let's display the current date-time and load the ini-file.

echo $(date '+%Y-%m-%d %H:%M:%S')

# load variables from .ini file:
. $2

Magic string . $2 executes the ini-file passed as the second parameter as a normal script, adding the values ​​specified in it to the environment variables.

Load the expected response from the file if the parameter is given MSMS_EXPECTED_FILE:

if [ -n "$MSMS_EXPECTED_FILE" ]; then
 MSMS_EXPECTED="$(cat "$MSMS_EXPECTED_FILE")"
fi

Let's check the service with sending notifications, if necessary:

RESPONSE="$(eval curl $MSMS_CURL_PARAMS "$MSMS_SERVICE_ENDPOINT")"
EXIT_CODE=$?
if [[ $EXIT_CODE != 0 ]]; then
    echo health-check "$MSMS_SERVICE_NAME" FAILED: CURL EXIT WITH $EXIT_CODE
    MESSAGE="$(cat ../templates/curl-fail.txt)"
    MESSAGE=$(eval echo $MESSAGE)
    send_message "$MESSAGE"
elif [[ "$RESPONSE" != "$MSMS_EXPECTED" ]]; then
    echo health-check "$MSMS_SERVICE_NAME" FAILED: "$RESPONSE"
    MESSAGE="$(cat ../templates/service-fail.txt)"
    MESSAGE=$(eval echo $MESSAGE)
    send_message "$MESSAGE"
else
    echo health-check "$MSMS_SERVICE_NAME": OK
fi

First assign to a variable RESPONSE the result of executing the curl command for the given service.

Expression EXIT_CODE=$? puts the result of the last command execution into a variable, i.e. curl. If it is necessary to send a notification, the template is read from the corresponding file and sent to recipients using send_message.

The last block handles the DAILY parameter:

if test "$1" = "DAILY"; then
    echo health-check "$MSMS_SERVICE_NAME" DAILY
    MESSAGE="$(cat ../templates/daily.txt)"
    MESSAGE=$(eval echo $MESSAGE)
    send_message "$MESSAGE"
fi

It sends a message confirming the health of the monitoring itself.

Getting a list of user ids

recipients-setup.sh calls the telegram API to get the latest messages addressed to the bot:

curl -s https://api.telegram.org/bot$(cat telegram-api-key.txt)/getUpdates 
| python recipients-setup.py

This is where the magic of python is used to make a pretty list output. This is optional, you can just take the desired id from the json that the command will display:

torsocks curl -s https://api.telegram.org/bot$(cat telegram-api-key.txt)/getUpdates

Conclusion

Thus, you can use ready-made scripts and message templates, setting up only observable services and lists for notifications; you can create a new "personality" for the bot; and you can make your own decision based on the proposed.

As options for further development, it suggests configuring and managing monitoring in the bot itself, but here you can’t do without python. If someone gets their hands on it before me, you know where to upload a pull request 🙂

Source: habr.com

Add a comment