Task History
On the one hand, small-sized companies need high-quality monitoring of their infrastructure (especially in the light of widespread virtualization), on the other hand, it is financially difficult for them to purchase new equipment. Also often there are problems with the server / hardware: often there are 1-3 tower-servers next to user workstations or in a small niche / closet.
It is easier to use a ready-made assembly (distribution kit), which is enough to upload to a microSD card and insert into a common single-board computer (beaglebone, raspberry pi and orange pi families, asus tinker board). In addition, such equipment is inexpensive and can be installed anywhere.
Formulation of the problem
In many ways, the project developed as a kind of laboratory work with the possibility of applying the results.
Zabbix was chosen as the monitoring system, as it is a powerful, free and well-documented system.
The question arose with the hardware platform. Putting a separate machine under monitoring is also not a good solution - either it is expensive to purchase new equipment, or to look for old equipment + in small firms there are frequent problems with the server / hardware.
Using the buildroot build system allows you to create customized solutions that can be operated by staff with minimal knowledge of the Linux family of OSes. This system is beginner-friendly, but at the same time gives ample opportunities for customization in the hands of an experienced developer. It is perfect for solving the problem of not expensive, but full-featured monitoring of the IT infrastructure, which requires minimal training for the personnel operating it.
Solution Steps
It was decided to initially create a firmware for x86_64 to run in qemu, since this is a convenient and fast debugging solution. Then port it to an arm single-board computer (I liked the asus tinker board).
buildroot was chosen as the build system. Initially, it lacks the zabbix package, so I had to port it. There were problems with the Russian locale, which were resolved by applying the appropriate patches (note: in newer buildroot versions, these patches are no longer needed).
Porting the zabbix package itself will be described in a separate article.
Since everything should work like a firmware (immutable system image + recoverable configuration files / databases), it was necessary to write our own systemd targets, services and timers (target, service, timer).
It was decided to split the media into 2 sections - a section with system files and a section with changeable configs and zabbix database files.
It turned out to be a little more difficult to solve problems related to the database. There was little desire to place it directly on the carrier. At the same time, the size of the database can reach a size that exceeds the size of a possible ramdisk. Therefore, a compromise solution was chosen: the database is located on the second partition of the sd card (modern SLC cards have up to 30 write cycles), but there is a setting that allows you to use external media (for example, usb-hdd).
Temperature monitoring was implemented through the RODOS-5 device. Of course, you can use the dallas 1820 directly, but it was faster and easier to plug in USB.
grub86 was chosen as the bootloader for x64_2. It was required to write a minimal config to run.
After debugging on qemu, porting to the asus tinker board was performed. In the structure of my overlay, cross-platform was initially laid down - the allocation of configs specific to each board (board defconfig, bootloader, image generation with a system partition) and maximum monotony in the file system setup / creating an image with data. In view of this preparation, porting went quickly.
It is highly recommended to read the introductory articles:
How to assemble
After cloning the repository, the following file structure is obtained:
[alexey@comp monitor]$ ls -1
buildroot-2019.05.tar.gz
overlay
README.md
run_me.shbuildroot-2019.05.tar.gz - clean buildroot archive
overlay is my external-tree directory. It is in it that everything you need is stored to assemble the firmware using buildroot
README.md - description of the project and manual in English.
run_me.sh is a script that prepares the build system. Expands the buildroot from the archive, attaches an overlay to it (via the external-tree mechanism) and allows you to select the target board for the build
[0] my_asus_tinker_defconfig
[1] my_beaglebone_defconfig
[2] x86_64_defconfig
Select defconfig, press A for abort. Default [0]After that, just go to the buildroot-2019.05 directory and run the make command.
After the build is complete, all build results will be in the output/images directory:
[alexey@comp buildroot-2019.05]$ ls -1 output/images/
boot.img
boot.vfat
bzImage
data
data.img
external.img
external.qcow2
grub-eltorito.img
grub.img
intel-ucode
monitor-0.9-beta.tar.gz
qemu.qcow2
rootfs.cpio
sdcard.img
sys
updateRequired files:
- sdcard.img - media image for writing to the sd card (via dd or rufus under wibdows).
- qemu.qcow2 - media image to run in qemu.
- external.qcow2 - external media image for the database
- monitor-0.9-beta.tar.gz - archive for updating via the web interface
Manual generation
It is not worth writing the same instruction several times. And the most logical thing is to write it once in markdown, and then convert it to PDF for download and html for the web interface. This is possible thanks to the pandoc package.
At the same time, you need to generate all these files before the system image is assembled, those post-build scripts are already useless. Therefore, the generation is made in the form of a package manuals. You can see it in overlay/package/manuals.
The manuals.mk file (which does all the work)
################################################################################
#
# manuals
#
################################################################################
MANUALS_VERSION:= 1.0.0
MANUALS_SITE:= ${BR2_EXTERNAL_monitorOverlay_PATH}/package/manuals
MANUALS_SITE_METHOD:=local
define MANUALS_BUILD_CMDS
pandoc -s -o ${TARGET_DIR}/var/www/manual_en.pdf ${BR2_EXTERNAL_monitorOverlay_PATH}/../README.md
pandoc -f markdown -t html -o ${TARGET_DIR}/var/www/manual_en.html ${BR2_EXTERNAL_monitorOverlay_PATH}/../README.md
endef
$(eval $(generic-package))systemd
The Linux world is actively moving to systemd, I had to do it too.
Of the pleasant innovations - the presence of timers. In general, a separate article is being written about them (and not only about them), but I will briefly talk about them.
There are actions that need to be performed periodically. I needed to run logrotate to clean up the lighttpd and php-fpm logs. The most common would be to write commands in cron, but I decided to use systemd's monotonous timer. So logrotate runs after a strict time interval.
Of course, it is possible to create timers that fire on certain dates, but I did not need it.
Timer example:
- Timer File
[Unit] Description=RODOS temp daemon timer
OnUnitActiveSec=1min [Install] WantedBy=timers.target
- Файл сервиса, вызываемого таймером:
```bash
[Unit]
Description=RODOS temp daemon
[Service]
ExecStart=/usr/bin/rodos.shSupported boards
Asus tinker board is the main board on which everything should work. Chosen as inexpensive and very powerful.
Beaglebone black is the first board on which the work was tested (during the selection of a more powerful board).
Qemu x86_64 - Used for debug development.
How does work
At startup, there is a two-stage restoration of settings:
- launching the settings_restore script (via a service). It restores the basic system settings - time zone, locale, network settings, etc.
- launching the prepare script (via the service) - zabbix, the database is being prepared here, the IP is displayed in the console.
At the first start, the size of the second section of the sd card is determined. If there is still unallocated space, the media is repartitioned, the data partition takes up all the free space. This is done in order to reduce the size of the installation image (sdcard.img). Also, the postgresql working directory is created at this point. That is why the first launch with a new media will be longer than subsequent ones.
When connecting an external drive, at the time of startup it looks for a free drive and formats it in ext4 with the label external.
Attention! When connecting an external drive (as well as disconnecting or replacing it), you need to backup and restore the settings!
The RODOS 5 device is used to monitor the temperature. The manufacturer provides the source codes of his utility for working with the device. When you turn on the system, the rodos timer starts, which runs this utility once a minute. The current temperature is written to the /tmp/rodos_current_temp file, after which zabbix can monitor this file as a sensor.
The configuration storage media is mounted to the /data directory.
When the system starts and prepares it for operation, the following message appears in the console:
System starting, please waitAfter the preparatory work is completed, it will change to the output of the IP address:
current ip 192.168.1.32
Ready to workSetting up zabbix for temperature monitoring
To monitor the temperature, it is enough to take 2 steps:
- connect RODOS device to usb port
- create data item in zabbix
Open the zabbix web interface:
- Open the section Configuration → Hosts
- Click on Items in the line of our zabbix server
- Click on Create item
Enter the following data:
- name - up to you (for example, serverRoomTemp )
- Type - zabbix agent
- Key - rodos
- Type-numeric
- Units-C
- History storage period - history storage period. left 10 days
- Trend storage period — storage period for the dynamics of changes. Left 30 days
- New application-server Room Temp
And press the ADD button.
Manage settings via web interface
The web interface is written in php. There are main functions:
- view device status
- changing network settings
- user password change
- time zone selection
- backup/restore/factory reset
- the ability to connect an external drive
- System update
Login to the web interface is protected by a password. Start page - manual.
zabbix interface address: ${ip/dns}/zabbix
Management interface address: ${ip/dns}/manage
Running in qemu
qemu-system-x86_64 -smp 4 -m 4026M -enable-kvm -machine q35,accel=kvm -device intel-iommu -cpu host -net nic -net bridge,br=bridge0 -device virtio-scsi-pci,id= scsi0 -drive file=output/images/qemu.qcow2,format=qcow2,aio=threads -device virtio-scsi-pci,id=scsi0 -drive file=output/images/external.qcow2,format=qcow2,aio=threads
This command will start a system with 4 cores, 2048 RAM, KVM enabled, bridge0 network card and two disks: for the system and external for postgresql.
Images can be converted and run in Virtualbox:
qemu-img convert -f qcow2 qemu.qcow2 -O vdi qcow2.vdi
qemu-img convert -f qcow2 external.qcow2 -O vdi external.vdiThen import them into virtualbox and connect via sata.
Conclusion
In the process, it became interesting for me to make a product ready for work - with a not very beautiful interface (I don’t like to write them), but it works and is easy to set up.
The last attempt to install zabbix-appliance in KVM showed the correctness of this step (after the installation is completed, the system does not start). Maybe I'm doing something wrong 😉
Source: habr.com