Remote monitoring and control of Linux/OpenWrt/Lede devices via port 80, continued

This is the final part of the article, here is the beginning habr.com/en/post/445568
Last time I wrote about how I implemented device monitoring, now we will talk about management. In discussions with "techies" on the part of the Customer, I often meet with a limited perception of the capabilities of such small devices (with low memory resources and performance), many believe that "the maximum that we need is to send a reboot, for something more serious - we will send a brigade" .
But practice shows that this is not entirely true. Here is a small list of common typical tasks:

  1. Network diagnostics and troubleshooting. Behind the ethernet port of your router, another piece of iron usually “lives”, which has its own internal ip-address. Sometimes, it can (should) be “pinged”. Or tunnel management - if the tunnel suddenly does not rise on the router working through the 3G modem, but we see the router itself.
  2. System service. Firmware update, service scripts upgrade.
  3. Equilibristics. This could be called "perversions", but the concept of "tightrope walker" as, I quote, "the ability of a circus performer to maintain balance in an unstable body position" - fits better. Such situations arise due to the limited budget of the customer. I gave a couple of examples below, but since they are not directly related to the topic of the story, I put them in the notes

wifi monitoringA fashionable topic for the last five years, mainly among federal retail chains. You are slowly walking around the trading floors, and your mobile phone with Wi-Fi turned on, in an attempt to “stick” to some thread of the network, regularly sends out Probe Request packets that can be analyzed in order to calculate for you: how often do you come to this store, for what trajectories you walk and so on. Further, the data is collected, analyzed, heat maps are drawn, and managers “knock out” money from management or investors for such pictures. In the meantime .... “there is no money, but you hold on ...”, and the result (real) should already be shown, the good old song “Yes, yes, then of course we will supply ciscos and whatever you want, but now we need to show the Customer the result! By the way, they forgot to say that the Customer allowed our equipment to be connected to his hotspot via Wi-Fi, but on a general basis, just as if we were guest clients. And now you have to make equilibrist routers - several WiFi subinterfaces rise, one of which it clings to the hotspot, and the second monitors the environment, frantically unloads the result of tcpdump into itself, then packs the contents of the file into an archive and risks dying from "overeating" trying to spit out the contents to the ftp server. It is not surprising that the equilibrist router often “breaks down” and somehow has to be “reanimated” remotely.

RadiusHere it is easier to describe the situation with something like this statement of the customer: “We want a decentralized network of hotspots that would work on equipment whose model is not known in advance, through channels, but which ones we do not yet know. Oh, we forgot to say, we not only want to show ads to customers, but also analyze everything around the hotspot installation site. No, we don’t know why yet, but we’ll come up with it, don’t hesitate, we were able to come up with this idea. ”

And we must not forget that due to a lot of uncertain circumstances in advance, management must be carried out in non-standard conditions, when we cannot connect to the router directly via ip: port and are forced to simply wait for activity to appear from it. If we abstract, then the dialogue between the server and the router can be represented like this:

  • Router: Hello. I am such and such a router, are there any tasks for me?
  • Server: such and such a router, I registered you that you are alive. Here's the challenge: show me the output of the ifconfig command?
  • Router: Hello. I am such and such a router, last time you asked to show the result of ifconfig, here it is. Are there tasks for me?
  • Server: such and such a router, I registered you that you are alive. There are no tasks for you.

The most interesting question is: how can a remote router send a certain amount of information? In the last part, I described that the router, due to limited resources, has only a “stripped down” wget, which works only through GET and nothing else, there is no ftp client or curl. More precisely, we need a universal way, regardless of the features of the image assembly. I settled on using wget. More precisely, how “stopped” - I just had no choice 🙂

Immediately reservationMy management solution is working, not very limited, and I'm sure it's crooked, even if it suits most of my customers. How WOULD it be possible to do it wisely - write a small utility that sends binary data via POST through the 80th port. Include it (utility) in the firmware of the router and use bash to access it. But the reality is that: a) you need to quickly b) you probably need to do everything on the existing "zoo of routers" c) "do no harm!" - if the router is working and performing other tasks, try to make changes that will not affect the existing functionality.

Let's move on to the implementation. Let's say your customer wants from zabbix to reboot the router easily and naturally, with a "mouse click". Today we will begin the description of the implementation with zabbix.
In the menu "Administration" -> "Scripts" add a new script. We call it "Reboot", as a command we write "php /usr/share/zabbix/reboot.php {HOST.HOST}"

Remote monitoring and control of Linux/OpenWrt/Lede devices via port 80, continued

Next: Menu "Monitoring" -> "Latest data" -> "Right-click on the desired host". This is how the menu will look after adding the script.

Remote monitoring and control of Linux/OpenWrt/Lede devices via port 80, continued
Accordingly, we put the reboot.php script in the /usr/share/zabbix directory (it may be different for you, I use the zabbixa root directory).

Safety disclaimerFor clarity of explanation in the script, I use only the id of the router, but do not use the password. In the working version, this is not recommended! Why did I do this: because the big question is where to store passwords for routers? In zabbixe itself in "inventory"? Contradictory practice. As an option: restrict external access to the reboot.php file itself

reboot.php file

<?php
	// присваиваем параметры с консоли переменным
	$user = $argv[1];
	// ВНИМАНИЕ. Вот здесь в целях безопасности все-таки прописывать пароль устройства! Но для демонстрации мы будем обращаться к базе данных без использования пароля. 
	//$password = $argv[2];
		
	$conn=new mysqli("localhost","db_user","db_password","db_name");
	if (mysqli_connect_errno()) {
		exit();
	}
	$conn->set_charset("utf8");
			
	// "Отправляем" команду reboot за счет изменения поля task таблицы users. В поле task можно отправлять любую команду.
	$sql_users=$conn->prepare("UPDATE users SET task='reboot' WHERE id=? AND status='active';");
	$sql_users->bind_param('s', $user);
	$sql_users->execute();
	$sql_users->close();
?>

Actually everything. The question “how to get the result of the command execution from the device side” remains open. Let's consider the task using the ifconfig command as an example. The following command can be sent to the device:

message=`ifconfig`; wget "http://xn--80abgfbdwanb2akugdrd3a2e5gsbj.xn--p1ai/a.php?u=user&p=password!&m=$message" -O /tmp/out.txt

, where:
message=`ifconfig` - we assign the result of the output of the ifconfig command to the $message variable
wget "xn--80abgfbdwanb2akugdrd3a2e5gsbj.xn--p1ai/a.php - our a.php script that registers routers and receives messages from them
u=user&p=password!&m=$message - credentials and the value of the query variable m - assigns the contents of the $message variable
-O /tmp/out.txt - we do not need output to the /tmp/out.txt file in this case, but if this parameter is not specified, wget does not work

Why does this work wrongBecause it's a potential security hole. the most innocuous mistake that can happen is if the output of your command, for example, contains the symbol "&". Therefore, it is necessary to filter everything that is sent from the routers and everything that comes to the server. Yeah, I'm ashamed, really. In my defense, I can only write that the entire article is devoted to how to manage routers with undefined firmware, with undefined communication channels.

Well, I’ve touched on the future: I haven’t figured out yet how to reflect the results (for example, the result of a command execution) that come to the server using standard zabbix tools.

I remind you that all sources can be taken from the Git repository at: github.com/BazDen/iotnet.online.git

Source: habr.com

Add a comment