The Banana Pi 64 is a single board computer similar to the Raspberry Pi, but with multiple Ethernet ports, making it a general-purpose Linux distribution router.
Yes, Openwrt already exists, but it has its own GUI and CLI; there is Mikrotik, but again it has its own GUI / CLI, and Wireguard does not work out of the box ... In general, I want a router with flexible settings, while remaining within the framework of standard Linux, which you work with every day.
In the article under the names BPI, R64, single-board, I will mean the same thing - the Banana Pi R64 single-board itself.
Image selection. Download via eMMC
The very first skill to acquire when working with in general, and with R64 in particular, it means learning how to load an OS into it and be able to interact with it, because R64 does not have a monitor port (HDMI, for example). When everything fell off - Wifi stopped working, Ethernet network, Bluetooth, USB, etc. there is a UART, through the interface of which you can always see what went wrong, and also run a couple of commands from the console, if necessary.
Connection algorithm to R64 via USB-UART:
- run to the radio parts store for a USB-UART cable (PL2303, Serial-to-USB)
- we connect one USB end to the computer, and the other, UART, to R64, with three out of four wires, as in the picture below
- in the computer console run
sudo minicom
After that, in most cases, the single-board console will appear = success.
More details can be seen .
Next, the easiest way is to load the operating system from the SD card: download by image and upload it:
unzip -p 2019-08-23-ubuntu-16.04-lite-preview-bpi-r64-sd-emmc.img.zip | pv | sudo dd of=/dev/mmcblk0 bs=10M status=noxferwe insert the card into the R64 SD-slot, turn it on, observe the loading of uboot first on the connected console, then the standard Linux boot.
An alternative boot option is using an 64Gb card already embedded in the R8, called eMMC. According to the instructions on the wiki, we rewrite the image to the device
/dev/mmcblk0 to BPI, reboot, remove SD card, enable BPI again... and it doesn't work. How to get there Boot select don't hold on.
The fact is that at least for BPI you need to set a special flag in order to be able to boot from an internal flash drive:
root@bpi-r64:~# ./mmc extcsd read /dev/mmcblk1 | grep 'PARTITION_CONFIG'
Boot configuration bytes [PARTITION_CONFIG: 0x00]
root@bpi-r64:~# ./mmc bootpart enable 1 1 /dev/mmcblk1
root@bpi-r64:~# ./mmc extcsd read /dev/mmcblk1 | grep 'PARTITION_CONFIG'
Boot configuration bytes [PARTITION_CONFIG: 0x48]Next, you need to write preloader to a special boot partition
root@bpi-r64:~# echo 0 > /sys/block/mmcblk0boot0/force_ro
root@bpi-r64:~# dd if=preloader_evb7622_64_foremmc.bin of=/dev/mmcblk0boot0R64 manufacturer (China) posted this binary . What he does is unknown (there are no source codes), but without him it will not work either.
In general, after that, the images begin to load from eMMC. If you want to figure it out and create images from scratch, then for both cases (SD / eMMC) you need to write a few more files (preloader for the SD card, ATF, u-boot), just to get to the kernel boot. This topic is still , but for us the main thing is that it works and okay.
Now, to be honest, I donβt use eMMC download, SD cards are enough, but I spent quite a lot of time making it work, so let it be in the article.
Choice of operating system. Armbian
The first application task is to launch a VPN, of course Wireguard. It immediately turned out that it was not assembled from the kernel side, and there were no headers. I rebuilt the kernel and, out of habit with x86, built the kernel module using DKMS. However, the assembly speed on arm64 of even small utilities unpleasantly surprised me. And then another kernel module was required, and so on. In general, it turns out that everything related to the kernel is better to be assembled on a warm-tube x86 laptop, then transferred to R64 by simple copying, rebooted and tested.
Another thing is the userspace part. In my case of choosing Debian, everything for the arm64 architecture is already on packages.debian.org and there is no need to rebuild anything.
In order not to produce another bicycle, I on BPI R64.
Or rather: the userspace part is Armbian, and the core is taken from the repository -A. The latest image can be downloaded .
All activity on the development of the software part of R64 is carried out on . Generally speaking, the manufacturer itself seeks to popularize the Openwrt router, but thanks to the activity of the developer Frank from Germany, all the features quickly end up in the Debian kernel. Surprisingly, Frank is active in every forum thread.
Workspace organization: wires
Separately, I want to tell you how to place the SBC (not just BPI) on the table during development / testing so as not to lead an Ethernet cable to it from the Internet source through the entire room / office. The fact is that, on the one hand, you need to provide the Internet to the piece of iron, and on the other hand, everything can break in this piece of iron, and especially Wifi.
First I decided to buy a cheap USB-Wifi "whistle", plug it into the only port on the BPI and forget about the wires. To do this, I purchased an inexpensive TP-LINK TL-WN725N USB 2.0, but very soon it became clear that it would not take off: for the whistle to work, a kernel driver was needed, which, of course, was not there (later I compiled the necessary RTL8XXXU driver, but itβs still impractical ). And the ethernet cable ruined the view of the room for a while.
As a result, I managed to get rid of the cable with the help of Tenda MW3 (Wifi mesh system): I simply placed one cube under the table and connected the BPI to the LAN port of the latter with a meter Ethernet cable. Success.
Wireguard, RKN, Bird
One of the things I use Banana PI for is to have free access to sites blocked by the RKN, in particular, so that Telegram and calls to Slack work. Articles on this topic have already been proposed on HabrΓ©: , , .
I implemented the deployment of just such a solution using Ansible: .
The VPS is supposed to be running Ubuntu 18.04. I checked the performance on two hosters in Europe: Amazon and Digital Ocean.
So, we installed the above Armbian on R64, it is available via ssh under the name hm-bananapi-1 and has internet access. We deploy sequentially ansible, automation scripts and run the installation itself on R64:
# Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠΈ Π΄Π»Ρ Debian-based Π΄ΠΈΡΡΡΠΈΠ±ΡΡΠΈΠ²ΠΎΠ²
$ sudo apt install --no-install-recommends python3-pip python3-setuptools python3-wheel git
$ which pip3
/usr/bin/pip3
# ansible Ρ pybook, ΡΠΊΡΠΈΠΏΡΠΎΠ²Π°Π½ΠΈΠ΅ Π½Π° Python
$ pip3 install https://github.com/muravjov/ansible/archive/ansible-2.10.0.dev0-pybook2019.tar.gz
$ export PATH=~/.local/bin:$PATH
$ which ansible-playbook
/home/sa/.local/bin/ansible-playbook
$ git clone https://github.com/muravjov/ansible-bpi-r64.git
$ cd ansible-bpi-r64
$ git submodule update --init
# ΡΠ±Π΅ΠΆΠ΄Π°Π΅ΠΌΡΡ Π² Π΄ΠΎΡΡΡΠΏΠ½ΠΎΡΡΠΈ hm-bananapi-1
$ ssh hm-bananapi-1 which python3
/usr/bin/python3
# ΡΠΎΠ±ΡΡΠ²Π΅Π½Π½ΠΎ ΡΡΡΠ°Π½ΠΎΠ²ΠΊΠ°
$ ansible-playbook ./router.py -l hm-bananapi-1Next, you need to deploy our VPN to the VPS in the same way:
ansible-playbook ./router.py -l current-vpnHere, the argument is always current-vpn, and the VPS name itself is configured in a variable (in this case, it is paris-vpn-aws-t2-micro-1):
$ grep current_vpn group_vars/all
current_vpn: paris-vpn-aws-t2-micro-1
#current_vpn: frankfurt-vpn-d0-starter-1Oh yes, before all these operations, you need to generate secrets (in particular, Wireguard keys) in a folder ./secrets, the directory should look like .
Ansible automation in Python
You may notice that instead of the YAML format, the Ansible commands are encoded in Python scripts. For comparison, how to enable the bird daemon in the usual way:
- name: start bird
systemd:
name: bird
state: started
enabled: yesand how the same via Python:
with mapping:
append("name", "start bird")
with mapping("systemd"):
append("name", "bird")
append("state", "started")
append("enabled", "yes")Writing Ansible commands with Python code allows you to reuse the code, and in general, all the possibilities of a general-purpose language are open. For example, installing bird on R64 and VPS:
install_bird("router/bird.conf.j2")
install_bird("vpn/bird.conf.j2")see function code .
This feature is called pybook implemented . There is no documentation on pybook yet, then I will correct this defect.
What does he think on this occasion.
Monitoring. Prometheus
Total: telegram works, linkedin and pornhub too, in general, user experience is ok. But everything can break, and Chinese pieces of iron too.
Kernel updates can also be interesting: for example, I wanted to update the kernel 5.4 => 5.6, well, there is Wireguard out of the box, no need to patch ... No sooner said than done: painstakingly transferred patches from 5.4 to 5.6, the kernel started, the tunnel to the VPS pinged, but bird cannot connect with error "BGP Error"β¦ "Rolled back in horror" (c) to 5.4; moving to 5.6 was postponed in TODO.
Therefore, in addition to installing the router and VPS, I added monitoring (on x86 Ubuntu 18.04), which is installed on a separate host with the following components:
- prometheus, alertmanager, blackbox_exporter - all in docker
- alerts are sent to the telegram channel using the metalmatze/alertmanager-bot bot - also in docker
- tor for the bot, so that the bot can alert situations when the Internet is available, but the telegram still does not work, and the bot itself cannot connect
- applied : NodeVPNTroubles (no ping to VPS), BirdVPNTroubles (no Bird session), AntifilterDownloadTroubles (failed to load blocked IP addresses), SiteTroubles (unfortunate telegram unavailable)
- system alerts like HostGrowingDiskReadLatency (cheap SD card stops being read)
Monitoring setup example:
ansible-playbook ./monitoring.py -l monitoring-preprodAuto Discovery for prometheus is set to the /etc/prometheus/auto_http folder, an example of adding a host to monitoring (hosts are not monitored by default):
bash << 'EOF'
HOSTNAME=hm-bananapi-1
IP_ADDRESS=`ssh -G $HOSTNAME | awk '/^hostname / { print $2 }'`
ssh monitoring-preprod sudo sponge /etc/prometheus/auto_http/$HOSTNAME.json << EOF2
[
{
"targets": ["$IP_ADDRESS:9100"],
"labels": {
"env": "prod",
"hostname": "$HOSTNAME"
}
}
]
EOF2
EOFTODO: 2 providers, 2 BPIs, anycast failover
In addition to everything, I planned to connect to two providers so that the Internet would continue to work, even if one provider had problems with the network, or they forgot to pay for the Internet, etc., and other human factors.
The most advanced user experience on the topic of multi-wan is described for Mwan3 system under Openwrt. This solution has rich functionality, but setting up and operating in general for multi-wan is rather troublesome. Just one example: if you come to some sites from two IP addresses at once, then they may not like it, they will stop working => "the Internet is not working."
Given this experience, I decided that multihoming is not a priority yet, only failover. Although it seems that in the latest versions of linux everything should work with one command like:
ip route add default
nexthop via 192.168.1.1 weight 10
nexthop via 192.168.2.1 weight 5So, so that there is no single point of failure, we take 2 BPIs, each connect to one provider, connect them together and make communication with each other by dynamic routing through bird / OSPF.
Further, on each we announce the same IP address if the service is available (Internet, DNS). That is, we will not put down the default route ourselves, but through bird. The solution spied .
This functionality has not yet been implemented, the insidious coronavirus messed up (not everything came from aliexpress; another online store, Layta, promised to deliver in a week, and more than a month has passed; the second provider did not manage to stretch the cable before quarantine, only managed to make a hole in wall to drill for the cable).
How to order R64
The board itself in the official store .
It is also better to order immediately:
- + inform EU or US plug standard
- heat sink: radiators/fans; because both the CPU is heated, and the switch chip
- wifi antenna,
There is a nuance - the price of delivery from some time has become inadequately high in the official store. Manager Judy Huang assured me that there was no mistake, and you can choose ePacket for $5, but I saw that for Russia there is only EMS for >33$. Unpleasant, but not critical. Moreover, if you choose any other country for delivery (I went through all the continents), delivery will cost ~5$. Russophobes?.. But then I found that for France the delivery price is also ~$30, and calmed down.
As a result, Judy offered to place an order, but not pay (hint: put less on the card so that the payment does not go through automatically); write to her and she will reduce the shipping price to normal. Success.
Issues
Not everything works perfectly yet.
Performance
Ansible=Python commands are executed slowly, even idle ones, for 20-30 seconds; an order of magnitude longer than on an x86 laptop. Moreover, at first they are performed quite quickly, ~ 3 seconds, then they slow down sharply. Perhaps this is due to the heating of the CPU (throttling). The Go code is also long running:
# Π·Π°ΠΏΡΠΎΡ ΠΌΠ΅ΡΡΠΈΠΊ Π΄Π»Ρ ΠΏΡΠΎΠΌΠ΅ΡΠ΅Ρ ΠΈΠ· node_exporter Π½Π° Go
$ time curl -s http://172.30.1.1:9100/metrics > /dev/null
real 0m6,118s
user 0m0,005s
sys 0m0,009s
# ΠΎΠ΄Π½Π°ΠΊΠΎ ΡΠ΅ΠΌΠΏΠ΅ΡΠ°ΡΡΡΠ° 51 Π³ΡΠ°Π΄ΡΡ, Π½Π΅ ΡΠ°ΠΊ ΠΈ ΠΌΠ½ΠΎΠ³ΠΎ
sa@bananapir64:~$ cat /sys/devices/virtual/thermal/thermal_zone0/temp
51700Wi-Fi
Wifi works, but stops on Armbian after about a day, writes:
sa@bananapir64:~$ dmesg | grep -E 'mt7622_wmac.*timeout'
[470303.802539] mt7622_wmac 18000000.wmac: Message 38 (seq 3) timeout
[470314.042508] mt7622_wmac 18000000.wmac: Message 50 (seq 4) timeout
...Only restart helps. We need to go further .
Ethernet
Ethernet works, but after ~ a day packets (DHCP) from R64 stop coming.
Restarting the interface helps:
ifdown br0; sleep 30; ifup br0The driver is new, the kernel has not yet been accepted, I hope the Chinese Landen Chao .
Source: habr.com