I would like to share my experience of combining networks in three geographically distant apartments, each of which uses routers with OpenWRT as a gateway, into one common network. When choosing a method for combining networks between L3 with subnet routing and L2 with bridging, when all network nodes will be in the same subnet, preference was given to the second method, which is more difficult to configure, but provides more opportunities, since transparent use of technologies was planned in the created network Wake-on-Lan and DLNA.
Part 1: Background
The protocol chosen to implement this task was initially OpenVPN, because, firstly, it can create a tap device that can be added to the bridge without any problems, and secondly, OpenVPN It supports TCP, which was also important, as none of the apartments had a dedicated IP address. I couldn't use STUN because my ISP, for some reason, blocks incoming UDP connections from its networks. TCP allowed me to forward the VPN server port to the rented VPS using SSH. While this approach creates a significant overhead, as the data is double-encrypted, I didn't want to integrate the VPS into my private network, as there was a risk of third parties gaining control over it. Therefore, having such a device on my home network was highly undesirable, so I decided to pay a significant overhead for security.
To forward the port on the router where the server was planned to be deployed, I used the sshtunnel program. I won't go into the details of its configuration—it's quite easy. I'll just note that its purpose was to forward TCP port 1194 from the router to the VPS. Next, I configured the server. OpenVPN On the tap0 device, which was connected to the br-lan bridge. After testing the connection to the newly created server from my laptop, it became clear that the port forwarding idea had worked, and my laptop had become a member of the router's network, even though it wasn't physically part of it.
The only thing left to do was to distribute IP addresses in different apartments so that they would not conflict and configure the routers as OpenVPN-clients.
The following router IP addresses and DHCP server ranges were selected:
- 192.168.10.1 with range 192.168.10.2 192.168.10.80 for the server
- 192.168.10.100 with range 192.168.10.101 192.168.10.149 for a router in apartment No. 2
- 192.168.10.150 with range 192.168.10.151 192.168.10.199 for a router in apartment No. 3
It was also necessary to assign these addresses to the client routers. OpenVPN-server, by adding the following line to its configuration:
ifconfig-pool-persist /etc/openvpn/ipp.txt 0and adding the following lines to the /etc/openvpn/ipp.txt file:
flat1_id 192.168.10.100
flat2_id 192.168.10.150
where flat1_id and flat2_id are the device names specified when creating certificates for connecting to OpenVPN
Next, the routers were configured OpenVPN- clients, tap0 devices on both were added to the br-lan bridge. At this point, everything seemed fine, as all three networks could see each other and function as a single unit. However, a rather unpleasant detail emerged: sometimes devices would receive an IP address from the wrong router, with all the ensuing consequences. For some reason, the router in one of the apartments failed to respond to DHCPDISCOVER in time, and the device received the wrong address. I realized I needed to filter such requests in tap0 on each router, but as it turned out, iptables can't work with a device if it's part of a bridge, so I needed to use ebtables. Unfortunately, my firmware didn't include it, so I had to rebuild the images for each device. After doing this and adding the following lines to /etc/rc.local on each router, the problem was solved:
ebtables -A INPUT --in-interface tap0 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
ebtables -A INPUT --in-interface tap0 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
ebtables -A FORWARD --out-interface tap0 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
ebtables -A FORWARD --out-interface tap0 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
This configuration lasted for three years.
Part 2: Getting to Know WireGuard
Lately, there has been increasing talk on the Internet about WireGuard, admiring its ease of configuration, high transfer speed, low ping, and comparable security. A search for additional information about it revealed that it doesn't support either bridge member or TCP protocol support, which led me to believe there was no alternative. OpenVPN for me it's still not there. So I put off getting to know WireGuard.
A few days ago, news spread through resources related to IT in one way or another that WireGuard will finally be included in the kernel Linux, starting with version 5.6. News articles, as always, were praised WireGuardI once again plunged into searching for ways to replace the good old OpenVPNThis time I ran into . It talked about creating an Ethernet tunnel over L3 using GRE. This article gave me hope. It remained unclear what to do with the UDP protocol. Searching led me to articles about using socat in conjunction with an SSH tunnel to forward a UDP port, however, they noted that this approach only works in single connection mode, which means that multiple VPN clients would be impossible. I came up with the idea to set up a VPN server on a VPS, and configure GRE for clients, but as it turned out, GRE does not support encryption, which will lead to the fact that if third parties gain access to the server, all traffic between my networks is in their hands which didn't suit me at all.
Again, the decision was made in favor of redundant encryption, by using VPN over VPN according to the following scheme:
Layer XNUMX VPN:
VPS is server with internal address 192.168.30.1
Elemental mastery is client VPS with internal address 192.168.30.2
MK2 is client VPS with internal address 192.168.30.3
MK3 is client VPS with internal address 192.168.30.4
Layer XNUMX VPN:
Elemental mastery is server with external address 192.168.30.2 and internal 192.168.31.1
MK2 is client Elemental mastery with the address 192.168.30.2 and has an internal IP of 192.168.31.2
MK3 is client Elemental mastery with the address 192.168.30.2 and has an internal IP of 192.168.31.3
* Elemental mastery - router-server in apartment 1, MK2 - router in apartment 2, MK3 - router in apartment 3
* Device configurations are published in the spoiler at the end of the article.
And so, pings between the nodes of the network 192.168.31.0/24 go, it's time to move on to setting up the GRE tunnel. Before that, in order not to lose access to routers, it is worth setting up SSH tunnels to forward port 22 to the VPS, so that, for example, a router from apartment 10022 will be available on port 2 of the VPS, and a router from apartment 11122 will be available on port 3 of the VPS. router from apartment XNUMX. It is best to configure the forwarding with the same sshtunnel, since it will restore the tunnel in case it falls.
The tunnel is configured, you can connect to SSH through the forwarded port:
ssh root@МОЙ_VPS -p 10022Next you should disable OpenVPN:
/etc/init.d/openvpn stopNow let's set up a GRE tunnel on the router from apartment 2:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.2
ip link set grelan0 up
And add the created interface to the bridge:
brctl addif br-lan grelan0
Let's perform a similar procedure on the server router:
ip link add grelan0 type gretap remote 192.168.31.2 local 192.168.31.1
ip link set grelan0 up
And, also, add the created interface to the bridge:
brctl addif br-lan grelan0
starting from this moment, pings begin to successfully go to the new network and I, with satisfaction, go to drink coffee. Then, to see how the network on the other end of the wire works, I try to SSH into one of the computers in apartment 2, but the ssh client freezes without prompting me for a password. I try to connect to this computer via telnet on port 22 and see a line from which you can understand that the connection is being established, the SSH server is responding, but for some reason it does not offer me to enter.
$ telnet 192.168.10.110 22
SSH-2.0-OpenSSH_8.1
I'm trying to connect to it via VNC and I see a black screen. I convince myself that the matter is in the remote computer, because I can easily connect to the router from this apartment using the internal address. However, I decide to SSH into this computer through the router and am surprised to find that the connection succeeds and the remote computer works fine but fails to connect to my computer either.
I take the grelan0 device out of the bridge and run it OpenVPN On the router in apartment 2, I confirmed that the network was working properly again and connections weren't dropping. Searching, I came across forums where people were complaining about the same issues, and where they were advised to raise the MTU. No sooner said than done. However, until the MTU was set high enough—7000 for gretap devices—I experienced either dropped TCP connections or low transfer speeds. Because of the high MTU for gretap, the MTU for connections WireGuard The first and second levels were set at 8000 and 7500 respectively.
I did a similar setup on the router from apartment 3, with the only difference being that a second gretap interface named grelan1 was added to the server router, which was also added to the br-lan bridge.
Everything is working. Now you can put the gretap assembly into autoload. For this:
Placed these lines in /etc/rc.local on the router in apartment 2:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.2
ip link set dev grelan0 mtu 7000
ip link set grelan0 up
brctl addif br-lan grelan0
Added this to /etc/rc.local on the router in apartment 3:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.3
ip link set dev grelan0 mtu 7000
ip link set grelan0 up
brctl addif br-lan grelan0
And on the server router:
ip link add grelan0 type gretap remote 192.168.31.2 local 192.168.31.1
ip link set dev grelan0 mtu 7000
ip link set grelan0 up
brctl addif br-lan grelan0
ip link add grelan1 type gretap remote 192.168.31.3 local 192.168.31.1
ip link set dev grelan1 mtu 7000
ip link set grelan1 up
brctl addif br-lan grelan1
After rebooting the client routers, I discovered that for some reason they weren't connecting to the server. After connecting to their SSH (thankfully, I'd previously configured sshtunnel for this), I discovered that WireGuard For some reason, it creates a route for the endpoint, but it's incorrect. For example, for 192.168.30.2, the route table specified a route through the pppoe-wan interface, i.e., through the internet, although the route to it should have been directed through the wg0 interface. After deleting this route, the connection was restored. Can I find instructions anywhere on how to force WireGuard I couldn't avoid creating these routes. Moreover, I didn't even understand whether this was a feature of OpenWRT or of the WireGuardWithout spending much time figuring out the problem, I simply added a line to the timer-loop script on both routers that deleted this route:
route del 192.168.30.2
Summing up
Complete rejection OpenVPN I haven't achieved this yet, as I occasionally need to connect to a new network from a laptop or phone, and setting up a gretap device on them is generally impossible. However, despite this, I've gained an advantage in data transfer speed between apartments, and using VNC, for example, is now hassle-free. Ping has decreased slightly but become more stable:
While using the OpenVPN:
[r0ck3r@desktop ~]$ ping -c 20 192.168.10.110
PING 192.168.10.110 (192.168.10.110) 56(84) bytes of data.
64 bytes from 192.168.10.110: icmp_seq=1 ttl=64 time=133 ms
...
64 bytes from 192.168.10.110: icmp_seq=20 ttl=64 time=125 ms
--- 192.168.10.110 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19006ms
rtt min/avg/max/mdev = 124.722/126.152/136.907/3.065 ms
While using the WireGuard:
[r0ck3r@desktop ~]$ ping -c 20 192.168.10.110
PING 192.168.10.110 (192.168.10.110) 56(84) bytes of data.
64 bytes from 192.168.10.110: icmp_seq=1 ttl=64 time=124 ms
...
64 bytes from 192.168.10.110: icmp_seq=20 ttl=64 time=124 ms
--- 192.168.10.110 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19003ms
rtt min/avg/max/mdev = 123.954/124.423/126.708/0.675 ms
It is mostly affected by high ping to VPS which is approximately 61.5ms
However, the speed has increased significantly. So, in the apartment with the router-server, I have an internet connection speed of 30 Mbps, and in the other apartments it's 5 Mbps. Moreover, during use OpenVPN I was unable to achieve a data transfer speed between networks greater than 3,8 Mbps according to iperf readings, while WireGuard "pumped" it up to the same 5 Mbit/sec.
Configuration WireGuard on VPS[Interface]
Address = 192.168.30.1/24
ListenPort = 51820
PrivateKey = <ЗАКРЫТЫЙ_КЛЮЧ_ДЛЯ_VPS>
[Peer]
PublicKey = <VPN_1_MS_PUBLIC_KEY>
AllowedIPs = 192.168.30.2/32
[Peer]
PublicKey = <VPN_2_MK2_PUBLIC_KEY>
AllowedIPs = 192.168.30.3/32
[Peer]
PublicKey = <VPN_2_MK3_PUBLIC_KEY>
AllowedIPs = 192.168.30.4/32
Configuration WireGuard on MS (added to /etc/config/network)
#VPN первого уровня - клиент
config interface 'wg0'
option proto 'wireguard'
list addresses '192.168.30.2/24'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_1_МС'
option auto '1'
option mtu '8000'
config wireguard_wg0
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_1_VPS'
option endpoint_port '51820'
option route_allowed_ips '1'
option persistent_keepalive '25'
list allowed_ips '192.168.30.0/24'
option endpoint_host 'IP_АДРЕС_VPS'
#VPN второго уровня - сервер
config interface 'wg1'
option proto 'wireguard'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_2_МС'
option listen_port '51821'
list addresses '192.168.31.1/24'
option auto '1'
option mtu '7500'
config wireguard_wg1
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_2_МК2'
list allowed_ips '192.168.31.2'
config wireguard_wg1ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.3
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_2_МК3'
list allowed_ips '192.168.31.3'
Configuration WireGuard on MK2 (added to /etc/config/network)
#VPN первого уровня - клиент
config interface 'wg0'
option proto 'wireguard'
list addresses '192.168.30.3/24'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_1_МК2'
option auto '1'
option mtu '8000'
config wireguard_wg0
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_1_VPS'
option endpoint_port '51820'
option persistent_keepalive '25'
list allowed_ips '192.168.30.0/24'
option endpoint_host 'IP_АДРЕС_VPS'
#VPN второго уровня - клиент
config interface 'wg1'
option proto 'wireguard'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_2_МК2'
list addresses '192.168.31.2/24'
option auto '1'
option listen_port '51821'
option mtu '7500'
config wireguard_wg1
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_2_МС'
option endpoint_host '192.168.30.2'
option endpoint_port '51821'
option persistent_keepalive '25'
list allowed_ips '192.168.31.0/24'
Configuration WireGuard on MK3 (added to /etc/config/network)
#VPN первого уровня - клиент
config interface 'wg0'
option proto 'wireguard'
list addresses '192.168.30.4/24'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_1_МК3'
option auto '1'
option mtu '8000'
config wireguard_wg0
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_1_VPS'
option endpoint_port '51820'
option persistent_keepalive '25'
list allowed_ips '192.168.30.0/24'
option endpoint_host 'IP_АДРЕС_VPS'
#VPN второго уровня - клиент
config interface 'wg1'
option proto 'wireguard'
option private_key 'ЗАКРЫТЫЙ_КЛЮЧ_VPN_2_МК3'
list addresses '192.168.31.3/24'
option auto '1'
option listen_port '51821'
option mtu '7500'
config wireguard_wg1
option public_key 'ОТКРЫТЫЙ_КЛЮЧ_VPN_2_МС'
option endpoint_host '192.168.30.2'
option endpoint_port '51821'
option persistent_keepalive '25'
list allowed_ips '192.168.31.0/24'
In the described configurations for the second level VPN, I indicate to clients WireGuard Port 51821. This shouldn't be necessary, as the client will establish a connection from any free, unprivileged port, but I did it this way so that I could deny all incoming connections on the wg0 interfaces of all routers, except for incoming UDP connections to port 51821.
I hope that the article will be useful to someone.
P.S. Also, I want to share my script that sends me a PUSH notification to my phone in the WirePusher application when a new device appears on my network. Here is a link to the script: .
UPDATE: Configuration OpenVPN-servers and clients
OpenVPN-server
client-to-client
ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/vpn-server.crt
dh /etc/openvpn/server/dh.pem
key /etc/openvpn/server/vpn-server.key
dev tap
ifconfig-pool-persist /etc/openvpn/ipp.txt 0
keepalive 10 60
proto tcp4
server-bridge 192.168.10.1 255.255.255.0 192.168.10.80 192.168.10.254
status /var/log/openvpn-status.log
verb 3
comp-lzoOpenVPN-client
client
tls-client
dev tap
proto tcp
remote VPS_IP 1194 # Change to your router's External IP
resolv-retry infinite
nobind
ca client/ca.crt
cert client/client.crt
key client/client.key
dh client/dh.pem
comp-lzo
persist-tun
persist-key
verb 3 I used easy-rsa to generate certificates.
Source: habr.com
