Hello. So, there's a network of 5,000 clients. Recently, an unpleasant issue arose — in the center of the network, we have a Brocade RX8, and it started sending a lot of unknown-unicast packets because the network is split into VLANs — partly, this isn't a problem, BUT there are special VLANs for white addresses, etc., and they are spread throughout the network. Now, imagine the incoming stream to a client's address, which the border doesn't learn about, and this stream is heading towards some (actually all) remote area — the channel is clogged — clients are upset — sadness…
The task is to turn the bug into a feature. I was thinking about q-in-q with a full client VLAN, but various devices like P3310 stop passing DHCP when dot1q is enabled, and they don’t support selective qinq. There are many underlying pitfalls like this. What is ip-unnumbered and how does it work? In short — gateway address + route on the interface. For our task, we need to: cut the shaper, distribute addresses to clients, and add routes for clients through specific interfaces. What do we use for this? The shaper — lisg, DHCP — db2dhcp on two independent servers, on the access servers, there’s dhcprelay, and ucarp runs on the access servers — for backup. But how do we add routes? We can pre-add everything with a large script — but that’s not true. So we'll be crafting a custom workaround.
After digging deep into the internet, I found a wonderful high-level library for C++ that allows you to sniff traffic elegantly. The algorithm for the program that adds routes is as follows: we listen for ARP requests on the interface; if we have an address on the lo interface that’s being queried, we add a route through this interface and add a static ARP entry for this IP — essentially, a bit of copy-pasting, some improvisation, and it's ready.
Source code for the 'router'
#include <stdio.h>
#include <sys/types.h>
#include <ifaddrs.h>
#include <netinet/in.h>
#include <string.h>
#include <arpa/inet.h>
#include <tins/tins.h>
#include <map>
#include <iostream>
#include <functional>
#include <sstream>
using std::cout;
using std::endl;
using std::map;
using std::bind;
using std::string;
using std::stringstream;
using namespace Tins;
class arp_monitor {
public:
void run(Sniffer &sniffer);
void reroute();
void makegws();
string iface;
map <string, string> gws;
private:
bool callback(const PDU &pdu);
map <string, string> route_map;
map <string, string> mac_map;
map <IPv4Address, HWAddress<6>> addresses;
};
void arp_monitor::makegws() {
struct ifaddrs *ifAddrStruct = NULL;
struct ifaddrs *ifa = NULL;
void *tmpAddrPtr = NULL;
gws.clear();
getifaddrs(&ifAddrStruct);
for (ifa = ifAddrStruct; ifa != NULL; ifa = ifa->ifa_next) {
if (!ifa->ifa_addr) {
continue;
}
string ifName = ifa->ifa_name;
if (ifName == "lo") {
char addressBuffer[INET_ADDRSTRLEN];
if (ifa->ifa_addr->sa_family == AF_INET) { // check it is IP4
// is a valid IP4 Address
tmpAddrPtr = &((struct sockaddr_in *) ifa->ifa_addr)->sin_addr;
inet_ntop(AF_INET, tmpAddrPtr, addressBuffer, INET_ADDRSTRLEN);
} else if (ifa->ifa_addr->sa_family == AF_INET6) { // check it is IP6
// is a valid IP6 Address
tmpAddrPtr = &((struct sockaddr_in6 *) ifa->ifa_addr)->sin6_addr;
inet_ntop(AF_INET6, tmpAddrPtr, addressBuffer, INET6_ADDRSTRLEN);
} else {
continue;
}
gws[addressBuffer] = addressBuffer;
cout << "GW " << addressBuffer << " is added" << endl;
}
}
if (ifAddrStruct != NULL) freeifaddrs(ifAddrStruct);
}
void arp_monitor::run(Sniffer &sniffer) {
cout << "RUNNED" << endl;
sniffer.sniff_loop(
bind(
&arp_monitor::callback,
this,
std::placeholders::_1
)
);
}
void arp_monitor::reroute() {
cout << "REROUTING" << endl;
map<string, string>::iterator it;
for ( it = route_map.begin(); it != route_map.end(); it++ ) {
if (this->gws.count(it->second) && !this->gws.count(it->second)) {
string cmd = "ip route replace ";
cmd += it->first;
cmd += " dev " + this->iface;
cmd += " src " + it->second;
cmd += " proto static";
cout << cmd << std::endl;
cout << "REROUTE " << it->first << " SRC " << it->second << endl;
system(cmd.c_str());
cmd = "arp -s ";
cmd += it->first;
cmd += " ";
cmd += mac_map[it->first];
cout << cmd << endl;
system(cmd.c_str());
}
}
for ( it = gws.begin(); it != gws.end(); it++ ) {
string cmd = "arping -U -s ";
cmd += it->first;
cmd += " -I ";
cmd += this->iface;
cmd += " -b -c 1 ";
cmd += it->first;
system(cmd.c_str());
}
cout << "REROUTED" << endl;
}
bool arp_monitor::callback(const PDU &pdu) {
// Retrieve the ARP layer
const ARP &arp = pdu.rfind_pdu<ARP>();
if (arp.opcode() == ARP::REQUEST) {
string target = arp.target_ip_addr().to_string();
string sender = arp.sender_ip_addr().to_string();
this->route_map[sender] = target;
this->mac_map[sender] = arp.sender_hw_addr().to_string();
cout << "save sender " << sender << ":" << this->mac_map[sender] << " want taregt " << target << endl;
if (this->gws.count(target) && !this->gws.count(sender)) {
string cmd = "ip route replace ";
cmd += sender;
cmd += " dev " + this->iface;
cmd += " src " + target;
cmd += " proto static";
// cout << cmd << std::endl;
/* cout << "ARP REQUEST FROM " << arp.sender_ip_addr()
<< " for address " << arp.target_ip_addr()
<< " sender hw address " << arp.sender_hw_addr() << std::endl
<< " run cmd: " << cmd << endl;*/
system(cmd.c_str());
cmd = "arp -s ";
cmd += arp.sender_ip_addr().to_string();
cmd += " ";
cmd += arp.sender_hw_addr().to_string();
cout << cmd << endl;
system(cmd.c_str());
}
}
return true;
}
arp_monitor monitor;
void reroute(int signum) {
monitor.makegws();
monitor.reroute();
}
int main(int argc, char *argv[]) {
string test;
cout << sizeof(string) << endl;
if (argc != 2) {
cout << "Usage: " << *argv << " <interface>" << endl;
return 1;
}
signal(SIGHUP, reroute);
monitor.iface = argv[1];
// Sniffer configuration
SnifferConfiguration config;
config.set_promisc_mode(true);
config.set_filter("arp");
monitor.makegws();
try {
// Sniff on the provided interface in promiscuous mode
Sniffer sniffer(argv[1], config);
// Only capture arp packets
monitor.run(sniffer);
}
catch (std::exception &ex) {
std::cerr << "Error: " << ex.what() << std::endl;
}
}Installation script for libtins
#!/bin/bash
git clone https://github.com/mfontanini/libtins.git
cd libtins
mkdir build
cd build
cmake ../
make
make install
ldconfig
Command to build the binary
g++ main.cpp -o arp-rt -O3 -std=c++11 -lpthread -ltinsHow do you run it?
start-stop-daemon --start --exec /opt/ipoe/arp-routes/arp-rt -b -m -p /opt/ipoe/arp-routes/daemons/eth0.800.pid -- eth0.800
Yes — it rebuilds the tables on the HUP signal. Why didn't you use netlink? Just lazy, and Linux is basically a script on a script — so everything's fine. Well, routes are routes, what's next? Next, we need to send the routes that are available on this server to the border — here, due to the outdated hardware, we took the path of least resistance — we pushed this task onto BGP.
BGP confighostname *******
password *******
log file /var/log/bgp.log
!
# номер ас-ки, адреса и сети выдуманы
router bgp 12345
bgp router-id 1.2.3.4
redistribute connected
redistribute static
neighbor 1.2.3.1 remote-as 12345
neighbor 1.2.3.1 next-hop-self
neighbor 1.2.3.1 route-map none in
neighbor 1.2.3.1 route-map export out
!
access-list export permit 1.2.3.0/24
!
route-map export permit 10
match ip address export
!
route-map export deny 20
Let's continue. For the server to respond to ARP requests, we need to enable proxy ARP.
echo 1 > /proc/sys/net/ipv4/conf/eth0.800/proxy_arp
Moving on — ucarp. We write the start scripts for this wonder ourselves.
Example of starting one daemon
start-stop-daemon --start --exec /usr/sbin/ucarp -b -m -p /opt/ipoe/ucarp-gen2/daemons/$iface.$vhid.$virtualaddr.pid -- --interface=eth0.800 --srcip=1.2.3.4 --vhid=1 --pass=carpasword --addr=10.10.10.1 --upscript=/opt/ipoe/ucarp-gen2/up.sh --downscript=/opt/ipoe/ucarp-gen2/down.sh -z -k 10 -P --xparam="10.10.10.0/24"
up.sh
#!/bin/bash
iface=$1
addr=$2
gw=$3
vlan=`echo $1 | sed "s/eth0.//"`
ip ad ad $addr/32 dev lo
ip ro add blackhole $gw
echo 1 > /proc/sys/net/ipv4/conf/$iface/proxy_arp
killall -9 dhcrelay
/etc/init.d/dhcrelay zap
/etc/init.d/dhcrelay start
killall -HUP arp-rt
down.sh
#!/bin/bash
iface=$1
addr=$2
gw=$3
ip ad d $addr/32 dev lo
ip ro de blackhole $gw
echo 0 > /proc/sys/net/ipv4/conf/$iface/proxy_arp
killall -9 dhcrelay
/etc/init.d/dhcrelay zap
/etc/init.d/dhcrelay start
For dhcprelay to work on the interface — it needs an address. Therefore, on the interfaces we use, we will add fake addresses — for example, 10.255.255.1/32, 10.255.255.2/32, etc. I won't explain how to configure the relay — it's all straightforward.
So, what do we have. Backup gateways, auto-configured routes, DHCP. This is the minimum set — in addition, we layer on LISG and we already have shaping. Why is everything so lengthy and convoluted? Isn't it easier to just use accel-pppd and PPPoE? No, it's not easier — people struggle to fit the patch cord into the router, let alone PPPoE. accel-ppp is great — but it didn't work for us — a bunch of errors in the code — it crashes, cuts awkwardly, and the saddest part is that if it crashes — people need to reboot everything — phones ringing red — in general, it didn't suit us. What’s the advantage of using ucarp over keepalived? It's everything — there are 100 gateways, keepalived, and one error in the config — nothing works. With ucarp, one gateway not working is not a big deal. Regarding security, if they write fake addresses and use it on the network — to control this moment on all switches/access points/databases, we configure dhcp-snooping + source-guard + arp inspection. If the client doesn't have DHCP and is using static — access-list on the port.
Why was all this done? To eliminate unwanted traffic. Now, each switch has its own VLAN, and unknown unicast is no longer a concern since it only needs to go through one port instead of all… Moreover, the side effects include a standardized equipment configuration and improved efficiency in address space allocation.
Configuring lisg is a separate topic. Links to the libraries are attached. The information above may help someone in implementing their tasks. We haven't deployed version 6 in our network yet, but there's a potential issue — we plan to rewrite lisg for version 6, and the program that adds routes will also need adjustments.
Source: habr.com
