The topic is pretty beaten up, I know. For example, there is a great article, but only the IP part of the blocklist is considered there. We will also add domains.

Due to the fact that the courts and the RKN block everything right and left, and the providers are trying hard not to fall under the fines issued by Revizorro, the associated losses from blocking are quite large. And among the "lawfully" blocked sites there are many useful ones (hello, rutracker)

I live outside the jurisdiction of the RKN, but my parents, relatives and friends remained at home. So it was decided to come up with an easy way for people far from IT to bypass blocking, preferably without their participation at all.

In this note, I will not describe the basic network things in steps, but I will describe the general principles of how this scheme can be implemented. So knowledge of how the network works in general and in Linux in particular is a must have.

Types of locks

First, let's refresh our memory of what is being blocked.

There are several types of locks in the unloaded XML from the RKN:

IP
Domain Name
URL

For simplicity, we will reduce them to two: IP and domain, and we will simply pull out the domain from blocking by URL (more precisely, they have already done this for us).

good people from Roskomsvoboda realized a wonderful API, through which we can get what we need:

Access to blocked sites

To do this, we need some small foreign VPS, preferably with unlimited traffic - there are many of these for 3-5 bucks. You need to take it in the near abroad so that the ping is not very large, but again, take into account that the Internet and geography do not always coincide. And since there is no SLA for 5 bucks, it’s better to take 2+ pieces from different providers for fault tolerance.

Next, we need to set up an encrypted tunnel from the client router to the VPS. I use Wireguard as the fastest and easiest to set up. I also have client routers based on Linux (APU2 or something in OpenWRT). In the case of some Mikrotik / Cisco, you can use the protocols available on them like OpenVPN and GRE-over-IPSEC.

Identification and redirection of traffic of interest

You can, of course, turn off all Internet traffic through foreign countries. But, most likely, the speed of working with local content will suffer greatly from this. Plus, the bandwidth requirements on VPS will be much higher.

Therefore, we will need to somehow allocate traffic to blocked sites and selectively direct it to the tunnel. Even if some of the "extra" traffic gets there, it's still much better than driving everything through the tunnel.

To manage traffic, we will use the BGP protocol and announce routes to the necessary networks from our VPS to clients. Let's take BIRD as one of the most functional and convenient BGP daemons.

IP

With blocking by IP, everything is clear: we simply announce all blocked IPs with VPS. The problem is that there are about 600 thousand subnets in the list that the API returns, and the vast majority of them are /32 hosts. This number of routes can confuse weak client routers.

Therefore, when processing the list, it was decided to summarize up to the network / 24 if it has 2 or more hosts. Thus, the number of routes was reduced to ~100 thousand. The script for this will follow.

Domains

It's more complicated and there are several ways. For example, you can install a transparent Squid on each client router and do HTTP interception there and peep into the TLS handshake in order to obtain the requested URL in the first case and the domain from SNI in the second.

But due to all sorts of newfangled TLS1.3 + eSNI, HTTPS analysis is becoming less and less real every day. Yes, and the infrastructure on the client side is becoming more complicated - you will have to use at least OpenWRT.

Therefore, I decided to take the path of intercepting responses to DNS requests. Here, too, any DNS-over-TLS / HTTPS begins to hover over your head, but we can (for now) control this part on the client - either disable it or use your own server for DoT / DoH.

How to intercept DNS?

Here, too, there may be several approaches.

Interception of DNS traffic via PCAP or NFLOG
Both of these methods of interception are implemented in the utility sidmat. But it has not been supported for a long time and the functionality is very primitive, so you still need to write a harness for it.
Analysis of DNS server logs
Unfortunately, the recursors known to me are not able to log responses, but only requests. In principle, this is logical, since, unlike requests, answers have a complex structure and it is difficult to write them in text form.
DNSTap
Fortunately, many of them already support DNSTap for this purpose.

What is DNSTap?

It is a client-server protocol based on Protocol Buffers and Frame Streams for transferring from a DNS server to a collector of structured DNS queries and responses. Essentially, the DNS server transmits query and response metadata (type of message, client/server IP, etc.) plus complete DNS messages in the (binary) form in which it works with them over the network.

It is important to understand that in the DNSTap paradigm, the DNS server acts as a client and the collector acts as a server. That is, the DNS server connects to the collector, and not vice versa.

Today DNSTap is supported in all popular DNS servers. But, for example, BIND in many distributions (like Ubuntu LTS) is often built for some reason without its support. So let's not bother with reassembly, but take a lighter and faster recursor - Unbound.

How to catch DNSTap?

There is some number CLI utilities for working with a stream of DNSTap events, but they are not suitable for solving our problem. Therefore, I decided to invent my own bicycle that will do everything that is necessary: dnstap-bgp

Work algorithm:

When launched, it loads a list of domains from a text file, inverts them (habr.com -> com.habr), excludes broken lines, duplicates and subdomains (i.e. if the list contains habr.com and www.habr.com, it will be loaded only the first one) and builds a prefix tree for fast searching through this list
Acting as a DNSTap server, it waits for a connection from a DNS server. In principle, it supports both UNIX and TCP sockets, but the DNS servers I know can only use UNIX sockets
Incoming DNSTap packets are first deserialized into a Protobuf structure, and then the binary DNS message itself, located in one of the Protobuf fields, is parsed to the level of DNS RR records
It is checked whether the requested host (or its parent domain) is in the loaded list, if not, the response is ignored
Only A/AAAA/CNAME RRs are selected from the response and the corresponding IPv4/IPv6 addresses are extracted from them
IP addresses are cached with configurable TTL and advertised to all configured BGP peers
When receiving a response pointing to an already cached IP, its TTL is updated
After the TTL expires, the entry is removed from the cache and from BGP announcements

Additional functionality:

Rereading the list of domains by SIGHUP
Keeping the cache in sync with other instances dnstap-bgp via HTTP/JSON
Duplicate the cache on disk (in the BoltDB database) to restore its contents after a restart
Support for switching to a different network namespace (why this is needed will be described below)
IPv6 support

Limitations:

IDN domains are not supported yet
Few BGP settings

I collected RPM and DEB packages for easy installation. Should work on all relatively recent OSes with systemd. they don't have any dependencies.

scheme

So, let's start assembling all the components together. As a result, we should get something like this network topology:

The logic of work, I think, is clear from the diagram:

The client has our server configured as DNS, and DNS queries must also go over the VPN. This is necessary so that the provider cannot use DNS interception to block.
When opening the site, the client sends a DNS query like “what are the IPs of xxx.org”
Unbound resolves xxx.org (or takes it from the cache) and sends a response to the client “xxx.org has such and such IP”, duplicating it in parallel via DNSTap
dnstap-bgp announces these addresses in IBRD via BGP if the domain is on the blocked list
IBRD advertises a route to these IPs with next-hop self client router
Subsequent packets from the client to these IPs go through the tunnel

On the server, for routes to blocked sites, I use a separate table inside BIRD and it does not intersect with the OS in any way.

This scheme has a drawback: the first SYN packet from the client, most likely, will have time to leave through the domestic provider. the route is not announced immediately. And here options are possible depending on how the provider does the blocking. If he just drops traffic, then there is no problem. And if he redirects it to some DPI, then (theoretically) special effects are possible.

It's also possible that clients don't respect DNS TTL miracles, which can cause the client to use some stale entries from its rotten cache instead of asking Unbound.

In practice, neither the first nor the second caused problems for me, but your mileage may vary.

Server Tuning

For ease of rolling, I wrote role for Ansible. It can configure both servers and clients based on Linux (designed for deb-based distributions). All settings are quite obvious and are set in inventory.yml. This role is cut from my large playbook, so it may contain errors - pull requests welcome :)

Let's go through the main components.

BGP

Running two BGP daemons on the same host has a fundamental problem: BIRD doesn't want to set up BGP peering with the localhost (or any local interface). From the word at all. Googling and reading mailing-lists did not help, they claim that this is by design. Perhaps there is some way, but I did not find it.

You can try another BGP daemon, but I like BIRD and it is used everywhere by me, I don’t want to produce entities.

Therefore, I hid dnstap-bgp inside the network namespace, which is connected to the root through the veth interface: it's like a pipe, the ends of which stick out in different namespaces. On each of these ends, we hang private p2p IP addresses that do not go beyond the host, so they can be anything. This is the same mechanism used to access processes inside loved by all Docker and other containers.

For this it was written script and the functionality already described above for dragging yourself by the hair to another namespace was added to dnstap-bgp. Because of this, it must be run as root or issued to the CAP_SYS_ADMIN binary via the setcap command.

Example script for creating namespace

#!/bin/bash

NS="dtap"

IP="/sbin/ip"
IPNS="$IP netns exec $NS $IP"

IF_R="veth-$NS-r"
IF_NS="veth-$NS-ns"

IP_R="192.168.149.1"
IP_NS="192.168.149.2"

/bin/systemctl stop dnstap-bgp || true

$IP netns del $NS > /dev/null 2>&1
$IP netns add $NS

$IP link add $IF_R type veth peer name $IF_NS
$IP link set $IF_NS netns $NS

$IP addr add $IP_R remote $IP_NS dev $IF_R
$IP link set $IF_R up

$IPNS addr add $IP_NS remote $IP_R dev $IF_NS
$IPNS link set $IF_NS up

/bin/systemctl start dnstap-bgp

dnstap-bgp.conf

namespace = "dtap"
domains = "/var/cache/rkn_domains.txt"
ttl = "168h"

[dnstap]
listen = "/tmp/dnstap.sock"
perm = "0666"

[bgp]
as = 65000
routerid = "192.168.149.2"

peers = [
    "192.168.149.1",
]

bird.conf

router id 192.168.1.1;

table rkn;

# Clients
protocol bgp bgp_client1 {
    table rkn;
    local as 65000;
    neighbor 192.168.1.2 as 65000;
    direct;
    bfd on;
    next hop self;
    graceful restart;
    graceful restart time 60;
    export all;
    import none;
}

# DNSTap-BGP
protocol bgp bgp_dnstap {
    table rkn;
    local as 65000;
    neighbor 192.168.149.2 as 65000;
    direct;
    passive on;
    rr client;
    import all;
    export none;
}

# Static routes list
protocol static static_rkn {
    table rkn;
    include "rkn_routes.list";
    import all;
    export none;
}

rkn_routes.list

route 3.226.79.85/32 via "ens3";
route 18.236.189.0/24 via "ens3";
route 3.224.21.0/24 via "ens3";
...

DNS

By default, in Ubuntu, the Unbound binary is clamped by the AppArmor profile, which forbids it from connecting to all sorts of DNSTap sockets. You can either delete this profile, or disable it:

# cd /etc/apparmor.d/disable && ln -s ../usr.sbin.unbound .
# apparmor_parser -R /etc/apparmor.d/usr.sbin.unbound

This should probably be added to the playbook. It is ideal, of course, to correct the profile and issue the necessary rights, but I was too lazy.

unbound.conf

server:
    chroot: ""
    port: 53
    interface: 0.0.0.0
    root-hints: "/var/lib/unbound/named.root"
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    access-control: 192.168.0.0/16 allow

remote-control:
    control-enable: yes
    control-use-cert: no

dnstap:
    dnstap-enable: yes
    dnstap-socket-path: "/tmp/dnstap.sock"
    dnstap-send-identity: no
    dnstap-send-version: no

    dnstap-log-client-response-messages: yes

Downloading and processing lists

Script for downloading and processing a list of IP addresses
It downloads the list, sums up to the prefix pfx. In dont_add и dont_summarize you can tell the IPs and networks to skip or not summarize. I needed it. the subnet of my VPS was in the blocklist 🙂

The funny thing is that the RosKomSvoboda API blocks requests with the default Python user agent. Looks like the script-kiddy got it. Therefore, we change it to Ognelis.

So far, it only works with IPv4. the share of IPv6 is small, but it will be easy to fix. Unless you have to use bird6 as well.

rkn.py

#!/usr/bin/python3

import json, urllib.request, ipaddress as ipa

url = 'https://api.reserve-rbl.ru/api/v2/ips/json'
pfx = '24'

dont_summarize = {
    # ipa.IPv4Network('1.1.1.0/24'),
}

dont_add = {
    # ipa.IPv4Address('1.1.1.1'),
}

req = urllib.request.Request(
    url,
    data=None, 
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
)

f = urllib.request.urlopen(req)
ips = json.loads(f.read().decode('utf-8'))

prefix32 = ipa.IPv4Address('255.255.255.255')

r = {}
for i in ips:
    ip = ipa.ip_network(i)
    if not isinstance(ip, ipa.IPv4Network):
        continue

    addr = ip.network_address

    if addr in dont_add:
        continue

    m = ip.netmask
    if m != prefix32:
        r[m] = [addr, 1]
        continue

    sn = ipa.IPv4Network(str(addr) + '/' + pfx, strict=False)

    if sn in dont_summarize:
        tgt = addr
    else:
        tgt = sn

    if not sn in r:
        r[tgt] = [addr, 1]
    else:
        r[tgt][1] += 1

o = []
for n, v in r.items():
    if v[1] == 1:
        o.append(str(v[0]) + '/32')
    else:
        o.append(n)

for k in o:
    print(k)

Script to update
I run it on the crown once a day, maybe it’s worth pulling it every 4 hours. this, in my opinion, is the renewal period that the RKN requires from providers. Plus, they have some other super-urgent blocking, which may arrive faster.

Does the following:

Runs the first script and updates the list of routes (rkn_routes.list) for BIRD
Reload BIRD
Updates and cleans up the list of domains for dnstap-bgp
Reload dnstap-bgp

rkn_update.sh

#!/bin/bash

ROUTES="/etc/bird/rkn_routes.list"
DOMAINS="/var/cache/rkn_domains.txt"

# Get & summarize routes
/opt/rkn.py | sed 's/(.*)/route 1 via "ens3";/' > $ROUTES.new

if [ $? -ne 0 ]; then
    rm -f $ROUTES.new
    echo "Unable to download RKN routes"
    exit 1
fi

if [ -e $ROUTES ]; then
    mv $ROUTES $ROUTES.old
fi

mv $ROUTES.new $ROUTES

/bin/systemctl try-reload-or-restart bird

# Get domains
curl -s https://api.reserve-rbl.ru/api/v2/domains/json -o - | jq -r '.[]' | sed 's/^*.//' | sort | uniq > $DOMAINS.new

if [ $? -ne 0 ]; then
    rm -f $DOMAINS.new
    echo "Unable to download RKN domains"
    exit 1
fi

if [ -e $DOMAINS ]; then
    mv $DOMAINS $DOMAINS.old
fi

mv $DOMAINS.new $DOMAINS

/bin/systemctl try-reload-or-restart dnstap-bgp

They were written without much thought, so if you see something that can be improved - go for it.

Client setup

Here I will give examples for Linux routers, but in the case of Mikrotik / Cisco it should be even easier.

First, we set up BIRD:

bird.conf

router id 192.168.1.2;
table rkn;

protocol device {
    scan time 10;
};

# Servers
protocol bgp bgp_server1 {
    table rkn;
    local as 65000;
    neighbor 192.168.1.1 as 65000;
    direct;
    bfd on;
    next hop self;
    graceful restart;
    graceful restart time 60;
    rr client;
    export none;
    import all;
}

protocol kernel {
    table rkn;
    kernel table 222;
    scan time 10;
    export all;
    import none;
}

Thus, we will synchronize the routes received from BGP with the kernel routing table number 222.

After that, it is enough to ask the kernel to look at this plate before looking at the default one:

# ip rule add from all pref 256 lookup 222
# ip rule
0:  from all lookup local
256:    from all lookup 222
32766:  from all lookup main
32767:  from all lookup default

Everything, it remains to configure DHCP on the router to distribute the server's tunnel IP address as DNS, and the scheme is ready.

Disadvantages

With the current algorithm for generating and processing the list of domains, it includes, among other things, youtube.com and its CDNs.

And this leads to the fact that all videos will go through the VPN, which can clog the entire channel. Perhaps it is worth compiling a list of popular domains-exclusions that block the RKN for the time being, the guts are thin. And skip them when parsing.

Conclusion

The described method allows you to bypass almost any blocking that providers currently implement.

In principle, dnstap-bgp can be used for any other purpose where some level of traffic control is needed based on the domain name. Just keep in mind that in our time, a thousand sites can hang on the same IP address (behind some Cloudflare, for example), so this method has a rather low accuracy.

But for the needs of bypassing locks, this is quite enough.

Additions, edits, pull requests - welcome!

Source: habr.com

Bypass ILV blocking with DNSTap and BGP