A detailed response to the comment, as well as some insights into the life of providers in Russia

Inspired me to write this post this particular comment.

I’m sharing it here:

kaleman today at 18:53

Today my provider pleased me. Together with the update of the website blocking system, the mail.ru service got blocked. I've been calling tech support since this morning, but they can't do anything. The provider is small, and it seems they're being blocked by larger providers. I've also noticed a slowdown in loading all websites; maybe some faulty DLP has been implemented? There were no access problems before. The destruction of the RuNet is happening right before my eyes…

The thing is, it seems we are that very provider 🙁

And indeed, kaleman I almost guessed the reason for the mail.ru issues (even though we took a long time to believe it).

The following will be divided into two parts:

1. the reasons for our current issues with mail.ru and an exciting quest to find them
2. the existence of ISPs in today's realities, the stability of a sovereign RuNet.

Issues with mail.ru accessibility

Oh, that's quite a long story.

The fact is that to meet government requirements (more on this in the second part), we acquired, configured, and installed some equipment — both for filtering prohibited resources and for carrying out NAT translations for subscribers.

Some time ago, we finally restructured our network core so that all subscriber traffic would pass through this equipment strictly in the necessary direction.

A few days ago, we enabled filtering of prohibited content on it (simultaneously leaving the old system operational) — everything seemed to go well.

Next, we gradually started enabling NAT for different segments of subscribers on this equipment. Seemingly, everything went fairly well.

But today, after enabling NAT for another segment of subscribers — we faced a good number of complaints about the inaccessibility or partial accessibility of mail.ru and other resources of Mail Ru Group.

We started checking: something somewhere sometimes, occasionally sends TCP RST in response to requests solely to mail.ru networks. Moreover — it sends incorrectly generated (without ACK), clearly artificial TCP RST. It looked something like this:

Naturally, the first thoughts were about the new equipment: the terrifying DPI, no trust in it whatsoever, who knows what it might do — after all, TCP RST is quite a common thing among blocking tools.

Assumption kaleman that someone 'higher up' is filtering, we proposed as well — but immediately dismissed it.

Firstly, we have sufficiently competent uplinks to avoid such issues 🙂

Secondly, we are connected to several IX in Moscow, and the traffic to mail.ru goes precisely through them — and they have neither obligations nor any other motivation to filter the traffic.

The next half of the day was spent on what is usually called shamanism — together with the equipment vendor, for which we are grateful, they did not abandon us 🙂

• filtering was completely turned off
• NAT was disabled by a new scheme
• the test PC was moved to a separate isolated pool
• IP addressing was changed

In the second half of the day, a virtual machine was allocated that connected to the network as a regular user, and both it and the equipment were made accessible to vendor representatives. The shamanism continued 🙂

In the end, the vendor representative confidently stated that the hardware was definitely not the issue: rst's are coming from somewhere above.

NoteAt this point, someone might say: but it would have been much simpler to take a dump not from the test PC, but from the trunk above the DPI?

No, unfortunately, taking a dump (and even just mirroring) 40+gbps is not trivial at all.

After that, in the evening — there was nothing left to do but return to the assumption of strange filtering somewhere above.

I checked which IX the traffic to the MRG networks is currently going through and simply shut down the bgp sessions to it. And — oh miracle! — everything immediately normalized 🙁

On the one hand — it’s very sad that the entire day was spent searching for the problem, although it was resolved in five minutes.

On the other hand:

— in my memory, this is unprecedented. As I wrote above — IXs really have no reason to filter transit traffic. They usually have hundreds of gigabits / terabits per second. I simply could not seriously assume such a thing until the last moment.

— an incredibly fortunate coincidence: new complex hardware, which is not especially trusted and from which it is unclear what to expect — specifically designed for blocking resources, including TCP RSTs

At the moment, the NOC of this internet exchange is looking for the problem. According to their statement (and I believe them) they have no specially deployed filtering system. But, thank heaven, the further quest is no longer our problem 🙂

This was a small attempt to justify myself, please understand and forgive 🙂

P.S.: I intentionally do not mention either the DPI/NAT manufacturer, or IX (I actually have no particular complaints against them, the main thing is to understand what happened)

Today's (as well as yesterday's and the day before yesterday's) reality from the ISP's perspective

I spent the last few weeks significantly restructuring the network core, performing a lot of live manipulations, risking to significantly impact live user traffic. Considering the goals, results, and consequences of all this — morally it is quite heavy. Especially listening once again to the noble speeches about protecting the stability of the RuNet, sovereignty, etc.

In this section, I will attempt to describe the "evolution" of the network core of a typical ISP over the last decade.

A decade ago.

In those blessed times, the core of an ISP's network could be as simple and reliable as a cork:

This very, very simplified picture lacks highways, rings, IP/MPLS routing.

The essence of it is that user traffic eventually reached the core switching — from where it went to BNG, from where, as a rule, it went back to the core switching, and then "out" — through one or more border gateways to the internet.

Such a scheme is very, very easily redundant both at L3 (dynamic routing) and at L2 (MPLS).

You can deploy N+1 of anything: access servers, switches, borders — and in one way or another reserve them for automatic failover.

A few years later it became clear to everyone in Russia that this way of living was no longer possible: it was necessary to urgently protect children from the corrupting influence of the network.

There arose a need to urgently find ways to filter user traffic.

There are different approaches to this.

In not very good cases — something is placed "in line": between user traffic and the internet. The traffic passing through this "something" is subject to analysis and, for example, a fake packet with a redirect is sent towards the subscriber.

In a slightly better scenario — if the traffic volumes allow — a little trick can be used: filter only the outgoing traffic from users to those addresses that need to be filtered (for this, you can either take the IP addresses specified in the registry or additionally resolve the domains listed in the registry).

At the time, I wrote a simple mini-DPI — although it’s hard to even call it that. It is very simple and not very efficient — however, it allowed us, and dozens (if not hundreds) of other providers, to avoid spending millions immediately on industrial DPI systems, giving us a few extra years of time.

By the way, about the DPI systems of then and nowIt’s worth mentioning that many who purchased the DPI systems available on the market at that time have already discarded them. They are just not suited for this: hundreds of thousands of addresses, tens of thousands of URLs.

At the same time, domestic manufacturers have significantly risen in this market. I’m not talking about hardware — that’s clear to everyone; however, the software — the main thing present in DPI — possibly, as of today, if not the most advanced in the world, is definitely a) developing by leaps and bounds, and b) in terms of boxed price — simply incomparable with foreign competitors.

I would like to take pride in this, but it’s a bit sad =)

Now everything looked like this:

A couple of years later everyone already had auditors; the resources in the registry were increasing more and more. For some old equipment (like Cisco 7600), the scheme with 'side filtering' became simply unworkable: the number of routes on 76 platforms is limited to around nine hundred thousand, while the number of just IPv4 routes today is approaching 800 thousand. And if we also consider IPv6… How many is it now? 900,000 separate addresses banned by the RKN? =)

Some switched to a scheme of mirroring all the backbone traffic to a filtering server, which must analyze the entire stream and, upon finding something suspicious, send RSTs in both directions (to the sender and the recipient).

However, the more traffic there is, the less applicable this scheme becomes. With the slightest delay in processing — mirrored traffic will just fly off unnoticed, and the provider will receive a penalty protocol.

More and more providers are forced to implement DPI systems of varying reliability into the backbone infrastructure.

A year or two ago rumor has it that practically all FSB now requires the actual installation of equipment SORM (previously, most providers managed by coordinating with the authorities the SORM plan — a plan of operational measures in case something needs to be found somewhere)

In addition to money (not that it was exactly astronomical, but still millions), SORM required many to perform additional manipulations with the network.

• SORM needs to see the 'gray' addresses of users before NAT translation
• SORM has a limited number of network interfaces

Therefore, we, in particular, had to significantly restructure part of the kernel—just to gather user traffic to access servers in one place. This was necessary to mirror it to SORM using several links.

In other words, to simplify it, it was (left) vs became (right):

Currently most providers are also required to implement SORM-3—which includes, among other things, the logging of NAT translations.

For this purpose, we had to add separate NAT equipment to the above scheme (the very one discussed in the first part). Moreover, it had to be added in a specific order: because SORM must 'see' traffic before the addresses are translated—the traffic must strictly follow this path: users -> switching, kernel -> access servers -> SORM -> NAT -> switching, kernel -> internet. For this, we had to literally 'flip' the traffic flows in the opposite direction on the fly, which was also quite challenging.

In total: over the decade, the core scheme of an average provider has significantly complicated, with additional points of failure (both in the form of hardware and as unified switching lines) greatly increased. The requirement to 'see everything' implies consolidating this 'everything' into one point.

I think this can be quite transparently extrapolated to current initiatives regarding the sovereignty of RuNet, its protection, stabilization, and improvement 🙂

And ahead lies Yarovaya.

Source: habr.com