A corporate conflict arose on 10.06.2019 due to the increased cost of SMS delivery to users of the Vimpelcom network by Mail.RU Group. In response, Mail.RU Group stopped "serving" direct Russian IP channels towards the Vimpelcom network.
Below is a brief analysis of the situation from the perspective of a network engineer.
Update: 14.06.2019 18:45 — focus on Russian routes to the Vimpelcom network, conclusions adjusted, explanation added Sergey Kubasov (CIO of Vkontakte).
Update: 14.06.2019 19:48 — description added on how to limit the propagation of routes via the "short" Russian path through Rostelecom, MTS, RETN.NET.
Introductory remarks:
Vimpelcom has an autonomous system AS3216, all others (8402 — home internet, 16345 — mobile internet) are located behind 3216.
Mail.RU Group consists of autonomous systems AS47541, AS47542, and AS47764. The main content generator is 47542, known as CDN VKONTAKTE (films, music). All autonomous systems are independent of each other (from the perspective of external autonomous systems).
First, let's look at the situation from the Vimpelcom network's side. For this, we will use .
Let's check the first autonomous system — AS47541.
2914 47541
79.104.32.251 (metric 10500) (79.104.32.251)
Origin IGP, metric 30, localpref 87, valid, internal, best, group-best, import-candidate, import suspect
Received Path ID 0, Local Path ID 1, version 2865394342
Community: 2914:410 2914:1214 2914:2213 2914:3200 3216:3000 3216:3103 47541:1 47541:40000 47541:50078 AS-PATH contains two autonomous systems — 2914 (NTT) and 47541 (VKONTAKTE-SPB-AS).
The localpref metric is set to 87, which according to the description in the RIPE DB for the AS3216 object corresponds to international peering.
…
remarks: International peer’s routes get local preference in the
remarks: range of 81-89.
…
This information is also confirmed by community 3216:3000 and 3216:3103 (source — ):
…
remarks: 3216:3000 Received from an international uplink or peer, specifically:
…
remarks: 3216:3103 AMS-IX
…
That is, Vimpelcom sees the route to Vkontakte through a European connection.
Let's look at another Vkontakte autonomous system — 47542 (VKONTAKTE-MSK-CDN-AS).
Everything is similar.
2914 47541
79.104.32.251 (metric 10500) (79.104.32.251)
Origin IGP, metric 30, localpref 87, valid, internal, best, group-best, import-candidate, import suspect
Received Path ID 0, Local Path ID 1, version 2865394338
Community: 2914:410 2914:1214 2914:2213 2914:3200 3216:3000 3216:3103 47541:1 47541:40000 47541:50078What about Mail.ru itself? Let's look at AS47764 (mailru-as).
3356 47764
194.67.0.215 (metric 10501) (194.67.0.215)
Origin IGP, metric 0, localpref 77, valid, internal, best, group-best, import-candidate
Received Path ID 0, Local Path ID 1, version 2867605667
Community: 3216:3000 3216:3007 3356:2 3356:22 3356:100 3356:123 3356:519 3356:2094 47764:1 47764:40000 47764:50077 VimpelCom sees Mail.ru via as3356 (uplink Level3, Tier1 operator). This information is confirmed by localpref 77:
…
remarks: Uplink’s routes get local preference in the range of 71-79.
remarks: Last Update: February 2012…
and community (3216:3000 and 3216:3007):
…
remarks: 3216:3000 Received from an international uplink or peer, specifically:
…
remarks: 3216:3007 Level 3 Communications
…
From the information received, it is evident that traffic from VimpelCom's network to VKontakte and Mail.ru is routed through European peering points according to the routes received via BGP protocol. There are no alternative routes through domestic Russian peering points in Looking Glass. No measures for artificially redirecting traffic through intentionally worse routes were found.
How does Mail.ru Group see VimpelCom's networks?
Let's use .
From routers in AS47764 (mailru-as):
Path #6: Received by speaker 0
Advertised to peers (in unique update groups):
188.93.60.188
1299 1299 1273 3216 3216
217.20.147.250 (metric 100) from 217.20.147.250 (217.20.147.253)
Origin IGP, metric 500, localpref 200, valid, internal, best, group-best
Received Path ID 0, Local Path ID 0, version 1172721494
Community: 1299:20000 47764:701 47764:41100 47764:41108 47764:50077AS-PATH contains AS1299 (Telia, Tier1 operator, uplink Mail.RU) and as1273 (Vodafon, Tier1 operator, uplink VimpelCom).
LocalPreference 200 is standard for Mail.ru's external peering (https://net.mail.ru/bgp.html), while MED 500 corresponds to all that is received not directly from the peering, not from IX, not from peering.
But why are there no local routes through Russian telecom operators???
There are, but their priority is 'non-standard'!
Here is a route through Rostelecom (as12389):
Path #1: Received by speaker 0
Not advertised to any peer
12389 3216
46.61.178.149 from 46.61.178.149 (213.59.207.79)
Origin IGP, metric 500, localpref 199, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 3216:2001 3216:2999 3216:4100 12389:5 12389:6 12389:1100 12389:1105 12389:1277 47764:701 47764:41100 47764:41104 47764:50077
Origin-AS validity: not-foundHere is one through MegaFon (as31133):
Path #2: Received by speaker 0
Not advertised to any peer
31133 3216
78.25.77.41 from 78.25.77.41 (10.222.253.97)
Origin IGP, metric 500, localpref 199, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 3216:2001 3216:2999 3216:4100 31133:300 31133:46170 47764:701 47764:41100 47764:41105 47764:50077
Origin-AS validity: not-foundHere is one through RETN.NET:
Path #3: Received by speaker 0
Not advertised to any peer
9002 9002 3216
87.245.253.24 from 87.245.253.24 (87.245.225.1)
Origin IGP, metric 500, localpref 199, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 9002:9002 9002:64667 47764:701 47764:41100 47764:41101 47764:50077
Origin-AS validity: not-foundAnd even through MTS!
Path #5: Received by speaker 0
Not advertised to any peer
8359 3216
212.188.61.105 from 212.188.61.105 (195.34.52.77)
Origin IGP, metric 500, localpref 199, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 8359:200 8359:609 8359:5012 47764:701 47764:41100 47764:41103 47764:50077
Origin-AS validity: not-foundThe localpref metric for these Russian routes is understated, meaning the routes are less favorable compared to foreign ones!
Additionally, There are restrictions on the part of Mail.Ru Group for distributing its prefixes in VimpelCom through Russian operators!
RETN.NET (http://lg.retn.net/):
On announcements from Mail.RU Group, the community 3216:65535 is noted.
Output from LG RETN.NETinet.0: 762737 destinations, 1734826 routes (762708 active, 222780 holddown, 277 hidden)
94.100.176.0/20 (1 entry, 1 announced)
*BGP Preference: 170/-201
…
…
AS path: 47764 I
AS path: Recorded
Communities: 3216:65535 9002:64667 9002:65530
…
Routes marked with this community are not accepted by VimpelCom into its network. An excerpt from the RIPE DB for AS3216:
...
Remarks: Internal communities are assigned only within the organization.
Remarks: They fall within the range of 3216:0000-3216:4999 and 3216:6000-3216:65535.
Remarks: These are always removed from incoming updates at the boundary.
Remarks: Routers.
...
Towards Rostelecom (http://lg.ip.rt.ru), Mail.RU Group gives similar routes with community 12389:8350.
Output from LG Rostelecom94.100.176.0/20 via 217.107.65.1 on eth0.9 [sr2 2019-06-13] * (100/?) [AS47764i]
Type: BGP unicast univ
BGP.origin: IGP
BGP.as_path: 47764
BGP.next_hop: 213.59.207.78
BGP.med: 0
BGP.local_pref: 850
BGP.community: (12389,1) (12389,1100) (12389,1105) (12389,1277) (12389,8350) (12389,8380) (47764,1) (47764,40000) (47764,50077)
BGP.originator_id: 213.59.207.78
BGP.cluster_list: 95.167.88.79 95.167.88.49 95.167.88.17
According to the records in RIPE DB for object as12389, this community means 'do not announce to network as3216':...
remarks: | 12389:835y When advertising to GoldenTelecom (AS3216) |
...
remarks: | ...y=0 - do not advertise |
...
Similarly towards MTS (http://lg.mtu.ru):Output from LG MTSBGP routing table entry for 94.100.176.0/20, version 161717219
Paths: (2 available, best #1, table default)
Multipath: eBGP
Advertised to update-groups:
6
47764, (received & used)
195.34.52.77 (metric 16) from 195.34.52.181 (195.34.52.181)
Origin IGP, metric 0, localpref 140, valid, internal, best
Community: 8359:2120 8359:2150 8359:5500 8359:55277
Originator: 195.34.52.77, Cluster list: 83.59.83.59
47764, (received & used)
195.34.52.77 (metric 16) from 195.34.52.182 (195.34.52.182)
Origin IGP, metric 0, localpref 140, valid, internal
Community: 8359:2120 8359:2150 8359:5500 8359:55277
Originator: 195.34.52.77, Cluster list: 83.59.2.77
The community 8359:2120 means:...
comments: 8359:212x when announcing to Sovam (Beeline)
...
comments: x=0 - do not announce
...
It is not possible to view Mail.RU Group's announcements towards MegaFon — the latter has no Looking Glass.
Let's take a look at AS47541 (VKONTAKTE-SPB-AS).
The output is too large.
Router: a9922-e-5
Command: show ip bgp 81.211.56.202
Last switch-over Thu Apr 5 04:25:09 2018: 1 year, 10 weeks, 6 hours, 9 minutes ago
Fri Jun 14 10:34:20.791 MSK
BGP routing table entry for 81.211.0.0/17
Versions:
Process bRIB/RIB SendTblVer
Speaker 913059757 913059757
Last Modified: May 21 05:20:38.536 for 1y03w
Paths: (6 available, best #4)
Advertised to update-groups (with more than one peer):
0.2
Advertised to peers (in unique update groups):
188.93.60.188
Path #1: Received by speaker 0
Not advertised to any peer
1299 1273 3216 3216
87.240.191.235 (metric 31) from 87.240.191.235 (87.240.191.235)
Origin IGP, metric 5000, localpref 150, valid, internal
Received Path ID 0, Local Path ID 0, version 0
Community: 1273:12752 1299:431 1299:4000 1299:20000 1299:20002 1299:20200 3216:2001 3216:2999 3216:4100 47541:701 47541:41100 47541:41111 47541:50078
Path #2: Received by speaker 0
Not advertised to any peer
1299 1273 3216 3216
87.240.191.248 (metric 31) from 87.240.191.248 (87.240.191.248)
Origin IGP, metric 5000, localpref 150, valid, internal
Received Path ID 0, Local Path ID 0, version 0
Community: 1273:12752 1299:431 1299:4000 1299:20000 1299:20002 1299:20200 3216:2001 3216:2999 3216:4100 47541:701 47541:41100 47541:41111 47541:50078
Path #3: Received by speaker 0
Not advertised to any peer
174 6762 3216 3216
87.240.191.249 (metric 31) from 87.240.191.249 (87.240.191.249)
Origin IGP, metric 5000, localpref 150, valid, internal
Received Path ID 0, Local Path ID 0, version 0
Community: 174:21100 174:22005 47541:701 47541:41100 47541:41108 47541:50078
Path #4: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Advertised to peers (in unique update groups):
188.93.60.188
174 6762 3216 3216
149.6.169.113 from 149.6.169.113 (38.28.1.236)
Origin IGP, metric 5000, localpref 150, valid, external, best, group-best
Received Path ID 0, Local Path ID 0, version 913059757
Community: 174:21100 174:22005 47541:701 47541:41100 47541:41108 47541:50078
Origin-AS validity: not-found
Path #5: Received by speaker 0
Not advertised to any peer
1273 1273 3216 3216
195.89.114.197 from 195.89.114.197 (195.2.1.107)
Origin IGP, metric 5005, localpref 150, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 1273:12752 3216:2001 3216:2999 3216:4100 47541:701 47541:41100 47541:41110 47541:50078
Origin-AS validity: not-found
Path #6: Received by speaker 0
Not advertised to any peer
3356 3356 3216 3216 3216
213.242.69.69 from 213.242.69.69 (4.69.177.130)
Origin IGP, metric 5000, localpref 150, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 3216:2001 3216:2999 3216:4100 3356:2 3356:22 3356:100 3356:123 3356:503 3356:2067 47541:701 47541:41100 47541:41107 47541:50078
Origin-AS validity: not-foundThe AS-PATH indicates AS174 — Cogent (uplink Mail.RU, Tier1), followed by AS6762 — Telecom Italia (uplink Vimpelcom). The Local Preference is consistently 150 across all external connections, regardless of .
Let's take a look at AS47542 (VKONTAKTE-MSK-CDN-AS).
Router: mx960-m9-0
Command: op lg-sh-bgp prefix 81.211.56.202
0.0.0.0/0 LP:151 MED: NH:87.240.191.222 AS path: 47541 I
Communities:
Accepted Best
0.0.0.0/0 LP:151 MED: NH:95.142.204.251 AS path: 47541 I
Communities:
Accepted
Inactive-reason: Interior > Exterior > Exterior via InteriorAnd from the second router:
Router: mx960-m9-1
Command: op lg-sh-bgp prefix 81.211.56.202
0.0.0.0/0 LP:151 MED: NH:87.240.191.224 AS path: 47541 I
Communities:
Accepted Best
0.0.0.0/0 LP:151 MED: NH:95.142.204.250 AS path: 47541 I
Communities:
Accepted
Inactive-reason: Interior > Exterior > Exterior via InteriorOnly default routes (0.0.0.0/0). This situation was explained by a Mail.RU Group employee , for which we thank him. In short: the Moscow segment of the VKontakte network acts as a caching (not generating) layer, aimed at optimizing the loading speed of popular and in-demand content. A concern for users, indeed.
If there is no route to a certain network, it means that this network is not served by the caching servers. Thus, the speed optimization does not work, and users suffer. But it should be emphasized — users not only from their own network but also from Vimpelcom.
Conclusions:
- From Vimpelcom's side, traffic towards Mail.RU Group is distributed naturally. No artificial redirections via Local Preference manipulations were detected.
- From Mail.RU Group, manipulations with Vimpelcom prefixes are observed.. Existing routes towards the Vimpelcom network through Russian operators have lower priorities compared to routes through foreign Tier1 operators.
- For routes transmitted to Russian operators (MTS, Rostelecom, RETN.NET), managing BGP communities have been added by Mail.RU Group to limit their propagation towards Vimpelcom.
Why does Mail.RU Group prioritize routes through Europe? Why does Mail.RU Group prohibit short internal connectivity with Vimpelcom?
Is it cheaper for them? To route traffic through foreign channels and pay tier-1 providers in currency?
Or is there a desire to push traffic further away, making it less convenient to retrieve?
This is unknown to the network engineer…
Source: habr.com
