“We set up a telephone connection between us and the guys at SRI…”, Kleinrock… said in an interview:
"We typed the L and we asked on the phone, 'Do you see the L?'"
"Yes, we see the L," came the response.
"We typed the O, and we asked, 'Do you see the O.'"
"Yes, we see the O."
"Then we typed the G, and the system crashed"...Yet a revolution had begun…
The beginning of the internet.
Hi all!
My name is Alexander, I am a network engineer at Linxdatacenter. In today's article, we will talk about traffic exchange points (Internet Exchange Point, IXP): about what preceded their appearance, what tasks they solve and how they are built. Also in this article, I will demonstrate how IXP works using the EVE-NG platform and the BIRD software router so that there is an understanding of how it works “under the hood”.
A bit of history
If you look , you can see that the rapid growth in the number of traffic exchange points began in 1993. This is due to the fact that most of the traffic of the telecom operators that existed at that time passed through the US backbone network. So, for example, when traffic went from an operator in France to an operator in Germany, it first got from France to the USA, and only then from the USA to Germany. The backbone network in this case acted as a transit between France and Germany. Even traffic within one country often passed not directly, but through the backbone networks of American operators.
This state of affairs affected not only the cost of transit traffic delivery, but also the quality of channels and delay. The number of Internet users increased, new operators appeared, the volume of traffic increased, the Internet matured. Operators around the world began to understand that a more rational approach to the organization of inter-operator interaction is needed. “Why would I, operator A, pay for transit through another country in order to deliver traffic to operator B, which is located on the next street?”. Approximately such a question was asked by telecom operators at that time. So, in different parts of the world, in points of concentration of operators, traffic exchange points began to appear:
- 1994 - LINX in London,
- 1995 - DE-CIX in Frankfurt,
- 1995 - MSK-IX, in Moscow, etc.
Internet and our days
Conceptually, the architecture of the modern Internet is a set of autonomous systems (autonomous system, AS) and many connections between them, both physical and logical, which determine the path of traffic from one AS to another.
As an AS, telecom operators, Internet providers, CDNs, data centers, and enterprise segment companies usually act. ASs organize logical connections (peering) among themselves, as a rule, by means of the BGP protocol.
How autonomous systems organize these connections is determined by a number of factors:
- geographical,
- economic,
- political,
- agreements and common interests between AS owners,
- etc.
Of course, in this scheme there is a certain structure and hierarchy. So, operators are divided into tier-1, tier-2 and tier-3, and if the clients for the local Internet provider (tier-3) are, as a rule, ordinary users, then, for example, for operators of the tier-1 level, the clients are other operators. Tier-3 operators aggregate the traffic of their subscribers on themselves, tier-2 telecom operators, in turn, aggregate the traffic of tier-3 operators, and tier-1 - all Internet traffic.
Schematically, this can be represented as follows:
This picture shows that traffic is aggregated from the bottom up, i.e. from end users to tier-1 operators. There is also a horizontal exchange of traffic between approximately equivalent ASs.
An integral part and at the same time a disadvantage of this scheme is a certain disorder of connections between autonomous systems located closer to the end user, within the geographical area. Consider the picture below:
Suppose that there are 5 telecom operators in a large city, peering between which, for one reason or another, is organized as shown above.
If the user Petya, connected to the Go ISP, wants to access a server connected to the ASM provider, then the traffic between them will be forced to go through 5 autonomous systems. Thus, the delay increases, because. increases the number of network devices through which traffic will go, as well as the amount of transit traffic on autonomous systems between Go and ASM.
How to reduce the number of transit ASs that traffic has to go through? That's right - a traffic exchange point.
Today, the emergence of new IXPs is driven by the same needs as in the early 90s-2000s, only on a smaller scale, in response to an increasing number of telecom operators, users and traffic, a growing amount of content generated by CDN networks and data centers.
What is a traffic exchange point?
A traffic exchange point is a place with a special network infrastructure where participants interested in mutual traffic exchange organize mutual peering. The main participants in traffic exchange points are telecom operators, Internet providers, content providers and data centers. At traffic exchange points, participants connect directly to each other. This allows you to solve the following tasks:
- reduce the delay
- reduce the amount of transit traffic,
- optimize routing between ASs.
Considering that IXPs are present in many large cities of the world, this all has a positive effect on the Internet as a whole.
If the above situation with Petya is solved with the help of IXP, then it will turn out something like this:
How is a traffic exchange point arranged?
Typically, an IXP is a separate AS with its own block of public IPv4/IPv6 addresses.
The IXP network is most often a continuous L2 domain. Sometimes it's just the VLAN that hosts all IXP clients. When it comes to larger, geographically distributed IXPs, then technologies such as MPLS, VXLAN, etc. can be used to organize an L2 domain.
IXP elements
- SCS. There is nothing unusual here: racks, optical cross-countries, patch panels.
- Switches - the basis of IXP. The switch port is the entry point to the IXP network. The switches also perform part of the security functions - they filter garbage traffic that should not be present on the IXP network. As a rule, switches are selected based on functional requirements - reliability, supported port speeds, security features, sFlow support, etc.
- Route server (RS) - an integral and necessary part of any modern traffic exchange point. By the principle of operation, it very much resembles a route reflector in iBGP or a designated router in OSPF and solves the same problems. As the number of participants in the traffic exchange point grows, the number of BGP sessions that each of the participants needs to support increases, i.e. this is reminiscent of the classic full-mesh topology in iBGP. RS solves the problem in the following way: it establishes a BGP session with each interested IXP, which becomes a client of RS. Upon receiving a BGP update from one of its clients, RS sends this update to all its other clients, of course, except for the one from which this update was received. Thus, RS eliminates the need to establish a full-mesh between all IXP members and elegantly solves the problem of scalability. It is worth noting that the route server transparently passes routes from one AS to another without changing the attributes transmitted by BGP, for example, does not add a number in its AS to the AS-path. Also, basic route filtering occurs on RS: for example, RS does not accept martians networks and prefixes of the IXP itself.
As a route server solution, an open source software router is often used - BIRD (bird internet routing daemon). It is good because it is free, quickly deployed on most linux distributions, has a flexible mechanism for setting routing / filtering policies, and is not demanding on computing resources. Also, a hardware / virtual router of Cisco, Juniper, etc. can be selected as RS.
- Security. Since the IXP network is a concentration of a large number of ASs, the security policy that all participants must follow must be well written. In general, all of the same mechanisms that apply when establishing a BGP neighbor between two separate BGP peers outside of an IXP apply here, plus some additional protections.
For example, it is good practice to allow traffic only from a specific mac-address of an IXP participant, which is negotiated in advance. Deny traffic with ethertype fields other than 0x0800(IPv4), 0x08dd(IPv6), 0x0806(ARP); this is done in order to filter out traffic that has no place in BGP peering. Mechanisms such as GTSM, RPKI, etc. can also be used.
Perhaps the above are the main components of any IXP, regardless of scale. Of course, large IXPs may use additional technologies and solutions.
It happens that IXP also provides its members with additional services:
- placed on IXP TLD DNS servers,
- install hardware NTP servers, enabling participants to accurately synchronize time,
- provide protection against DDoS attacks, etc.
Principle of operation
Let's analyze the principle of operation of the traffic exchange point using the example of the simplest IXP, modeled by means of EVE-NG, and then consider the basic configuration of the BIRD software router. To simplify the diagram, we will omit such important things as redundancy and fault tolerance.
The network topology is shown in the figure below.
Let's assume that we administer a small exchange point and provide the following peering options:
- public peering,
- private peering,
- peering through route server.
Our AS number is 555, we own a block of IPv4 addresses - 50.50.50.0/24, from which we issue IP addresses for those who want to connect to our network.
50.50.50.254 – IP address configured on the route server's interface, with this IP, clients will establish a BGP session in case of peering via RS.
Also, for peering through RS, we have developed the simplest routing policy based on the BGP community, which allows IXP members to control to whom and which routes to send:
BGP community
Description
LOCAL_AS:PEER_AS
Pass prefixes only PEER_AS
LOCAL_AS:IXP_AS
Send prefixes to all IXP members
3 clients wish to connect to our IXP and exchange traffic; Let's say they are ISPs. All of them want to establish peering through the route server. Below is a diagram with client connection parameters:
Customer
Customer AS Number
Prefixes announced by the client
ip address given to the client to connect to the IXP
ISP#1
AS 100
1.1.0.0/16
50.50.50.10/24
ISP#2
AS 200
2.2.0.0/16
50.50.50.20/24
ISP#3
AS 300
3.3.0.0/16
50.50.50.30/24
Basic BGP setup on client router:
router bgp 100
no bgp enforce-first-as
bgp log-neighbor-changes
neighbor 50.50.50.254 remote-as 555
address-family ipv4
network 1.1.0.0 mask 255.255.0.0
neighbor 50.50.50.254 activate
neighbor 50.50.50.254 send-community both
neighbor 50.50.50.254 soft-reconfiguration inbound
neighbor 50.50.50.254 route-map ixp-out out
exit-address-family
ip prefix-list as100-prefixes seq 5 permit 1.1.0.0/16
route-map bgp-out permit 10
match ip address prefix-list as100-prefixes
set community 555:555
The no bgp enforce-first-as setting is worth noting here. By default, BGP requires that the as-path of an update received by BGP contains the as-bgp number of the peer from which the update was received. But since the route server does not make changes to the as-path, its number will be missing from the as-path and the update will be discarded. This setting is used to make the router ignore this rule.
We also see that the client has set bgp community 555:555 to this prefix, which, according to our policy, means that the client wants to advertise this prefix to all other participants.
For routers of other clients, the setting will be similar, except for their unique parameters.
BIRD configuration example:
define ixp_as = 555;
define ixp_prefixes = [ 50.50.50.0/24+ ];
template bgp RS_CLIENT {
local as ixp_as;
rs client;
}
The following describes a filter that does not accept martians prefixes, as well as prefixes from IXP itself:
function catch_martians_and_ixp()
prefix set martians;
prefix set ixp_prefixes;
{
martians = [
0.0.0.0/8+,
10.0.0.0/8+,
100.64.0.0/10+,
127.0.0.0/8+,
169.254.0.0/16+,
172.16.0.0/12+,
192.0.0.0/24+,
192.0.2.0/24+,
192.168.0.0/16+,
198.18.0.0/15+,
198.51.100.0/24+,
203.0.113.0/24+,
224.0.0.0/4+,
240.0.0.0/4+ ];
if net ~ martians || net ~ ixp_prefixes then return false;
return true;
}
This function implements the routing policy we described earlier.
function bgp_ixp_policy(int peer_as)
{
if (ixp_as, ixp_as) ~ bgp_community then return true;
if (ixp_as, peer_as) ~ bgp_community then return true;
return false;
}
filter reject_martians_and_ixp
{
if catch_martians_and_ixp() then reject;
if ( net ~ [0.0.0.0/0{25,32} ] ) then {
reject;
}
accept;
}
We set up peering, apply the appropriate filters and policies.
protocol as_100 from RS_CLIENT {
neighbor 50.50.50.10 as 100;
ipv4 {
export where bgp_ixp_policy(100);
import filter reject_martians_and_ixp;
}
}
protocol as_200 from RS_CLIENT {
neighbor 50.50.50.20 as 200;
ipv4 {
export where bgp_ixp_policy(200);
import filter reject_martians_and_ixp;
}
}
protocol as_300 from RS_CLIENT {
neighbor 50.50.50.30 as 300;
ipv4 {
export where bgp_ixp_policy(300);
import filter reject_martians_and_ixp;
}
}
It is worth noting that on a route server it is good practice to add routes from different peers to different RIBs. BIRD allows you to do this. In our example, for simplicity, all updates received from all clients are added to one common RIB.
So, let's check what we got.
On the route server, we see that a BGP session has been established with all three clients:
We see that we receive prefixes from all clients:
On the as 100 router, we see that if there is only one BGP session with the route server, we receive prefixes from both as 200 and as 300, while the BGP attributes have not changed, as if peering between clients was carried out directly:
Thus, we see that the presence of a route server greatly simplifies the organization of peering on the IXP.
I hope this demo has helped you better understand how traffic exchange points work and how a route server works on an IXP.
Linxdatacenter IX
At Linxdatacenter, we built our own IXP based on a fault-tolerant infrastructure of 2 switches and 2 route servers. Now our IXP is running in test mode, and we invite everyone to connect to Linxdatacenter IX and take part in testing. When connecting, you will be provided with a port with a bandwidth of 1 Gbit / s, the possibility of peering through our route servers, as well as access to the personal account of the IX portal, available at .
Write in comments or private messages to get access to testing.
Hack and predictor Aviator
Traffic exchange points emerged at the dawn of the Internet as a tool for solving the issue of non-optimal traffic between telecom operators. Now, with the advent of new global services and an increase in the amount of CDN traffic, exchange points continue to optimize the operation of the global network. The increase in the number of IXPs in the world benefits both the end user of the service and telecom operators, content operators, etc. For IXP participants, the benefit is expressed in reducing the cost of organizing external peering, reducing the amount of traffic that upstream operators have to pay for, routing optimization, and the ability to have a direct interface with content operators.
Useful links
- View a map of the location of traffic exchange points:
- View detailed statistics on BGP peering, including presence on IXP:
Source: habr.com