The scale of the Amazon Web Services network consists of 69 zones across 22 regions worldwide: the USA, Europe, Asia, Africa, and Australia. Each zone contains up to 8 data centers (DCs), and each DC holds thousands or hundreds of thousands of servers. The network is built to account for all unlikely scenarios of outages. For example, all regions are isolated from each other, and availability zones are distributed several kilometers apart. Even if a cable is cut, the system will switch to backup channels, and data loss will amount to mere packets. More about the principles on which the network is built and its structure will be explained by Vasily Pantyukhin.
Vasily Pantyukhin started as a Unix admin in .ru companies, spent 6 years working with large hardware from Sun Microsystems, and preached datacenter-centricity at EMC for 11 years. He naturally evolved into private clouds, then moved to public ones. Now, as an architect at Amazon Web Services, he provides technical advice to help thrive and grow in the AWS cloud.
In the previous part of the trilogy about the structure of AWS, Vasily delved into the architecture of physical servers and database scaling. Nitro cards, a custom hypervisor based on KVM, and the Amazon Aurora database — all this is covered in the article "". Read it to immerse yourself in the context, or watch the presentations.
This part will discuss network scaling — one of the most complex systems in AWS. The evolution from a flat network to a Virtual Private Cloud and its architecture, internal services like Blackfoot and HyperPlane, the noisy neighbor problem, and finally — the scale of the network, backbone, and physical cables. All this is covered below.
Disclaimer: everything below is Vasily's personal opinion and may not reflect the views of Amazon Web Services.
Network Scaling
AWS cloud was launched in 2006. Its network was quite primitive — with a flat structure. The range of private addresses was shared among all tenants of the cloud. When launching a new virtual machine, you would randomly receive an available IP address from this range.
This approach was simple to implement but fundamentally limited cloud usage. In particular, it made it quite challenging to develop hybrid solutions that merged private networks on-premises and in AWS. The most common issue was overlapping IP address ranges.
Virtual Private Cloud
The cloud has proven to be in demand. It's time to consider scalability and the ability to use it for tens of millions of tenants. A flat network has become the main obstacle. Therefore, we thought about how to isolate users from each other at the network level so that they could independently choose IP ranges.
What comes to mind first when you think of network isolation? Of course, VLAN and VRF — Virtual Routing and Forwarding.
Unfortunately, this didn't work. VLAN ID is only 12 bits, which gives us just 4096 isolated segments. Even in the largest switches, a maximum of 1-2 thousand VRFs can be used. The combination of VRF and VLAN gives us only a few million subnets. This is definitely insufficient for tens of millions of tenants, each of whom should be able to use multiple subnets.
Moreover, we simply cannot afford to buy the required number of large boxes, for example, from Cisco or Juniper. There are two reasons: it’s incredibly expensive, and we do not want to become dependent on their development and patching policies.
One conclusion is clear – we need to build our own solution.
In 2009, we announced VPC — Virtual Private Cloud. The name took hold, and now many cloud providers use it as well.
VPC is a virtual network SDN (Software Defined Network). We decided not to invent special protocols at L2 and L3 levels. The network operates on standard Ethernet and IP. To transmit, the traffic of virtual machines is encapsulated in a wrapper of our own protocol. It specifies the ID that belongs to the VPC tenant.
It sounds simple. However, several serious technical challenges need to be addressed. For example, where and how to store data on the mapping of virtual MAC/IP addresses, VPC IDs, and corresponding physical MAC/IP. At the scale of AWS, this is a huge table that must work with minimal latency when accessed. This is handled by the mapping service, which is spread thinly across the entire network.
In next-generation machines, encapsulation is performed by Nitro cards at the hardware level. In older instances, encapsulation and decapsulation are done through software.
Let's understand how this works in general terms. Starting with the L2 level, suppose we have a virtual machine with IP 10.0.0.2 located on a physical server 192.168.0.3. It sends data to another virtual machine 10.0.0.3, which resides on 192.168.1.4. An ARP request is formed and reaches the network Nitro card. For simplicity, let's assume both virtual machines are within the same 'blue' VPC.
The card replaces the source address with its own and forwards the ARP frame to the mapping service.
The mapping service returns the information necessary for transmission over the physical L2 network.
The Nitro card replaces the MAC in the physical network with the address in the VPC in the ARP response.
When data is transmitted, we wrap logical MAC and IP in a VPC envelope. We send all of this over the physical network using the respective IPs of the source and destination Nitro cards.
The physical machine intended for the packet performs a check. This is to prevent the possibility of address spoofing. The machine sends a special request to the mapping service asking, 'From the physical machine 192.168.0.3, I've received a packet destined for 10.0.0.3 in the
The mapping service checks its resource mapping table and either allows or denies the passage of the packet. In all new instances, additional validation is embedded in the Nitro cards. It cannot be circumvented even theoretically. Therefore, spoofing resources in another VPC will not work.
Then, the data is sent to the virtual machine for which it is intended.
The mapping service also acts as a logical router for data transmission between virtual machines in different subnets. Conceptually, it is straightforward, and I won't delve into details.
Thus, with each packet transmission, servers refer to the mapping service. How do we handle the inevitable delays? By caching., of course.
The beauty of it is that it's not necessary to cache the entire massive table. The physical server hosts virtual machines from a relatively small number of VPCs. Information needs to be cached only about these VPCs. Data transmission to other VPCs in 'default' configuration is still not legitimate. If functionalities like VPC peering are used, information about the corresponding VPCs is additionally loaded into the cache.
We've got a handle on data transmission in the VPC.
Blackfoot
What to do when traffic needs to be sent externally, for example, to the Internet or through a VPN? Here we are helped by Blackfoot — an internal AWS service. It was developed by our South African team. Thus, the service is named after the penguin that lives in South Africa.
Blackfoot decapsulates traffic and handles it as required. Data is sent to the Internet as is.
Data is decapsulated and re-encapsulated in an IPsec wrapper when using a VPN.
When using Direct Connect, traffic is tagged and sent to the appropriate VLAN.
HyperPlane
This is an internal flow control service. Many network services require control of data flow state. For example, when using NAT, flow control must ensure that each pair of "IP: destination port" corresponds to a unique outgoing port. In the case of a load balancer, NLB — Network Load Balancer, the data flow must always be directed to the same target virtual machine. Security Groups act as a stateful firewall. They monitor incoming traffic and implicitly open ports for outgoing packet flows.
In the AWS cloud, latency requirements are extremely high. Therefore, HyperPlane it is critical for the functionality of the entire network.
HyperPlane is built on EC2 virtual machines. There is no magic here, just cleverness. The trick is that these are virtual machines with large RAM. The transactions are conducted solely in memory. This allows achieving latencies of just a few microseconds. Working with disk would kill all performance.
HyperPlane is a distributed system made up of a vast number of such EC2 machines. Each virtual machine has a throughput of 5 GB/s. Across the entire regional network, this provides immense terabits of bandwidth and allows processing millions of connections per second..
HyperPlane works only with streams. VPC packet encapsulation is completely transparent to it. Any potential vulnerability in this internal service will still not breach the isolation of the VPC. Security is managed at lower levels.
Noisy neighbor
There is also the issue of the noisy neighbor — noisy neighborAssuming we have 8 nodes. These nodes handle traffic for all cloud users. It seems fine, and the load should be evenly distributed across all nodes. The nodes are very powerful, and overloading them is difficult.
However, we design our architecture even considering unlikely scenarios.
A low probability does not mean impossibility.
We can imagine a situation where one or more users generate too much load. All HyperPlane nodes are involved in handling this load, and other users may potentially feel some performance degradation. This undermines the cloud concept, where tenants cannot influence one another.
How can we solve the noisy neighbor problem? The first thing that comes to mind is sharding. Our 8 nodes are logically divided into 4 shards with 2 nodes each. Now the noisy neighbor will only affect a quarter of all users, but significantly.
Let's approach it differently. We'll allocate just 3 nodes per user.
The trick is to assign nodes to different users randomly. In the image below, the blue user shares nodes with one of two other users — the green and orange ones.
With 8 nodes and 3 users, the probability of the noisy neighbor overlapping with one of the users is 54%. This is the likelihood that the blue user will impact other tenants, but only with part of their load. In our example, this influence will not be noticeable to everyone, but only to a third of all users. This is already a decent result.
The number of users who will overlap
The probability in percentage
0
18%
1
54%
2
26%
3
2%
Let's bring the situation closer to reality — take 100 nodes and 5 users spread across 5 nodes. In this case, none of the nodes will overlap with a probability of 77%.
The number of users who will overlap
The probability in percentage
0
77%
1
21%
2
1,8%
3
0,06%
4
0,0006%
5
0,00000013%
In a real situation, with a vast number of HyperPlane nodes and users, the potential impact of a noisy neighbor on other users is minimal. This method is called shuffle sharding — . It minimizes the negative effect of node failures.Many services are built on HyperPlane: Network Load Balancer, NAT Gateway, Amazon EFS, AWS PrivateLink, AWS Transit Gateway.
The scale of the network
Network Scale
Now let's talk about the scale of the network itself. As of October 2019, AWS offers its services in 22 regions, with 9 more planned.
- Each region contains several Availability Zones — AZ. There are a total of 69 worldwide.
- Each AZ consists of Data Centers. There are no more than 8 of them.
- The data centers house an enormous number of servers, some with up to 300,000.
Now let's average all this, multiply, and we will get an impressive number that reflects the scale of the Amazon cloud..
Between the availability zones and data centers, there are many optical links. In one of our largest regions alone, there are 388 links for communication between AZs and transit centers with other regions. In total, this gives a staggering 5000 Tbps..
The AWS backbone is specifically built for the cloud and optimized for its operations. We build it on links 100 Gbps.. We control them completely, except in the regions within China. The traffic is not shared with loads from other companies.
Of course, we are not the only cloud provider with a private backbone network. More and more large companies are going down this path. This is confirmed by independent researchers, such as from .
The graph shows that the share of content providers and cloud providers is growing. As a result, the share of Internet traffic from backbone providers is continuously decreasing.
Let me explain why this is happening. Previously, most web services were available and consumed directly from the Internet. Now, more and more servers are located in the cloud and accessed through CDN — Content Distribution Network.To access a resource, a user goes through the Internet only to the nearest CDN PoP — Point of Presence.Most often, this is somewhere nearby. After that, it leaves the public Internet and travels over a private backbone across the Atlantic, for example, directly to the resource.
It's interesting to think about how the Internet will change in 10 years if this trend continues.
Physical channels.
Scientists have not yet figured out how to increase the speed of light in the universe, but they have made significant advances in the methods of transmitting it through fiber optics. Currently, we use cables with 6912 fibers. This helps to significantly optimize the cost of their installation.
In some regions, we have to use special cables. For example, in the Sydney region, we use cables with special termite-resistant coating.
No one is immune to troubles, and sometimes our channels get damaged. In the photo to the right, you can see optical cables in one of the American regions that were cut by builders. As a result of the incident, only 13 data packets were lost, which is remarkable. Once again — just 13! The system literally switched to backup channels instantly — the scale works.
We took a quick look at some Amazon cloud services and technologies. I hope you now have at least some idea of the scale of the challenges that our engineers face. Personally, I find this very exciting.
This is the final part of the trilogy by Vasily Pantiukhin about the AWS infrastructure. In part, server optimization and database scaling were described, and in — serverless functions and Firecracker.
At In November, Vasily Pantiukhin will share new details about Amazon's infrastructure. He discusses the reasons for failures and designing distributed systems at Amazon. Until October 24, you can still a ticket at a good price, and pay later. We look forward to seeing you at HighLoad++, come — let's chat!
Source: habr.com
