Hello, my name is Kostya Kramlih, I am the lead developer of the Virtual Private Cloud division at Yandex.Cloud. I work with virtual networks, and as you might guess, in this article I will talk about the structure of Virtual Private Cloud (VPC) in general and virtual networks in particular. You will also learn why we, the service developers, value feedback from our users. But let's take it step by step.
What is VPC?
Nowadays, there are many options for deploying services. I am sure some people still have a server under the administrator's desk, although I hope such stories are becoming less common.
Currently, services are trying to move to public clouds, and this is where they encounter VPC. VPC is a part of the public cloud that connects user, infrastructure, platform, and other resources into one, regardless of whether they are located within our Cloud or outside it. Moreover, VPC allows you to keep these resources from being exposed to the internet unnecessarily, they remain within your isolated network.
How a virtual network looks from the outside
In the context of VPC, we primarily understand it as an overlay network and network services like VPNaaS, NATaaS, LBaaS, etc. And all of this operates over a fault-tolerant network infrastructure, which was already discussed in an excellent article here on Habr.
Let’s take a closer look at the virtual network and its structure.
Let’s consider two availability zones. We provide a virtual network – what we call VPC. In fact, it defines the uniqueness space of your ‘gray’ addresses. Within each virtual network, you have complete control over the address space that you can assign to computational resources.
The network is global. It is projected onto each availability zone as an entity called a Subnet. For each Subnet, you assign a CIDR size of 16 or less. Each availability zone can have more than one such entity, with transparent routing always present between them. This means that all your resources within a single VPC can 'communicate' with each other, even if they are in different availability zones. They can 'communicate' without going out to the internet, using our internal channels, 'thinking' they are within a single private network.
The diagram above shows a typical scenario: two VPCs that overlap somewhere in their addresses. Both may belong to you. For instance, one for development and the other for testing. There could simply be different users — in this case, it doesn’t matter. Each VPC has one virtual machine plugged into it.
Let’s complicate the diagram. You can configure one virtual machine to connect to multiple Subnets. And not just that, but across different virtual networks.
If you need to expose machines to the internet, you can do this via API or UI. You need to set up NAT translation for your 'gray' internal address to a 'white' public address. You cannot choose the 'white' address; it is randomly assigned from our pool of addresses. Once you stop using the external IP, it returns to the pool. You only pay for the time you use the 'white' address.
There is also the option to grant a machine internet access through a NAT instance. You can route traffic to the instance through a static routing table. We have accounted for this case, as it is needed by some users, and we know this. Accordingly, we have a specially configured NAT image in our catalog.
However, even when a ready-made NAT image is available, the setup can be complicated. We understood that this might not be the most convenient option for some users, which is why we ultimately made it possible to enable NAT for the desired Subnet with just one click. This feature is still in closed preview access, being tested with community participants.
How a virtual network is structured internally
How does the user interact with the virtual network? The network looks outward through its API. The user comes to the API and works with the desired state. Through the API, the user sees how everything should be structured and configured, while also seeing the status of how much the actual state differs from the desired one. This is the user's picture. But what happens inside?
We record the desired state in Yandex Database and proceed to configure different parts of our VPC. The overlay network in Yandex.Cloud is built on selected components of OpenContrail, which has recently been renamed Tungsten Fabric. Network services are implemented on a single platform, CloudGate. In CloudGate, we also used several open-source components: GoBGP for handling control information, and VPP for implementing a software router that operates over DPDK for the data path.
Tungsten Fabric communicates with CloudGate through GoBGP. It reports what happens in the overlay network. CloudGate, in turn, connects overlay networks to each other and to the internet.
Now let's look at how the virtual network addresses scalability and availability challenges. Consider a simple case. There is one availability zone containing two VPCs. We deployed one instance of Tungsten Fabric, which can handle several tens of thousands of networks. The networks are linked to CloudGate. As mentioned, CloudGate ensures their connectivity with each other and the internet.
Suppose a second availability zone is added. It should fail completely independently of the first. Therefore, in the second availability zone, we need to deploy a separate instance of Tungsten Fabric. This will be a separate system dealing with the overlay and knowing little about the first system. The visibility that our virtual network is global is created by our VPC API; that is its task.
VPC1 is projected into availability zone B if there are resources in availability zone B that connect to VPC1. If there are no resources from VPC2 in availability zone B, we do not materialize VPC2 in that zone. Conversely, since resources from VPC3 exist only in zone B, VPC3 does not exist in zone A. It's simple and logical.
Let's delve a bit deeper and see how a specific host is organized in Y.Cloud. The main point to note is that all hosts are structured the same way. We ensure that only the essential minimum of services runs on the hardware, while all others operate on. virtual machines. We build higher-order services based on basic infrastructure services and use the Cloud to address some engineering tasks, such as in Continuous Integration.
If we look at a specific host, we will see that three components run in the host's OS:
- Compute – the part responsible for distributing computational resources on the host.
- VRouter – part of Tungsten Fabric, which organizes the overlay, meaning it tunnels packets through the underlay.
- VDisk – these are pieces of storage virtualization.
In addition to this, virtual machines services are running: Cloud infrastructure services, platform services, and customer capacities. Customer resources and platform services always go through the overlay via VRouter.
Infrastructure services can hook into the overlay, but mainly prefer to operate in the underlay. In the underlay, they connect using SR-IOV. Essentially, we slice the card into virtual network cards (virtual functions) and plug them into infrastructure VMs to avoid losing performance. For example, that very CloudGate runs as one of these infrastructure virtual machines.
Now that we have described the global tasks of the virtual network and the structure of the basic cloud components, let's look at how different parts of the virtual network interact with each other.
We identify three layers in our system:
- Config Plane – defines the desired state of the system. This is what the user configures via the API.
- Control Plane – ensures the user-defined semantics, meaning it brings the Data Plane state to what was described by the user in the Config Plane.
- Data Plane – directly processes the user's packets.
As I mentioned earlier, it all starts when a user or an internal platform service accesses the API and describes a specific desired state.
This state is immediately recorded in the Yandex Database, returns the ID of the asynchronous operation via the API, and initiates our internal machinery to deliver the state the user desired. Configuration tasks are sent to the SDN controller, informing Tungsten Fabric of what needs to be done in the overlay. For example, they reserve ports, virtual networks, and so on.
The Config Plane in Tungsten Fabric delivers the required state to the Control Plane. It is also through this that the Config Plane communicates with the hosts, indicating what will soon run on them.
Now let's look at how the system appears on the hosts. In virtual machine there is a network adapter plugged into the VRouter. The VRouter is a core module of Tungsten Fabric that examines the packets. If there is already a flow for a given packet, the module processes it. If there's no flow, the module performs what is known as punting, meaning it sends the packet to a user-mode process. This process parses the packet and either responds to it directly, as with DHCP and DNS, or instructs the VRouter on what to do with it. After this, the VRouter can handle the packet.
Traffic between virtual machines within a single virtual network flows transparently; it doesn't route through the CloudGate. The hosts that have the virtual machines deployed communicate directly with each other. They tunnel traffic and pass it to one another through the underlay.
The Control Plane communicates with each other between availability zones via BGP, just like with another router. They relay information about which machines are running where, allowing virtual machines in one zone to directly interact with other virtual machines.
Additionally, the Control Plane communicates with CloudGate. It similarly reports where and which virtual machines are running and their addresses. This enables external traffic and traffic from load balancers to be directed to them.
Traffic exiting the VPC arrives at CloudGate in the data path, where it is quickly processed by VPP with our plugins. The traffic is then sent either to other VPCs or out to border routers, which are configured through the Control Plane of CloudGate itself.
Plans for the near future
In summary of all that has been said above in a few sentences, we can state that VPC in Yandex.Cloud addresses two important tasks:
- It provides isolation between different clients.
- Combines resources, infrastructure, platform services, other clouds, and on-premise into a single network.
And to address these tasks effectively, scalability and resilience at the internal architecture level must be ensured, which is what VPC does.
Gradually, VPC is acquiring new functions; we are implementing new capabilities and striving to improve user convenience. Some ideas are voiced and make it to the priority list thanks to the participants of our community.
Here is approximately the list of our plans for the near future:
- VPN as a service.
- Private DNS instances – images for the quick setup of virtual machines with a pre-configured DNS server.
- DNS as a service.
- Internal load balancer.
- Adding a 'white' IP address without recreating the virtual machine.
The balancer and the ability to switch the IP address for an already created virtual machine ended up on this list at the request of users. To be honest, without explicit feedback we would have approached these functions a bit later. Now we are already working on the task regarding addresses.
Initially, the 'white' IP address could only be added at the creation of the machine. If the user forgot to do this, the virtual machine had to be recreated. The same applied if there was a need to remove the external IP. Soon it will be possible to enable and disable the public IP without recreating the machine.
Feel free to share your ideas and support the suggestions of other users. You help us make Cloud better and receive important and useful features faster!
Source: habr.com
