Evaluate the connections in the middle part of the diagram. We will return to them below.
At some point, you may find that large, complex L2-based networks are irreparably sick. The primary issues stem from handling BUM traffic and the functioning of the STP protocol. Secondly, there’s the generally outdated architecture. This leads to unpleasant problems in the form of downtimes and management inconveniences.
We had two parallel projects where the clients realistically assessed all the pros and cons of the options and chose two different overlay solutions, which we implemented.
We had the opportunity to specifically compare the implementations. Not the operation, which will be relevant in about two to three years.
So, what is a network fabric with overlay networks and SDN?
What to do about the longstanding issues of classic network architecture?
Every year, new technologies and ideas emerge. In practice, the pressing need to reorganize networks has not arisen for quite some time because it’s also possible to do everything by hand using the old trusty methods. So what if we are in the twenty-first century? After all, an admin is supposed to work, not just sit in their office.
Then the boom of large-scale data center construction began. It became clear that the limitations of classic architecture were reached not just in terms of functionality, fault tolerance, and scalability. One solution to these challenges was the idea of building overlay networks over a routable backbone.
In addition, as the scale of networks increases, the problem of managing such fabrics became acute, leading to the emergence of software-defined networking solutions capable of managing the entire network infrastructure as a whole. When the network is managed from a single point, it becomes easier for other IT infrastructure components to interact with it, and automating these interactions becomes simpler.
Virtually every major manufacturer of not only networking equipment but also virtualization has such solutions in its portfolio.
It only remains to determine what will suit which needs. For instance, large companies with a solid development and operational team often find that out-of-the-box solutions from vendors do not meet all their requirements, prompting them to develop their own software-defined (SD) solutions. This is especially true for cloud providers who are continually expanding the range of services offered to their clients, and packaged solutions simply cannot keep up with their demands.
For medium-sized companies, the functionality offered by vendors in the form of packaged solutions is sufficient in 99 percent of cases.
What are overlay networks?
What is the idea behind overlay networks? Essentially, you take a classic routable network and build another network on top of it to gain additional features. Most often, this relates to effectively distributing the load across equipment and communication lines, significantly increasing scalability limits, enhancing reliability, and providing a plethora of security benefits (through segmentation). Additionally, SDN solutions offer very, very, very convenient flexible administration, making the network more transparent for its users.
In general, if local networks had been invented in the 2010s, they would not have looked anything like the legacy systems we inherited from the military in the 1970s.
From the perspective of technologies for building factories using overlay networks, there are currently many implementations from manufacturers and internet projects such as RFC (EVPN+VXLAN, EVPN+MPLS, EVPN+MPLSoGRE, EVPN+Geneve, and others). While there are standards, the implementation of these standards can vary among manufacturers, so completely moving away from vendor lock-in in the creation of such factories is still theoretically possible only on paper.
The situation with SD solutions is even more complex; each vendor has its own vision. There are fully open solutions that can theoretically be modified independently, and there are completely closed solutions.
Cisco offers its own version of SDN for data centers — ACI. Naturally, this is a 100% vendor-lock solution in terms of network equipment selection, but it fully integrates with virtualization, containerization, security, orchestration systems, load balancers, and more. Nevertheless, it is essentially a black box, with no complete access to all internal processes. Not all customers agree to such an option, as you are entirely dependent on the quality of the code written for the solution and its implementation. On the other hand, the manufacturer boasts one of the best technical support teams in the world and has a dedicated team focused solely on this solution. Cisco ACI was chosen as the solution for the first project.
For the second project, a solution based on Juniper was selected. The manufacturer also offers its own SDN for data centers, but the customer decided against implementing SDN. For network construction, an EVPN VXLAN fabric was chosen without the use of centralized controllers.
What is it for
Creating a fabric allows for building a scalable, fault-tolerant, reliable network. The architecture (leaf-spine) considers the specifics data processing centers (traffic paths, minimizing delays and bottlenecks in the network). SD solutions in data centers allow for easy, quick, and flexible management of such a fabric, integrating it into the data center ecosystem.
Both customers needed to build backup data centers to ensure fault tolerance; in addition, traffic between data centers needed to be encrypted.
The first customer had already considered solutions without a fabric as a potential standard for their networks but encountered compatibility issues with STP between multiple vendors' hardware during testing. Downtimes occurred, leading to service outages, which were critical for the customer.
Cisco had already been the client's corporate standard; they considered ACI and other options and decided to go with this solution. They liked the one-click management automation through a single controller. Services configure faster, and management is quicker. They opted to ensure traffic encryption by implementing MACSec between the IPN and SPINE switches. This way, they avoided a bottleneck with a crypto gateway, saved on costs, and maximized bandwidth utilization.
The second client chose a controller-less solution from Juniper since they already had a small installation in their existing data center implementing an EVPN VXLAN fabric. However, it was not redundant (only one switch was used). They decided to expand the infrastructure of the main data center and build a fabric at the backup data center. The existing EVPN was not fully utilized: VXLAN encapsulation was not actually applied, as all hosts were connected to a single switch, and all MAC addresses and /32 addresses of hosts were local, with that same switch serving as the gateway, and there were no other devices where VXLAN tunnels needed to be built. They decided to secure traffic encryption using IPSEC technology between the firewalls (the performance of the MSE was sufficient).
They also explored ACI but decided that vendor lock-in would require purchasing too much hardware, including replacing recently acquired new equipment, which simply made no economic sense. Yes, Cisco's fabric integrates with everything, but only its own devices are possible within the fabric.
On the other hand, as mentioned earlier, mixing an EVPN VXLAN fabric with any neighboring vendor is not straightforward because implementations of the protocol differ. It's like trying to mesh Cisco and Huawei on the same network — while the standards may be shared, it requires extra effort. Since this is a bank, and compatibility tests would take a very long time, they decided it was better to procure from the same vendor now rather than get too carried away with functionality beyond the basics.
Migration Plan
Two data centers based on ACI:
Organization of interaction between data centers. A Multi-Pod solution was chosen — each data center acts as a pod. Requirements for scaling the number of switches and latency between the pods (RTT less than 50 ms) were taken into account. It was decided not to build a Multi-Site solution for easier management (a single management interface is used for the Multi-Pod solution, whereas two interfaces would be required for a Multi-Site solution, or a Multi-Site Orchestrator would be needed), and since geographical redundancy of sites was not required.
From the perspective of migrating services from the Legacy network, the most transparent option was chosen: to gradually transfer VLANs corresponding to specific services.
For migration, a corresponding EPG (End-point-group) was created for each VLAN on the fabric. Initially, the network was stretched between the old network and the fabric over L2, and after all hosts were migrated, the gateway was moved to the fabric, with interaction between the EPG and the existing network carried out via L3OUT. The interaction between L3OUT and EPG was described using contracts. A schematic example:
An approximate structure of most ACI fabric policies is shown in the figure below. All configurations are built on policies nested within other policies, and so on. At first, it is very challenging to understand, but gradually, as practice shows, network administrators get accustomed to this structure in about a month, and only then do they realize how convenient it is.
Comparison
In the Cisco ACI solution, more equipment needs to be purchased (separate switches for Inter-Pod interactions and APIC controllers), which made it more expensive. The Juniper solution did not require the purchase of controllers and auxiliary equipment, allowing partial use of the client's existing hardware.
Here is the architecture of the EVPN VXLAN fabric for the two data centers of the second project:
With ACI, you get a ready-made solution — no digging, no optimization needed. During the initial introduction of the client to the factory, developers and maintenance personnel for code and automation are not necessary. Simple operation is enough, many settings can be done through a wizard, which is not always a plus, especially for those used to the command line. In any case, time is needed to shift one's mindset to new frameworks, to the specifics of settings through policies and managing numerous nested policies. It is also very desirable to have a clear naming structure for policies and objects. If any logic issue arises with the controller, it can only be resolved through technical support.
In EVPN — the console. Suffer or rejoice. A familiar interface for the old guard. Yes, there are standard configurations and guides. You’ll need to study the manuals. Various constructs, everything is clear and detailed.
Naturally, in both cases, it’s best to migrate first the less critical services, such as testing environments, and only after catching all bugs proceed to production. And don’t set it up on a Friday evening. Don’t believe the vendor that everything will be okay, it’s always better to be cautious.
With ACI, you pay more, although Cisco is actively promoting this solution and often offers good discounts, but you save on maintenance. Managing and automating the EVPN factory without a controller requires investments and regular costs — monitoring, automation, integrating new services. At the same time, the initial setup for ACI takes 30–40 percent longer. This is because it takes longer to create the entire set of necessary profiles and policies that will be used later. However, as the network grows, the number of configurations needed decreases. You use already created policies, profiles, and objects. You can flexibly configure segmentation and security, centrally manage contracts that govern the permissions for interactions between EPGs — the workload decreases sharply.
In EVPN, each device in the factory must be configured, increasing the likelihood of errors.
If ACI is slower to implement, then EVPN took almost twice as long to debug. In the case of Cisco, you can always call a support engineer and ask about the network as a whole (because it’s covered as a solution), but with Juniper Networks, you only buy the hardware, and that’s what is covered. Did the packets leave the device? Well, okay, that's your problem. However, you can open a question about the choice of solution or network design — and then they will advise you to purchase professional services for an additional fee.
ACI support is very cool because it's separate: a dedicated team sits just for that. There are, among them, Russian-speaking specialists. The guide is detailed, and the solutions are predetermined. They review and advise. They quickly validate the design, which is often important. Juniper Networks does the same thing, but much slower (it was so for us, but it should be better now by hearsay), which forces you to do everything on your own where you could have been advised by a solution engineer.
Cisco ACI supports integration with virtualization and containerization systems (VMware, Kubernetes, Hyper-V) and centralized management. It also includes network services and security services — load balancing, firewalls, WAF, IPS, and more... Good microsegmentation out of the box. In the second solution, integration with network services is more complicated, so it's better to research forums with those who have done it.
Summary
For each specific case, it is necessary to choose a solution not only based on the cost of the equipment, but also to consider the ongoing operational expenses and the main problems the customer is facing now, along with plans for the development of the IT infrastructure.
ACI turned out more expensive due to the additional equipment, but it’s a ready solution without the need for tweaking; the second solution is more complex and costly in terms of operation, but cheaper.
If you want to discuss how much implementing a network fabric with different vendors might cost and what architecture is needed — we can meet and talk. We will provide advice up to a rough draft of the architecture (with which you can calculate budgets) for free; detailed development, of course, will be paid.
Vladimir Klepche, corporate networks.
Source: habr.com
