Today I will tell you about how the idea of creating a new internal network for our company appeared and was implemented. The position of the management is to make the same full-fledged project for yourself as for the client. If we do well for ourselves, we will be able to invite the customer and show how well what we offer him is arranged and works. Therefore, we approached the development of the concept of a new network for the Moscow office very thoroughly, using the full production cycle: analysis of the needs of departments → selection of a technical solution → design → implementation → testing. So let's start.
Choice of technical solution: mutant sanctuary
The procedure for working on a complex automated system is best described in GOST 34.601-90 “Automated systems. Stages of creation”, so we worked on it. And already at the stages of formation of requirements and development of the concept, we encountered the first difficulties. Organizations of various profiles - banks, insurance companies, software developers, etc. - for their tasks and standards need certain types of networks, the specifics of which are understandable and standardized. However, this will not work with us.
Why?
Jet Infosystems is a large diversified IT company. At the same time, our internal support department is small (but proud), it ensures the performance of basic services and systems. The company contains many departments that perform different functions: these are several powerful outsourcing teams, and in-house developers of business systems, and information security, and architects of computer systems - in general, who is not. Accordingly, their tasks, systems and security policies are also different. Which, as expected, created difficulties in the process of needs analysis and their standardization.
For example, the development department: its employees write and test code for a large number of customers. Often there is a need to quickly organize test environments, and frankly, it is not always possible to form requirements for each project, request resources and build a separate test environment in accordance with all internal regulations. This gives rise to curious situations: once your obedient servant looked into the development room and found a properly working Hadoop cluster of 20 desktops under the table, which was inexplicably connected to the public network. I think it is not necessary to specify that the IT department of the company did not know about its existence. This circumstance, like many others, became the culprits of the fact that during the development of the project we had the term "reserve of mutants", describing the state of the long-suffering infrastructure of the office.
Or here's another example. Periodically, a test stand is set up inside a division. This was the case with Jira and Confluence, which were used to a limited extent by the Software Development Center in some projects. After some time, other departments learned about these useful resources, appreciated them, and at the end of 2018 Jira and Confluence moved from the status of “local toy of programmers” to the status of “company resources”. Now the owner should be assigned to these systems, SLA, access / information security policies, backup policy, monitoring, rules for routing requests for troubleshooting should be defined - in general, all the attributes of a full-fledged information system should be present.
Each of our divisions is also an incubator that grows its own products. Some of them die at the development stage, some we use while working on projects, while others take root and become replicated solutions that we begin to use ourselves and sell to clients. For each such system, it is desirable to have its own network environment, where it will develop without interfering with other systems, and at some point be able to be integrated into the company's infrastructure.
In addition to development, we have a very large with more than 500 employees formed into teams for each customer. They are involved in the maintenance of networks and other systems, remote monitoring, settlement of claims, and so on. That is, the infrastructure of the SC is, in fact, the infrastructure of the customer with whom they are currently working. The peculiarity of working with this section of the network is that their workstations for our company are partly external, and partly internal. Therefore, we implemented the following approach for the SC - the company provides the corresponding department with network and other resources, considering the workstations of these departments as external connections (similar to branches and remote users).
Highway design: we are the operator (surprise)
After evaluating all the pitfalls, we realized that we were getting a telecommunications operator network within one office, and began to act accordingly.
We have created a core network, with the help of which any internal, and in the future, external consumer is provided with the required service: L2 VPN, L3 VPN or regular L3 routing. Some departments need secure Internet access, others need clean access without firewalls, but at the same time protecting our corporate resources and the core network from their traffic.
We informally “signed an SLA” with each division. In accordance with it, all emerging incidents must be eliminated within a certain, pre-agreed period of time. The company's requirements for its network turned out to be strict. The maximum incident response time for telephone and email outages was 5 minutes. The recovery time of the network in case of typical failures is no more than a minute.
Since we have a carrier-class network, you can only connect to it in strict accordance with the rules. Service departments set policies and provide services. They do not even need information about the connections of specific servers, virtual machines and workstations. But at the same time, protection mechanisms are needed, because no connection should disable the network. If a loop is accidentally created, other users should not notice this, that is, an adequate network reaction is necessary. Any telecom operator constantly solves such seemingly complex tasks within its core network. It provides a service to many clients with different needs and traffic. At the same time, different subscribers should not experience inconvenience from the traffic of others.
At home, we solved this problem as follows: we built a backbone L3 network with full redundancy, using the IS-IS protocol. An overlay network was built on top of the reference network based on technology /, using the routing protocol . BFD technology was used to accelerate the convergence of routing protocols.
Network structure
In tests, this scheme proved to be excellent - when any channel or switch is turned off, the convergence time is no more than 0.1-0.2 s, a minimum of packets is lost (often not a single one), TCP sessions are not torn, telephone conversations are not interrupted.
Underlay layer - routing
Overlay layer - routing
Huawei CE6870 switches with VXLAN licenses were used as distribution switches. This device has an optimal price / quality ratio, allows you to connect subscribers at a speed of 10 Gb / s, and connect to the backbone at speeds of 40-100 Gb / s, depending on the transceivers used.
Switches Huawei CE6870
Huawei CE8850 switches were used as core switches. Their task is to quickly and reliably transfer traffic. They do not connect to any devices other than distribution switches, they do not know anything about VXLAN, so a model with 32 ports 40/100 Gb / s was chosen, with a basic license that provides L3 routing and supports IS-IS and MP-BGP protocols .
The lowest one is the Huawei CE8850 core switch
At the design stage, a discussion broke out within the team about technologies that can be used to implement a fault-tolerant connection to the core network nodes. Our Moscow office is located in three buildings, we have 7 cross rooms, in each of which two Huawei CE6870 distribution switches were installed (only access switches were installed in several more cross rooms). When developing the network concept, two redundancy options were considered:
- Fault-tolerant stacking of distribution switches in each cross-room. Pros: simplicity and ease of setup. Cons: higher probability of failure of the entire stack when errors occur in the firmware of network devices (“memory leaks” and the like).
- Apply M-LAG and Anycast gateway technologies to connect devices to distribution switches.
In the end, we settled on the second option. It is somewhat more difficult to set up, but in practice it has shown its performance and high reliability.
Consider first connecting end devices to distribution switches:
cross
An access switch, a server, or any other device that requires a failover connection is included in two distribution switches. M-LAG technology provides redundancy at the link level. It is assumed that two distribution switches appear to the connected equipment as one device. Redundancy and load balancing are carried out using the LACP protocol.
Anycast gateway technology provides redundancy at the network level. A sufficiently large number of VRFs are configured on each of the distribution switches (each VRF is designed for its own purposes - separately for “regular” users, separately for telephony, separately for various test and development environments, etc.), and in each VRF has multiple VLANs configured. In our network, distribution switches are the default gateway for all devices connected to them. The IP addresses corresponding to the VLAN interfaces are the same for both distribution switches. Traffic is routed through the nearest switch.
Now consider connecting distribution switches to the core:
Fault tolerance is provided at the network level, using the IS-IS protocol. Please note that a separate L3 communication line is provided between the switches, at a speed of 100G. Physically, this communication line is a Direct Access cable, you can see it on the right in the photo of Huawei CE6870 switches.
An alternative would be to organize a "fair" fully meshed "double star" topology, but, as mentioned above, we have 7 distribution rooms in three buildings. Accordingly, if we chose the “double star” topology, then we would need exactly twice as many “long-range” 40G transceivers. The savings here are very significant.
A few words need to be said about how VXLAN and Anycast gateway technologies work together. VXLAN, without going into details, is a tunnel for transporting Ethernet frames inside UDP packets. The loopback interfaces of the distribution switches are used as the destination IP address of the VXLAN tunnel. Each cross-connect has two switches with the same addresses of loopback interfaces, respectively, a packet can arrive at any of them, and an Ethernet frame can be extracted from it.
If the switch knows the destination MAC address of the retrieved frame, the frame will be correctly delivered to its destination. The M-LAG mechanism is responsible for ensuring that both distribution switches installed in the same cross-connect have up-to-date information about all MAC addresses “arriving” from access switches, which provides for synchronization of MAC address tables (as well as ARP tables) on both M-LAG-pair switches.
Traffic balancing is achieved due to the presence in the underlay network of several routes to the loopback interfaces of distribution switches.
Instead of a conclusion
As mentioned above, in tests and in operation, the network showed high reliability (recovery time for typical failures is no more than hundreds of milliseconds) and good performance - each cross-connect is connected to the core by two 40 Gb / s channels. Access switches in our network are stacked and connected to distribution switches via LACP/M-LAG with two 10 Gb/s links. The stack usually has 5 switches with 48 ports each, up to 10 access stacks are connected to the distribution in each cross-connect. Thus, the backbone provides about 30 Mbit / s per user even at the maximum theoretical load, which at the time of writing is enough for all our practical applications.
The network allows you to easily organize the pairing of any arbitrary connected devices both via L2 and L3, providing complete isolation of traffic (which the information security service likes) and fault domains (which the operation service likes).
In the next part, we will tell you how we migrated to the new network. Stay tuned!
Maxim Klochkov
Senior Consultant, Network Audit and Complex Projects Group
Network Solutions Center
Jet Infosystems
Source: habr.com