I am sure that all Habr readers at least once ordered goods in online stores abroad and then went to receive parcels at the Russian Post office. Can you imagine the scale of this task in terms of organizing logistics? Multiply the number of buyers by the number of their purchases, imagine a map of our vast country, and on it - more than 40 thousand post offices ... By the way, in 2018, Russian Post processed 345 million international parcels.
In this article, we will tell you what issues the Post faced and how the LANIT-Integration team solved them, creating a new IT infrastructure for data centers.
One of the modern logistics centers of Russian Post
Before the project
Due to a sharp increase in the number of parcels from foreign stores in China, Western Europe and North America, the load on the logistics facilities of the Russian Post has increased. Therefore, a new generation of logistics centers have been built, which use high-capacity sorting machines. They require support from the computing infrastructure.
The data center infrastructure was outdated and did not provide the necessary performance and reliability in the operation of enterprise information systems. Also, Russian Post experienced a lack of computing power to launch new services.
Customer data centers and their problems
Russian Post data centers serve more than 40 objects, 000 territorial offices. Dozens of round-the-clock business services operate in data centers, including e-commerce services.
Already today, the enterprise uses systems for storing, analyzing and processing big data. For such systems, the use of artificial intelligence and machine learning algorithms plays an important role. To date, one of the most important cases for the enterprise is the optimization of logistics flow management and the acceleration of customer service in post offices.
Prior to the start of the upgrade project, there were about 3000 virtual machines in the main and backup data centers, the amount of stored information exceeded 2 petabytes. Data centers had a complex traffic routing structure associated with the division into different segments according to security levels.
With the development of applications and the introduction of new services, the existing bandwidth of network equipment in data centers has become insufficient. A transition to interfaces with new speeds was required: 10 Gb / s, instead of 1 Gb / s for access and 40 Gb / s at the core level, with full redundancy of equipment and communication channels.
From the information security department, a requirement was received to divide the infrastructure into segments with a high level of information security of traffic and applications (PN - Private Network and DMZ - Demilitarized Zone). Firewalls (ITU) passed traffic that was not necessary to filter. VRF was not used on the switches for such traffic. The rules at the ITU were suboptimal (tens of thousands of rules in each data center).
Seamless migration of virtual machines (VMs) between data centers while maintaining the IP address and the optimal path for traffic between segments, including the corporate data network (CDTN), was not possible.
MSTP was used for redundancy, some ports were blocked (hot standby). The core and access switches were not failover clustered, and no interface aggregation (LAG) was used.
With the advent of the third data center, a new architecture and equipment configuration was required to operate the ring between data centers (EVPN was proposed).
There was no single concept for the development of data centers, documented in the form of a project and agreed with all departments of the customer. The current network operation documentation was incomplete and out of date.
Customer expectations
The project team had the following tasks:
- prepare the architecture and development concept for building the network and server infrastructure of the third data center;
- conduct an operational audit of the customer's existing network;
- expand network core capacity by more than 1500 10/40 Gb/s Ethernet ports in each data center (4500 ports in total);
- ensure the operation of the ring between three data centers with the possibility of increasing the speed up to 80 Gb / s in each of the segments in order to combine the customer's computing resources from different data centers into a single IT system;
- provide 100% double reserve of all network elements to achieve the target Uptime at the level of 99,995%;
- minimize traffic delays between virtual machines to speed up business applications;
- collect statistics, analyze and further optimize traffic filtering rules in data centers (initially there were about 80 rules);
- develop a target architecture to ensure seamless migration of the customer's critical business applications to any of the three data centers.
Thus, we had something to work on.
Equipment
Let's take a closer look at what equipment we used in the project.
Firewall (NGWF) USG9560:
- division by VSYS;
- up to 720 Gbps;
- up to 720 million simultaneous sessions;
- 8 slots.
Router NE40E-X8:
- up to 7,08 Tbit/s Switching Capacity;
- up to 2,880 Mpps Forwarding Performance;
- 8 slots for line cards (LPU);
- up to 10M BGP IPv4 routes per MPU;
- up to 1500K OSPF IPv4 routes per MPU;
- up to 3000K - IPv4 FIB (depends on LPU).
CE12800 Series Switches:
- Device Virtualization: VS (1:16 virtualization), Cluster Switch System (CSS), Super Virtual Fabric (SVF);
- Network Virtualization: M-LAG, TRILL, VXLAN and VXLAN bridging, QinQ in VXLAN, EVN (Ethernet Virtual Network);
- starting with VRP V2, EVPN support is included;
- M-LAG - analogue of vPC (virtual Port Channel) for Cisco Nexus;
- Virtual Spanning Tree Protocol (VSTP) - Compatible with Cisco PVST.
CE12804
CE12808
Software
In the project we used:
- converter of configuration files for firewalls of other vendors into command format for new equipment;
- scripts of our own design to optimize and transform the configuration of firewalls.
Appearance of the converter for converting configuration files
Communication scheme between data centers (EVPN VXLAN)
The nuances of setting up equipment
CE12808
- EVPN (standard) instead of EVN (Huawei proprietary) for communication between data centers:
○ L2 over L3 using iBGP in Control plane;
○ MAC training and announcement via iBGP EVPN family (MAC routes, type 2);
○ automatic construction of VXLAN tunnels for broadcast / unknown unicast traffic (Inclusive Multicast Routes, type 3). - Two division modes on VS:
○ based on ports (port-mode port) or based on ASIC (port-mode group, display device port-map);
○ port split dimension interface 40GE ONLY works in Admin VS (regardless of port-mode).
USG9560
- possibility of dividing by VSYS,
- between VSYS dynamic routing and route leaking is impossible!
CE12804
All Active GW (VRRP Master/Master/Master) with MAC VRRP filtering between data centers
acl number 4000
rule 5 deny source-mac 0000-5e00-0100 ffff-ffff-ff00
rule 10 deny destination-mac 0000-5e00-0100 ffff-ffff-ff00
rule 15 permit
interface Eth-Trunk1
traffic-filter acl 4000 outbound
Scheme of interaction of resources between data centers (VXLAN EVPN and All Active GW)
Project complexity
The main difficulty was the need to back up existing applications using computing infrastructure. The customer had more than 100 different applications, some of which were written almost 10 years ago. For example, if Yandex can easily shut down several hundred virtual machines without harming end users, then in Russian Post such an approach would require the development of a number of applications from scratch and changes in the architecture of enterprise information systems. We solved the problems arising in the process of migration and optimization at the stage of a joint audit of the computing infrastructure. All networking technologies new to the enterprise (such as EVPN) have been pre-tested in the laboratory.
Results of the project
The project team included specialists , the customer and his partners in the operation of the computing infrastructure. Dedicated support teams from vendors (Check Point and Huawei) were also formed. The project took two years. Here is what has been done during this time.
- A strategy for the development of a network of data centers, a corporate data transmission network (CSTN) and a ring between data centers was developed and agreed with all departments of the customer.
- Increased service availability. This was noted by the customer's business and led to an even greater increase in traffic due to the introduction of new services.
- More than 40 rules have been migrated and optimized from FWSM/ASA to USG 000. Different ASA contexts on UGG 9560 have been merged into a single security-policy.
- The throughput of data center ports has been increased from 1G to 10/40G through the use of CE12800/CE6850. This made it possible to eliminate interface overloads and loss of packets.
- Carrier-class routers NE40E-X8 fully covered the needs of the customer's data center and KSPD, taking into account future business development.
- Eight new Feature Requests have been requested for USG 9560. Of these, seven have already been implemented and are included in the current version of VRP. 1 FR is under implementation at Huawei R&D. This is a cluster for eight chassis with the ability to configure the necessary functionality for synchronizing the configuration without synchronizing sessions. Required if the traffic delay to one of the data centers is too high (Adler - Moscow 1300 km along the main route and 2800 km along the backup route).
The project has no analogues in comparison with other postal companies in Russia.
The modernization of the data center network infrastructure has opened up new opportunities for the enterprise to develop digital services.
- Providing a personal account and a mobile application for individuals and legal entities.
- Integration with electronic stores to provide goods delivery services.
- Fulfillment is the storage of goods, the formation and delivery of orders from electronic stores.
- Expansion of points of issue of orders, including with the use of partner networks.
- Legally significant document flow with contractors. This will eliminate the slow and costly delivery of paper documents.
- Acceptance of registered letters in electronic form with delivery both in electronic and paper form (with printing of items as close as possible to the final recipient). Service of electronic registered letters on the portal of public services.
- Platform for the provision of telemedicine services.
- Simplified acceptance and simplified delivery of registered postal items using a simple electronic signature.
- Digitization of the post office network.
- Processing of self-service services (terminals and parcel machines).
- Creation of a digital platform for managing the courier service and a new mobile application for courier service customers.
Come to work with us!
Source: habr.com