🥇The Story of a Switch

In our local network aggregation, we had six pairs of Arista DCS-7050CX3-32S switches and one pair of Brocade VDX 6940-36Q switches. While the Brocade switches were functioning fine in this network, we were preparing for full automation of certain actions, which those switches could not provide. Additionally, we wanted to transition from 40GE interfaces to the capability of using 100GE, to accommodate the next 2-3 years. Thus, we decided to replace the Brocade with Arista.

These switches serve as aggregation switches for each data center. They are directly connected to distribution switches (the second level of aggregation), which aggregate the Top-of-Rack switches in server racks.

Each server is connected to one or two access switches. The access switches are connected to a pair of distribution switches (two distribution switches and two physical links from the access switch to different distribution switches are used for redundancy).

Each server can be utilized by its client, so a separate VLAN is assigned to each client. This VLAN is then mapped to another server of the same client in any rack. The data center consists of several such rows (PODs), with each row having its own distribution switches. These distribution switches are then connected to the aggregation switches.

Clients can order a server in any row, and it is not possible to predict in advance that a server will be assigned or installed in a specific row or particular rack. Therefore, the aggregation switches have around 2500 VLANs in each data center.

The equipment for DCI (Data-Center Interconnect) connects to the aggregation switches. It can be intended for L2 connectivity (a pair of switches forming a VXLAN tunnel to another data center) or for L3 connectivity (two MPLS routers).

As I mentioned before, to unify the processes of automating the configuration of services on equipment in a single data center, it became necessary to replace the central aggregation switches. We installed new switches next to the existing ones, connected them in an MLAG pair, and began preparing for the work. They were immediately connected to the existing aggregation switches, creating a shared L2 domain across all customer VLANs.

Diagram Details

For specificity, let’s name the old aggregation switches A1 and A2, and the new ones — N1 and N2. Let’s assume that in POD 1 and POD 4 are the servers of one client C1, where the client's VLAN is highlighted in blue. This client is using the L2 connectivity service with another data center, therefore their VLAN is routed to a pair of VXLAN switches.

Client C2 hosts servers in POD 2 and POD 3, where the client's VLAN is marked in dark green. This client also utilizes connectivity service with another data center, but L3, hence their VLAN is routed to a pair of L3VPN routers.

We need the client VLANs to understand what is happening at each stage of the replacement works, where connection interruptions occur, and what their potential duration might be. The STP protocol is not used in this scheme, as its tree width becomes large in such a case, and the protocol’s convergence grows geometrically with the number of devices and links between them.

All devices connected by dual links form a stack, an MLAG pair, or a VCS Ethernet fabric. Such technologies are not used for the pair of L3VPN routers since there is no need for L2 redundancy; it is sufficient that they have L2 connectivity to each other via the aggregation switches.

Implementation Options

When analyzing the options for future events, we realized that there are several ways to carry out this work. From a global interruption across the entire local network, to small interruptions lasting literally 1-2 seconds in parts of the network.

Network, halt! Switches, swap out!

The simplest way is, of course, to declare a global connection outage across all PODs and all DCI services and switch all links from the switches A to the switches N.

In addition to the downtime, the duration of which we cannot reliably predict (yes, we know the number of links, but we don't know how many times things might go wrong — from a broken patch cord or damaged connector to a faulty port or transceiver), we also cannot foresee in advance whether the lengths of the patch cords, DACs, AOCs connected to the old switches A will be sufficient to reach the new switches N, which, although standing close by, are still just a bit off to the side, and whether the same transceivers/DACs/AOCs from Brocade switches will work in Arista switches.

And all this under intense pressure from clients and tech support ('Natasha, get up! Natasha, everything is down! Natasha, we’ve already contacted tech support, honestly! Natasha, everything is already crashed! Natasha, how much longer will this be down? Natasha, when will it be back?!'). Even with the pre-announced downtime and notifications sent to clients, a surge of inquiries during such times is guaranteed.

Wait, 1-2-3-4!

And what if we don't announce a global downtime, but instead announce a series of small downtime periods by POD and DCI services. During the first downtime, switch to the switches N only POD 1, then a couple of days later — POD 2, then again a few days after that POD 3, and further POD 4…[N], then the VXLAN switches and then the L3VPN routers.

With this approach to switching work, we reduce the complexity of simultaneous tasks and give ourselves more time to solve problems if something suddenly goes wrong. The connectivity of POD 1 after switching with other PODs and DCI remains intact. However, this work is prolonged, and during the time of these operations, it requires the presence of an engineer for the physical execution of switches in the data center. Moreover, during the work (which is typically performed at night, from 2 to 5 a.m.), there is a need for a highly qualified online network engineer. But we achieve short downtime periods, generally, the work can be carried out within a half-hour interval with a break of up to 2 minutes (in practice, often 20-30 seconds under expected equipment behavior).

In the example provided for the client C1 or the client C2 There will be at least three notifications about the work involving connectivity interruption — the first for work on one POD that contains one of its servers, the second for the second one, and the third during the equipment switch for DCI services.

Switching aggregated communication channels

Why we discuss the expected behavior of the equipment and how aggregated channels can switch with minimized connectivity interruption. Let's imagine the following scenario:

On one side of the link — distribution switches of the POD — D1 and D2, they form an MLAG pair (stack, VCS fabric, vPC pair), on the other side two links — Link 1 and Link 2 — are included in an MLAG pair of old aggregation switches A. On the switch side D an aggregated interface named Port-channel A, on the aggregation switches side A — an aggregated interface named Port-channel D.

Aggregated interfaces use LACP in their operation, meaning that switches on both sides regularly exchange LACPDU packets over both links to ensure that the links are:

operational;
included in one pair of devices on the remote side.

When exchanging packets, the value system-idis carried in the packet, indicating the device to which these links are connected. For an MLAG pair (stack, fabric, etc.), the system-id value for the devices forming the aggregated interface is the same. The switch D1 sends to Link 1 the value system-id D, and the switch D2 sends to Link 2 the value system-id D.

Switches A1 and A2 analyze the LACPDU packets received over one Po D interface and check the matching system-id in them. If the received system-id on any link suddenly differs from the current operational value, then this link is removed from the aggregated interface until the situation is resolved. Currently, we have the current system-id value from the LACP partner on the switches — D , while on the switches A— the current system-id value from the LACP partner — A If it is necessary to switch the aggregated interface, we can proceed in two different ways: D.

Method 1 — Simple

Disconnect both links from switches A.
In this case, the aggregated channel does not work.Reconnect both links sequentially to the switches,

then the LACP operational parameters will be renegotiated, forming the interface NPo D on the switches and transmitting the value N system-id N system-id N.

Method 2 — Minimizing Downtime
Disconnect from switch A2 link Link 2. Meanwhile, traffic between A and D will continue to be transmitted simply through one of the links that remains part of the aggregated interface.

Connect Link 2 to switch N2. On the switch N an aggregated interface is already configured Po DN, and the switch N2 will start transmitting in LACPDU system-id N. At this point, we can already check that the switch N2 is correctly working with the transceiver used for Link 2, that the connection port has transitioned to the state Up, and that no errors occur on the connection port during LACPDU transmission.

However, the fact that the switch D2 for the aggregated interface Po A on the side of Link 2 receives a system-id N value different from the current working value system-id A, does not allow the switches D to introduce Link 2 into the aggregated interface. Po AThe switch N cannot be integrated Link 2 into operation, as it does not receive confirmation of functionality from the LACP partner switch D2. Ultimately, traffic is not transmitted through Link 2 .

And now we disconnect Link 1 from switch A1, thus depriving the switches of A and D a working aggregated interface. Therefore, on the switch side D the current working value of system-id for the interface disappears. Po A.

This allows the switches D and N to agree on exchanging system-id A-N on the interfaces, Po A and Po DNso that traffic starts to be transmitted through the link. Link 2The interruption in this case amounts, in practice, to up to 2 seconds.

And now we smoothly switch Link 1 to switch N1, restoring capacity and redundancy levels of the interfaces. Po A and Po DN. Since the current value of system-id is not changed on either side when connecting this link, there is no interruption.

Additional Links

But the switch can be done without the presence of an engineer at the moment of switching. For this, we need to lay additional links between the distribution switches D and the new aggregation switches. N.

We lay new links between the aggregation switches N and the distribution switches of all PODs. This requires ordering and laying additional patch cords, and installing additional transceivers in N, and in D. We can do this since we have free ports in the switches D of each POD (or we free them up in advance). As a result, each POD is physically connected by two links to the old switches A and the new switches N.

On the switch D two aggregated interfaces have been formed — Po A with links Link 1 and Link 2, and Po N — with links Link N1 and Link N2. At this stage, we check the correctness of the interface and link connections, the optical signal levels at both ends of the links (using DDM information from the switches), and we can even check the link functionality under load or monitor the states of optical signals and transceiver temperatures for a couple of days.

Traffic is still being transmitted through the interface Po A, while the interface Po N is idle. The settings on the interfaces are approximately as follows:

Interface Port-channel A
Switchport mode trunk
Switchport allowed vlan C1, C2

Interface Port-channel N
Switchport mode trunk
Switchport allowed vlan none

Switches D generally support session-based configuration changes, and such models of switches that have this functionality are used. Therefore, we can make changes to the settings of interfaces Po A and Po N in one go:

Configure session
Interface Port-channel A
Switchport allowed vlan none
Interface Port-channel N
Switchport allowed vlan C1, C2
Commit

Then the configuration change will occur quickly, and the downtime will be practically no more than 5 seconds.

This method allows us to carry out all the preparatory work in advance, conduct all necessary checks, coordinate work with the process participants, thoroughly forecast actions needed for the work, without creative flights when 'everything goes wrong', and have a rollback plan to the previous configuration at hand. Work according to this plan is performed by a network engineer without the presence of a data center engineer on-site who physically carries out the switching.

What is also important with this switching method is that all new links have already been set up for monitoring in advance. Errors, activating links in the aggregate, link load — all necessary information is already in the monitoring system, and this has already been visualized on maps.

D-Day

POD

We chose the least painful path for clients, which is least prone to 'something went wrong' scenarios with additional links for switching. Thus, we switched all PODs to the new aggregation switches over a couple of nights.

But we still need to switch the equipment that provides DCI services.

L2

In the case of equipment providing L2 connectivity, we were unable to carry out similar work with additional links. There are at least two reasons for this:

The lack of free ports of the required speed on the VXLAN switches.
The absence of session configuration change functionality on the VXLAN switches.

We decided not to switch the links 'one at a time' with an interruption only during the negotiation of the new system-id pair, as we lacked 100% confidence that the procedure would go smoothly, and laboratory testing showed that if 'something goes wrong', we still experience a connection interruption, and what’s worse — not only for clients with L2 connectivity to other data centers, but for all clients of this data center.

We had conducted promotional work in advance to transition from L2 channels, so the number of clients affected by work on the VXLAN switches was already several times lower than a year ago. As a result, we decided on a connection interruption for the L2 connectivity service on the condition that we maintain the normal operation of local network services in one data center. Moreover, the SLA for this service provides for the possibility of scheduled maintenance with interruption.

L3

Why did we recommend everyone switch to L3VPN for DCI services? One reason is the ability to perform work on one of the routers providing this service, simply by reducing the level of redundancy to N+0, without a connection interruption.

Let’s take a closer look at the service delivery scheme. In this service, the L2 segment goes from client servers only to the L3VPN routers at Selectel. The client network is terminated at the routers.

Each client server, for example, S2 and S3 in the diagram provided, has its private an IP address — 10.0.0.2/24 for server S2 and 10.0.0.3/24 for server S3. The addresses 10.0.0.252/24 and 10.0.0.253/24 are assigned by Selectel to the routers L3VPN-1 and L3VPN-2, respectively. The IP address 10.0.0.254/24 is the VRRP VIP address on the Selectel routers.

More details about the L3VPN service can be found read in our blog.

Until the moment of switching, everything looked approximately as shown in the diagram:

Two routers L3VPN-1 and L3VPN-2 were connected to the old aggregation switch A. The master for the VRRP VIP address 10.0.0.254 is the router L3VPN-1It has a higher priority set for this address than the router. L3VPN-2.

unit 1006 {
    description C2;
    vlan-id 1006;
    family inet {       
        address 10.0.0.252/24 {
            vrrp-group 1 {
                priority 200;
                virtual-address 10.100.0.254;
                preempt {
                    hold-time 120;
                }
                accept-data;
            }
        }
    }
}

Server S2 uses gateway 10.0.0.254 to communicate with servers in other locations. Thus, disconnecting from the L3VPN-2 router (after previously disconnecting it from the MPLS domain) does not affect the connectivity of client servers. At that moment, the level of redundancy in the scheme simply decreases.

After this, we can safely reconnect the router. L3VPN-2 to a pair of switches. NLay the links, change the transceivers. The logical interfaces of the router, which are essential for the operation of client services, remain disabled until we confirm that everything is functioning as it should.

After checking the links, transceivers, signal levels, and error levels on the interfaces, the router is put into operation but already connected to the new pair of switches.

Next, we lower the VRRP priority of the L3VPN-1 router, and the VIP address 10.0.0.254 moves to the L3VPN-2 router. These operations are also performed without interruption in connectivity.

The transfer of the VIP address 10.0.0.254 to the router L3VPN-2 allows us to disconnect the router L3VPN-1 without interrupting connectivity for the client and connect it to the new pair of aggregation switches. N.

Whether to return the VRRP VIP to the L3VPN-1 router or not is another question, and even if we return it, it is done without interrupting connectivity.

Total

After all these actions, we indeed replaced the aggregation switches in one of our data centers while minimizing interruptions for our clients.

Next, all that remains is the dismantling. Dismantling the old switches, dismantling the old links between switches A and D, dismantling transceivers from these links, correcting monitoring, and fixing the network scheme in documentation and monitoring.