Judging by the number of questions that began to arrive to us via SD-WAN, the technology began to take root thoroughly in Russia. Vendors, of course, do not sleep and offer their concepts, and some brave pioneers are already implementing them on their networks.
We work with almost all vendors, and for several years in our laboratory, I managed to delve into the architecture of every major developer of software-defined solutions. SD-WAN from Fortinet stands a little apart here, which simply built the functionality of balancing traffic between communication channels into firewall software. The solution is rather democratic, so it is usually looked at in companies where they are not yet ready for global changes, but want to use their communication channels more efficiently.
In this article, I want to tell you how to set up and work with Fortinet SD-WAN, who this solution is suitable for, and what pitfalls you can encounter here.
The most prominent players in the SD-WAN market can be classified into one of two types:
1. Startups that created SD-WAN solutions from scratch. The most successful of these receive a huge impetus to development being bought by large companies - this is the story of Cisco / Viptela, VMWare / VeloCloud, Nuage / Nokia
2. Large network vendors who created SD-WAN solutions, developing the programmability and manageability of their traditional routers - this is the story of Juniper, Huawei
Fortinet has found its way. The firewall software had a built-in functionality that allows you to combine their interfaces into virtual channels and balance the load between them using complex, compared to the usual routing, algorithms. This functionality is called SD-WAN. Can what Fortinet have done be called SD-WAN? An understanding is gradually emerging on the market that Software-Defined means the separation of the Control Plane from the Data Plane, dedicated controllers, orchestrators. Fortinet has none of that. Centralized management is optional and offered through the traditional Fortimanager tool. But in my opinion, you should not look for abstract truth and waste time arguing about terms. In the real world, either approach has its advantages and disadvantages. The best way out is to understand them and be able to choose solutions that match the tasks.
I’ll try to tell you with screenshots in my hands how it looks and what Fortinet SD-WAN can do.
How it all works
Suppose you have two branches connected by two data channels. These data channels are combined into a group, similar to how ordinary Ethernet interfaces are combined into a LACP-Port-Channel. Old-timers will remember PPP Multilink - also a suitable analogy. Channels can be physical ports, VLAN SVIs, and VPN or GRE tunnels.
VPN or GRE is typically used when connecting branch LANs over the Internet. And physical ports - if there are L2 connections between sites, or when connecting over a dedicated MPLS / VPN, if we are satisfied with communication without Overlay and encryption. Another scenario in which physical ports are used in an SD-WAN group is balancing the local access of users to the Internet.
Our booth has four firewalls and two VPN tunnels operating through two "carriers". The schema looks like this:
VPN tunnels are configured in interface mode, so that it looks like Point-to-Point connections between devices with IP addresses on P2P interfaces that can be pinged and verified that the connection through a particular tunnel is working. In order for the traffic to be encrypted and go to the opposite side, it is enough to route it into the tunnel. The alternative, selecting traffic for encryption using subnet lists, is highly confusing for the administrator as the configuration becomes more complex. In a large network, you can use ADVPN technology to build a VPN, this is an analogue of DMVPN from Cisco or DVPN from Huawei, which makes it easy to set up.
Site-to-Site VPN config for two devices with BGP routing on both sides
«ЦОД» (DC)
«Филиал» (BRN)
config system interface
edit "WAN1"
set vdom "Internet"
set ip 1.1.1.1 255.255.255.252
set allowaccess ping
set role wan
set interface "DC-BRD"
set vlanid 111
next
edit "WAN2"
set vdom "Internet"
set ip 3.3.3.1 255.255.255.252
set allowaccess ping
set role lan
set interface "DC-BRD"
set vlanid 112
next
edit "BRN-Ph1-1"
set vdom "Internet"
set ip 192.168.254.1 255.255.255.255
set allowaccess ping
set type tunnel
set remote-ip 192.168.254.2 255.255.255.255
set interface "WAN1"
next
edit "BRN-Ph1-2"
set vdom "Internet"
set ip 192.168.254.3 255.255.255.255
set allowaccess ping
set type tunnel
set remote-ip 192.168.254.4 255.255.255.255
set interface "WAN2"
next
end
config vpn ipsec phase1-interface
edit "BRN-Ph1-1"
set interface "WAN1"
set local-gw 1.1.1.1
set peertype any
set net-device disable
set proposal aes128-sha1
set dhgrp 2
set remote-gw 2.2.2.1
set psksecret ***
next
edit "BRN-Ph1-2"
set interface "WAN2"
set local-gw 3.3.3.1
set peertype any
set net-device disable
set proposal aes128-sha1
set dhgrp 2
set remote-gw 4.4.4.1
set psksecret ***
next
end
config vpn ipsec phase2-interface
edit "BRN-Ph2-1"
set phase1name "BRN-Ph1-1"
set proposal aes256-sha256
set dhgrp 2
next
edit "BRN-Ph2-2"
set phase1name "BRN-Ph1-2"
set proposal aes256-sha256
set dhgrp 2
next
end
config router static
edit 1
set gateway 1.1.1.2
set device "WAN1"
next
edit 3
set gateway 3.3.3.2
set device "WAN2"
next
end
config router bgp
set as 65002
set router-id 10.1.7.1
set ebgp-multipath enable
config neighbor
edit "192.168.254.2"
set remote-as 65003
next
edit "192.168.254.4"
set remote-as 65003
next
end
config network
edit 1
set prefix 10.1.0.0 255.255.0.0
next
end
config system interface
edit "WAN1"
set vdom "Internet"
set ip 2.2.2.1 255.255.255.252
set allowaccess ping
set role wan
set interface "BRN-BRD"
set vlanid 111
next
edit "WAN2"
set vdom "Internet"
set ip 4.4.4.1 255.255.255.252
set allowaccess ping
set role wan
set interface "BRN-BRD"
set vlanid 114
next
edit "DC-Ph1-1"
set vdom "Internet"
set ip 192.168.254.2 255.255.255.255
set allowaccess ping
set type tunnel
set remote-ip 192.168.254.1 255.255.255.255
set interface "WAN1"
next
edit "DC-Ph1-2"
set vdom "Internet"
set ip 192.168.254.4 255.255.255.255
set allowaccess ping
set type tunnel
set remote-ip 192.168.254.3 255.255.255.255
set interface "WAN2"
next
end
config vpn ipsec phase1-interface
edit "DC-Ph1-1"
set interface "WAN1"
set local-gw 2.2.2.1
set peertype any
set net-device disable
set proposal aes128-sha1
set dhgrp 2
set remote-gw 1.1.1.1
set psksecret ***
next
edit "DC-Ph1-2"
set interface "WAN2"
set local-gw 4.4.4.1
set peertype any
set net-device disable
set proposal aes128-sha1
set dhgrp 2
set remote-gw 3.3.3.1
set psksecret ***
next
end
config vpn ipsec phase2-interface
edit "DC-Ph2-1"
set phase1name "DC-Ph1-1"
set proposal aes128-sha1
set dhgrp 2
next
edit "DC2-Ph2-2"
set phase1name "DC-Ph1-2"
set proposal aes128-sha1
set dhgrp 2
next
end
config router static
edit 1
set gateway 2.2.2.2
et device "WAN1"
next
edit 3
set gateway 4.4.4.2
set device "WAN2"
next
end
config router bgp
set as 65003
set router-id 10.200.7.1
set ebgp-multipath enable
config neighbor
edit "192.168.254.1"
set remote-as 65002
next
edit "192.168.254.3"
set remote-as 65002
next
end
config network
edit 1
set prefix 10.200.0.0 255.255.0.0
next
end
I give the config in text form, because, in my opinion, it is more convenient to configure VPN this way. Almost all settings are the same on both sides, in text form they can be made copy-paste. If you do the same in the web interface, it's easy to make a mistake - forget a tick somewhere, enter the wrong value.
After we added interfaces to the bundle
all routes and security policies can refer to it, and not to the interfaces included in it. At a minimum, you need to allow traffic from internal networks to SD-WAN. When you create rules for them, you can apply protective measures such as IPS, antivirus, and HTTPS disclosure.
For the bundle, SD-WAN Rules are configured. These are the rules that define the balancing algorithm for specific traffic. They are similar to the routing policies in Policy-Based Routing, only as a result of traffic falling under the policy, not the next-hop or the usual outgoing interface is installed, but the interfaces added to the SD-WAN bundle plus the traffic balancing algorithm between these interfaces.
Traffic can be separated from the general flow by L3-L4 information, by recognized applications, Internet services (URL and IP), as well as by recognized users of workstations and laptops. After that, one of the following balancing algorithms can be assigned to dedicated traffic:
In the Interface Preference list, those interfaces from those already added to the bundle that will serve this type of traffic are selected. By adding not all interfaces, you can limit which channels you use, say, e-mail, if you do not want to load expensive channels with a high SLA. In FortiOS 6.4.1, it became possible to group interfaces added to the SD-WAN bundle into zones, creating, for example, one zone for communication with remote sites, and another for local Internet access using NAT. Yes, yes, the traffic that goes to the regular Internet can also be balanced.
About balancing algorithms
In terms of how Fortigate (a firewall from Fortinet) can split traffic between channels, there are two interesting options that are not very common on the market:
Lowest Cost (SLA) – from all interfaces that meet the SLA at the moment, the one with a lower weight (cost) manually set by the administrator is selected; this mode is suitable for "bulky" traffic such as backups and file transfers.
Best Quality (SLA) – this algorithm, in addition to the usual delay, jitter and Fortigate packet loss, can also use the current channel load to assess the quality of channels; this mode is suitable for "sensitive" traffic such as voice over IP (VoIP) and video conferencing.
These algorithms require setting up a communication channel performance meter - Performance SLA. This meter periodically (check interval) monitors information on SLA compliance: packet loss, latency and jitter in the communication channel, and can “reject” those channels that do not currently meet the quality thresholds – Lose too many packets or generate too much delay. In addition, the meter monitors the status of the channel, and can temporarily remove it from the bundle in case of repeated loss of responses (failures before inactive). When restoring, after several successive responses (restore link after), the meter will automatically return the channel to the bundle, and data will begin to be transmitted over it again.
This is how the "meter" setting looks like:
In the web interface, ICMP-Echo-request, HTTP-GET and DNS request are available as a test protocol. There are a little more options on the command line: TCP-echo and UDP-echo options are available, as well as a specialized quality measurement protocol - TWAMP.
Measurement results can also be seen in the web interface:
And on the command line:
Troubleshooting
If you have created a rule, but everything does not work as expected, you should look at the Hit Count value in the SD-WAN Rules list. It will show if the traffic falls into this rule at all:
On the settings page of the meter itself, you can see the change in the channel parameters over time. The dotted line indicates the threshold value of the parameter
In the web interface, you can see how traffic is distributed by the amount of transmitted/received data and the number of sessions:
In addition to all this, there is an excellent opportunity to track the passage of packets with maximum detail. When working in a real network, a device configuration accumulates a lot of routing, firewalling, and traffic distribution policies on SD-WAN ports. All this interacts with each other in a complex way, and although the vendor provides detailed block diagrams of packet processing algorithms, it is very important to be able not to build and test theories, but to see where the traffic really goes.
For example, the following set of commands
diagnose debug flow filter saddr 10.200.64.15
diagnose debug flow filter daddr 10.1.7.2
diagnose debug flow show function-name
diagnose debug enable
diagnose debug trace 2
Allows tracking two packets with source address 10.200.64.15 and destination address 10.1.7.2.
We ping 10.7.1.2 from 10.200.64.15 twice and look at the output on the console.
First package:
Second package:
Here is the first packet received by the firewall:
id=20085 trace_id=475 func=print_pkt_detail line=5605 msg="vd-Internet:0 received a packet(proto=1, 10.200.64.15:42->10.1.7.2:2048) from DMZ-Office. type=8, code=0, id=42, seq=0."
VDOM – Internet, Proto=1 (ICMP), DMZ-Office – название L3-интерфейса. Type=8 – Echo.
A new session has been created for it:
msg="allocate a new session-0006a627"
And there was a match in the routing policy settings
msg="Match policy routing id=2136539137: to 10.1.7.2 via ifindex-110"
It turns out that the packet needs to be sent to one of the VPN tunnels:
"find a route: flag=04000000 gw-192.168.254.1 via DC-Ph1-1"
An allow rule is found in the firewall policies:
msg="Allowed by Policy-3:"
The encrypted packet is sent to the VPN tunnel:
func=ipsecdev_hard_start_xmit line=789 msg="enter IPsec interface-DC-Ph1-1"
func=_ipsecdev_hard_start_xmit line=666 msg="IPsec tunnel-DC-Ph1-1"
func=esp_output4 line=905 msg="IPsec encrypt/auth"
The encrypted packet is sent to the gateway address for this WAN interface:
msg="send to 2.2.2.2 via intf-WAN1"
For the second packet, everything happens in the same way, but it is sent to another VPN tunnel and leaves through a different firewall port:
func=ipsecdev_hard_start_xmit line=789 msg="enter IPsec interface-DC-Ph1-2"
func=_ipsecdev_hard_start_xmit line=666 msg="IPsec tunnel-DC-Ph1-2"
func=esp_output4 line=905 msg="IPsec encrypt/auth"
func=ipsec_output_finish line=622 msg="send to 4.4.4.2 via intf-WAN2"
Pros of the solution
Reliable functionality and user-friendly interface. The feature set that was available in FortiOS before the advent of SD-WAN has been fully preserved. That is, we do not have newly developed software, but a mature system from a proven firewall vendor. With a traditional set of network functions, a convenient and easy-to-learn web interface. How many SD-WAN vendors have, say, Remote-Access VPN functionality on end devices?
Level 80 security. FortiGate is one of the top firewall solutions. There are many materials on the Internet on configuring and administering firewalls, and there are many security specialists on the labor market who have already mastered the vendor's solutions.
Zero price for SD-WAN functionality. Building an SD-WAN network on FortiGate costs the same as building a regular WAN network on it, since no additional licenses are needed to implement the SD-WAN functionality.
Low entry threshold for the price. Fortigate has a good gradation of devices for different performance. The youngest and most inexpensive models are quite suitable for raising an office or point of sale, say, by 3-5 employees. Many vendors simply do not have such low-performance and affordable models.
High performance. Reducing the SD-WAN functionality to traffic balancing allowed the company to release a specialized SD-WAN ASIC, thanks to which the operation of SD-WAN does not reduce the performance of the firewall as a whole.
The ability to implement the entire office on Fortinet equipment. This is a pair of firewalls, switches, Wi-Fi access points. It is easy and convenient to manage such an office - switches and access points are registered on firewalls and controlled from them. This is how, for example, the switch port from the firewall interface that this switch is controlled by can look like:
Lack of controllers as a single point of failure. The vendor himself focuses on this, but this can only be called a plus, because for those vendors that have controllers, ensuring their fault tolerance is inexpensive, most often at the price of a small amount of computing resources in a virtualization environment.
What to look for
No separation of Control Plane and Data Plane. This means that the network must be configured either manually or using the traditional management tools already available - FortiManager. For vendors who have implemented such a separation, the network is assembled by itself. The administrator may only need to adjust its topology, prohibit something somewhere, nothing more. However, the trump card of FortiManager is that it can manage not only firewalls, but also Wi-Fi switches and access points, that is, almost the entire network.
Conditional increase in manageability. Because traditional tools are used to automate network configuration, there is little improvement in network manageability with the introduction of SD-WAN. On the other hand, new functionality becomes available faster, since the vendor first releases it only for the firewall operating system (which immediately makes it possible to use it), and only then supplements the management system with the necessary interfaces.
Part of the functionality may be available from the command line, but not available from the web interface. It's not so scary sometimes to go to the command line to configure something, it's scary not to see in the web interface that something has already been configured by someone from the command line. But this usually applies to the newest features and gradually, with FortiOS updates, the capabilities of the web interface are being tightened.
To suit
Those who do not have a lot of branches. Implementing an SD-WAN solution with complex central components on a network of 8-10 branches may not be worth the candle - you will have to spend money on licenses for SD-WAN devices and virtualization system resources to host the central components. Free computing resources in a small company, as a rule, are running out. In the case of Fortinet, it's easy enough to buy firewalls.
Those who have a lot of small branches. For many vendors, the minimum price of a solution for a branch is quite high and may turn out to be uninteresting from the point of view of the end customer's business. Fortinet offers small devices at a very attractive price.
For those who are not yet ready to step too far. Implementing SD-WAN with controllers, proprietary routing, and a new approach to network planning and management may be too big a step for some customers. Yes, such an implementation will ultimately help optimize the use of communication channels and the work of administrators, but first you have to learn a lot of new things. For those who are not yet ready for a paradigm shift, but want to squeeze more out of their communication channels, the solution from Fortinet is just right.
Source: habr.com