As IPv4 addresses become exhausted, many service providers have faced the need to provide their clients with network access through address translation. In this article, I will explain how to achieve Carrier Grade NAT performance on commodity servers.
A Bit of History
The issue of IPv4 address space exhaustion is not new. At one point, waiting lists appeared within RIPE, followed by exchanges where address blocks were traded and leasing deals were made. Gradually, service providers began offering Internet access services using address and port translation. Some operators were unable to acquire enough addresses to assign a 'white' address to each subscriber, while others started saving money by opting out of purchasing addresses on the secondary market. Equipment manufacturers supported this idea since this functionality usually requires additional extension modules or licenses. For instance, Juniper's MX router series (except for the latest MX104 and MX204) performs NAPT on a separate MS-MIC service card, Cisco ASR1k requires a CGN license, and Cisco ASR9k necessitates a separate A9K-ISM-100 module along with the A9K-CGN-LIC license. Overall, this service comes at a significant cost.
IPTables
The task of performing NAT does not require specialized computing resources; it can be handled by general-purpose processors found in any home router. At the scale of a service provider, this task can be solved using commodity servers running FreeBSD (ipfw/pf) or GNU/Linux (iptables). We will not consider FreeBSD as I have long abandoned this OS, so let's focus on GNU/Linux.
Enabling address translation is straightforward. First, you need to add a rule in iptables to the nat table:
iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -j SNAT --to - --persistent
The operating system will load the nf_conntrack module, which will monitor all active connections and perform necessary transformations. There are a few nuances. Firstly, since this concerns NAT on a carrier scale, it's necessary to adjust the timeouts because the default values will quickly cause the translation table size to grow to catastrophic levels. Below is an example of the settings I used on my servers:
net.ipv4.ip_forward = 1
net.ipv4.ip_local_port_range = 8192 65535
net.netfilter.nf_conntrack_generic_timeout = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 45
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 60
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15
net.netfilter.nf_conntrack_checksum=0
Secondly, since the default size of the translation table is not designed for carrier-grade operation, it needs to be increased:
net.netfilter.nf_conntrack_max = 3145728
It is also necessary to increase the number of buckets for the hash table that stores all translations (this is an option of the nf_conntrack module):
options nf_conntrack hashsize=1572864
After these straightforward adjustments, a fully operational construct is obtained, capable of translating a large number of client addresses to an external pool. However, the performance of this solution leaves much to be desired. In my early attempts to use GNU/Linux for NAT (around 2013), I managed to achieve a performance of about 7 Gbit/s at 0.8 Mpps on one server (Xeon E5-1650v2). Since then, many different optimizations have been made in the GNU/Linux kernel's network stack, and the performance of a single server on the same hardware has increased to nearly 18-19 Gbit/s at 1.8-1.9 Mpps (these were peak values), but the demand for the volume of traffic handled by a single server has grown much faster. As a result, load balancing schemes across different servers were developed, but all this increased the complexity of configuration, maintenance, and quality assurance of the services provided.
NFTables
Currently, the trendy approach in programmatically "packet forwarding" is the use of DPDK and XDP. A lot of articles have been written on this topic, numerous presentations have been made, and commercial products are emerging (for example, SKAT by VasExperts). However, given the limited resources of programmers at telecommunications operators, developing any kind of "custom solution" based on these frameworks is quite challenging. Furthermore, maintaining such a solution later on will be significantly more difficult; for instance, tools for diagnostics will need to be developed. For example, the regular tcpdump won’t work with DPDK as it is, and it won’t "see" packets sent back into the wires with XDP. Amidst all the discussions about new technologies for forwarding packets in user-space, unnoticed have remained and Pablo Neira Ayuso, the maintainer of iptables, on the development of flow offloading in nftables. Let's take a closer look at this mechanism.
The main idea is that if a router has allowed packets from one session to pass in both directions of the stream (the TCP session has moved to the ESTABLISHED state), it is unnecessary to pass subsequent packets of this session through all firewall rules, as all these checks will ultimately end in routing the packet. Moreover, there is no need to choose a route — we already know which interface and to which host the packets within this session should be forwarded. We only need to save this information and use it for routing at the early stage of packet processing. When performing NAT, we also need to maintain information about the address and port changes made by the nf_conntrack module. Yes, of course, in this case, various policers and other informational-statistical rules in iptables cease to work, but within the scope of the task for a standalone NAT or, for example, a border router — this is not very important, because the services are distributed across devices.
Configuration
To utilize this feature, we need to:
- Use a recent kernel. Although the functionality itself appeared in kernel 4.16, it was quite "raw" for a long time and regularly caused kernel panic. Everything stabilized around December 2019, when the LTS kernels 4.19.90 and 5.4.5 were released.
- Rewrite iptables rules in the nftables format, using a sufficiently recent version of nftables. It works accurately in version 0.9.0.
If the first point is generally clear, just don't forget to include the module in the configuration during the build (CONFIG_NFT_FLOW_OFFLOAD=m), the second point requires some explanation. nftables rules are described in a completely different way than in iptables. covers almost all aspects, and there are also special for rules from iptables to nftables. Therefore, I will only provide an example of configuring NAT and flow offload. A small legend for the example: , — these are the network interfaces through which traffic flows; there may actually be more than two. , — the starting and ending addresses of the range of 'white' addresses.
NAT configuration is very simple:
#! /usr/sbin/nft -f
table nat {
chain postrouting {
type nat hook postrouting priority 100;
oif <o_if> snat to <pool_addr_start>-<pool_addr_end> persistent
}
}
Flow offload is a bit more complex, but quite understandable:
#! /usr/sbin/nft -f
table inet filter {
flowtable fastnat {
hook ingress priority 0
devices = { <i_if>, <o_if> }
}
chain forward {
type filter hook forward priority 0; policy accept;
ip protocol { tcp , udp } flow offload @fastnat;
}
}
And that's the entire configuration. Now all TCP/UDP traffic will go to the fastnat table and be processed much faster.
Results
To show how much 'faster' this is, I will attach a screenshot of the load on two real servers, with identical configurations (Xeon E5-1650v2), configured the same way, using the same Linux kernel, but performing NAT in iptables (NAT4) and in nftables (NAT5).
The screenshot does not show a graph of packets per second, but in the load profile of these servers, the average packet size is around 800 bytes, so the values reach up to 1.5Mpps. As seen, the performance reserve of the server with nftables is enormous. Currently, this server processes up to 30Gbit/s at 3Mpps and is clearly capable of hitting the physical network limit of 40Gbps, while having free CPU resources.
I hope this material will be useful for network engineers trying to improve the performance of their servers.
Source: habr.com
