The Calico network plugin provides a wide range of network policies with a unified syntax to protect hardware hosts, virtual machines and pods. These policies can be applied within a namespace or be global network policies applicable to (to protect applications running directly on the host - the host can be directly a server or a virtual machine) or to (for protecting applications running in containers or hosted virtual machines). Calico policies allow you to apply security measures to different packet path points using options such as preDNAT, unraracked, and applyOnForward. Understanding how these options work can help improve overall system security and performance. This article explains the nature of these Calico policy settings (preDNAT, unraracked, and applyOnForward) applied to host endpoints, with a focus on what happens in the packet processing paths (iptabels chains).
This article assumes that you have a basic understanding of how Kubernetes and Calico network policies work. If not, we recommend trying ΠΈ using Calico before reading this article. We also expect you to have a basic understanding of how in linux.
Calico allows you to apply a set of access rules by labels (to groups of hosts and workloads/pods). This is very useful if you use heterogeneous systems together - virtual machines, a system directly on the hardware, or a kubernetes infrastructure. In addition, you can protect your cluster (nodes) with a set of declarative policies and apply network policies to incoming traffic (for example, through the NodePorts or External IPs service).
At a fundamental level, when Calico connects a pod to the network (see diagram below), it connects it to the host using a virtual Ethernet interface (veth). The traffic sent by the pod comes to the host from this virtual interface and is processed in the same way as if it came from the physical network interface. By default, Calico names these interfaces caliXXX. Since the traffic is coming through the virtual interface, it goes through iptables as if the pod was one hop away. So when traffic comes in/out of a pod, it is forwarded from the host's point of view.
On a Kubernetes node running Calico, you can map a virtual interface (veth) to a workload like this. In the example below, you can see that veth#10 (calic1cbf1ca0f8) is connected to cnx-manager-* in the calico-monitoring namespace.
[centos@ip-172-31-31-46 K8S]$ sudo ip a
...
10: calic1cbf1ca0f8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
...
[centos@ip-172-31-31-46 K8S]$ calicoctl get wep --all-namespaces
...
calico-monitoring cnx-manager-8f778bd66-lz45m ip-172-31-31-46.ec2.internal 192.168.103.134/32
calic1cbf1ca0f8
...
Given that Calico creates a veth interface for each workload, how does it enforce policies? To do this, Calico creates hooks in various packet processing chains using iptables.
The diagram below shows the chains involved in packet processing in iptables (or the netfilter subsystem). When a packet arrives on a network interface, it first goes through the PREROUTING chain. A routing decision is then made, and based on that, the packet passes either through INPUT (directed to host processes) or through FORWARD (directed to a pod or another node on the network). From the local process, the packet passes through the OUTPUT and then POSTROUTING chain before being sent over the wire.
Note that the pod is also an external entity (attached to veth) in terms of iptables processing. Let's summarize:
- Forwarded traffic (nat, routed, or to/from a pod) goes through the PREROUTING - FORWARD - POSTROUTING chains.
- Traffic to the local host process goes through the PREROUTING - INPUT chain.
- Traffic from the local host process goes through the OUTPUT - POSTROUTING chain.
Calico provides options for policies with which you can apply policies to all chains. With that in mind, let's take a look at the different policy options available in Calico. The numbers in the options list below correspond to the numbers in the diagram above.
- Workload endpoint (pod) policy
- host endpoint policy
- ApplyOnForward option
- PreDNAT Policy
- Untracked Policy
Let's start by looking at how policies apply to workload endpoints (Kubernetes pods or OpenStack VMs) and then look at the policy options for host endpoints.
Workload Endpoints
Workload Endpoint Policy (1)
This is an option to secure your kubernetes pods. Calico supports working with Kubernetes NetworkPolicy, but it also provides additional policies - Calico NetworkPolicy and GlobalNetworkPolicy. Calico creates a chain for each pod (workload) and hooks in the INPUT and OUTPUT chains for the workload to the filter table of the FORWARD chain.
Host Endpoints
Host Endpoint Policy (2)
In addition to CNI (container network interface), Calico policies provide the ability to protect the host itself. In Calico, you can create a host endpoint by specifying a combination of the host's interface and, optionally, port numbers. Policing for this entity is achieved using a filter table in the INPUT and OUTPUT chains. As you can see from the diagram, (2) they apply to local processes on the node/host. That is, if you have created a policy that applies to the host endpoint, it will not affect traffic to/from your pods. But it does provide a single interface/syntax for blocking traffic to your host and pods using Calico policies. This greatly simplifies the policy management process for a heterogeneous network. Setting up host endpoint policies to harden the cluster is another important use case.
ApplyOnForward Policy (3)
The ApplyOnForward option is available in the Calico global network policy to enable the policy to be applied to all traffic passing through the host endpoint, including traffic that will be forwarded by the host. This traffic includes forwarding to the local pod or anywhere else on the network. Calico requires this setting to be enabled for policies using PreDNAT and untracked, see the following sections. In addition, ApplyOnForward can be used to track host traffic in cases where a virtual router or software NAT is used.
Note that if you need to apply the same network policy to both host processes and pods, then you don't have to use the ApplyOnForward option. You just need to create a label for the desired hostendpoint and workload endpoint (pod). Calico is smart enough to enforce policy based on labels regardless of the type of endpoint (hostendpoint or workload).
PreDNAT Policy (4)
In Kubernetes, the ports of the service entity can be exposed to the outside using the NodePorts option or, optionally (when using Calico), by declaring them through the Cluster IPs or External IPs options. Kube-proxy balances incoming service-bound traffic to corresponding service pods using DNAT. With that in mind, how do you enforce policies for traffic coming through NodePorts? In order for these policies to be applied before traffic is processed by DNAT (which is a mapping of a host:port and the corresponding service), Calico provides a setting for the globalNetworkPolicy called "preDNAT: true".
When pre-DNAT is enabled, these policies are implemented at (4) in the diagram - in the mangle table of the PREROUTING chain - just before DNAT. The usual order of policies (order) is not observed here, since the application of these policies occurs much earlier along the traffic processing path. However, preDNAT policies respect the application order (order) among themselves.
When creating policies with pre-DNAT, it's important to be mindful of the traffic you want to process and let the majority be rejected. Traffic marked as 'allow' in the pre-DNAT policy will no longer be checked by the hostendpoint policy, while traffic that fails the pre-DNAT policy will continue on through the rest of the chains.
Calico has made it mandatory to include the applyOnForward option when using preDNAT because, by definition, the traffic destination has not yet been selected. Traffic can be directed to the host process, or it can be redirected to a pod or another node.
Untracked Policy (5)
Networks and applications can have large differences in behavior. In some extreme cases, applications may generate many short lived connections. This can cause conntrack (a core component of the Linux networking stack) to run out of memory. Traditionally, to run these types of applications on Linux, you have to manually configure or disable conntrack, or write iptables rules to bypass conntrack. Untracked policy in Calico is a simpler and more efficient option if you want to process connections as quickly as possible. For example if you use massive or as an additional measure of protection against .
Read this (or ) for more information, including performance tests using untracked policy.
When you set the "doNotTrack: true" option on the Calico globalNetworkPolicy, it becomes an **untracked** policy and is applied very early in the Linux packet processing pipeline. Looking at the diagram above, untracked policies are applied in the PREROUTING and OUTPUT chains in the raw table before connection tracking (conntrack) is started. When a packet is allowed untracked by the policy, it is marked to disable connection tracking for that packet. It means:
- The untracked policy is applied per package. There is no notion of connection (or flow). The lack of connections has several important consequences:
- If you want to allow both request and response traffic, you need a rule for both inbound and outbound traffic (since Calico usually uses conntrack to mark response traffic as allowed).
- The untracked policy does not work for Kubernetes workloads (pods), because in this case there is no way to track the outgoing connection from the pod.
- NAT does not work correctly with untracked packets (because the kernel stores the NAT mapping in conntrack).
- When passing through the "allow all" rule in the untracked policy, all packets will be marked as untracked. This is almost always not what you want, so it's important to be very selective about which packets are allowed by untracked policies (and allow most traffic to go through normal tracked policies).
- Untracked policies are applied at the very beginning of the packet processing pipeline. This is very important to understand when creating Calico policies. You can have a pod policy with order:1 and an untracked policy with order:1000. It won't matter. The untracked policy will be applied before the policy for the pod. Untracked policies respect the order of execution only among themselves.
Because one of the goals of the doNotTrack policy is to enforce the policy very early in the Linux packet processing pipeline, Calico makes it mandatory to specify the applyOnForward option when using doNotTrack. Referring to the packet processing diagram, note that the untracked(5) policy is applied before any routing decisions. Traffic can be directed to the host process, or it can be redirected to a pod or another node.
Results
We looked at the different policy options (Host endpoint, ApplyOnForward, preDNAT, and Untracked) in Calico and how they are applied in the packet processing path. Understanding how they work helps you develop effective and secure policies. With Calico, you can use a global network policy that applies to a label (a group of nodes and pods) and apply policies with different options. This allows security and network design professionals to conveniently secure βeverythingβ (endpoint types) at once using a single policy language with Calico policies.
Acknowledgment: I would like to thank ΠΈ for their review and for valuable information.
Source: habr.com