When launching a Kubernetes cluster for a specific application, you need to understand what the application itself, the business, and the developers pose to this resource. With this information, you can start making an architectural decision and, in particular, choosing a specific Ingress controller, of which there are already a large number today. In order to get a basic idea of the available options without having to go through a lot of articles / documentation, etc., we have prepared this overview, including the main (production ready) Ingress controllers.
We hope that it will help colleagues in choosing an architectural solution - at least it will become a starting point for obtaining more detailed information and practical experiments. Previously, we studied other similar materials on the net and, oddly enough, did not find a single more or less complete, and most importantly - structured - review. So let's fill that gap!
Criteria
In principle, in order to make a comparison and get any useful result, you need to understand not just the subject area, but also have a specific list of criteria that will set the research vector. Without pretending to analyze all possible cases of using Ingress / Kubernetes, we tried to highlight the most general requirements for controllers - be prepared that in any case you will have to study all your specifics and particulars separately.
But I'll start with the characteristics that have become so familiar that they are implemented in all solutions and are not considered:
- dynamic discovery of services (service discovery);
- SSL termination;
- working with websockets.
Now for the points of comparison:
Supported protocols
One of the fundamental selection criteria. Your software may not work on standard HTTP, or it may require work on multiple protocols at once. If your case is non-standard, be sure to take this factor into account so that you do not have to reconfigure the cluster later. For all controllers, the list of supported protocols varies.
software at the core
There are several variations of applications that the controller is based on. Popular ones are nginx, traefik, haproxy, envoy. In the general case, it may not have much effect on how traffic is received and transmitted, but it is always useful to know the potential nuances and features of what is “under the hood”.
Traffic routing
On the basis of what it is possible to make a decision about the direction of traffic to a particular service? Usually these are host and path, but there are additional possibilities.
Namespace within a cluster
Namespace (namespace) - the ability to logically split resources in Kubernetes (for example, on stage, production, etc.). There are Ingress controllers that must be installed separately in each namespace (and then it can direct traffic only to the pods of this space). And there are those (and their clear majority) that work globally for the entire cluster - in them traffic is directed to any pod of the cluster, regardless of the namespace.
Samples for upstreams
How is traffic directed to healthy instances of the application, services? There are options with active and passive checks, retries, circuit breakers (For more details, see, for example, ), custom health checks, etc. A very important parameter if you have high requirements for availability and timely removal of failed services from balancing.
Balancing algorithms
There are many options: from traditional to the exotic , as well as individual features like .
Authentication
What authorization schemes does the controller support? Basic, digest, oauth, external-auth - I think these options should be familiar. This is an important criterion if there are many developer (and/or just private) loops that are accessed through Ingress.
Traffic distribution
Does the controller support such commonly used mechanisms for traffic distribution as canary rollouts (canary), A / B testing, traffic mirroring (mirroring / shadowing)? This is a really sore subject for applications that require accurate and precise traffic management for productive testing, debugging product bugs off-line (or with minimal loss), traffic analysis, and so on.
Paid subscription
Is there a paid option for the controller, with advanced functionality and / or technical support?
Graphical user interface (Web UI)
Is there any GUI to manage controller configuration? Mainly for "handiness" and / or for those who need to make some changes to the Ingress'a configuration, but working with "raw" templates is inconvenient. It can be useful if developers want to conduct some experiments with traffic on the fly.
JWT validation
The presence of built-in validation of JSON web tokens for authorization and validation of the user to the end application.
Possibilities for config customization
Template extensibility in the sense of having mechanisms that allow you to add your own directives, flags, etc. to standard configuration templates.
Basic DDOS protection mechanisms
Simple rate limit algorithms or more complex traffic filtering options based on addresses, whitelists, countries, etc.
Request trace
The ability to monitor, track and debug requests from Ingresses to specific services / pods, and ideally between services / pods too.
WAF
Support .
Controllers
The list of controllers was formed based on и . We excluded some of them from the review due to specificity or low prevalence (early stage of development). The rest are discussed below. Let's start with a general description of the solutions and continue with a summary table.
Ingress from Kubernetes
Website:
License: Apache 2.0
This is the official controller for Kubernetes and is being developed by the community. Obviously from the name, it is based on nginx and is complemented by a different set of Lua plugins used to implement additional features. Due to the popularity of nginx itself and minimal modifications to it when used as a controller, this option may be the easiest and easiest to configure for the average engineer (with web experience).
Ingress by NGINX Inc.
Website:
License: Apache 2.0
The official product of the nginx developers. Has a paid version based on . The main idea is a high level of stability, constant backward compatibility, the absence of any extraneous modules and the declared increased speed (compared to the official controller), achieved due to the rejection of Lua.
The free version is significantly reduced, including even when compared with the official controller (due to the lack of the same Lua modules). At the same time, the paid one has a fairly wide additional functionality: real-time metrics, JWT validation, active health checks, and more. An important advantage over NGINX Ingress is full support for TCP / UDP traffic (and in the community version too!). Minus - traffic distribution feature, which, however, “has the highest priority for developers,” but takes time to implement.
Kong Ingress
Website:
License: Apache 2.0
Product developed by Kong Inc. in two versions: commercial and free. Based on nginx, which has been extended with a large number of Lua modules.
Initially, it was focused on processing and routing API requests, i.e. as an API Gateway, but at the moment it has become a full-fledged Ingress controller. Main advantages: many additional modules (including those from third-party developers) that are easy to install and configure and with the help of which a wide range of additional features is implemented. However, built-in functions already offer many possibilities. Job configuration is done using CRD resources.
An important feature of the product - working within the same contour (instead of cross-namespaced) is a controversial topic: for some it will seem like a disadvantage (you have to produce entities for each contour), and for someone it’s a feature (bоGreater level of isolation, as if one controller is broken, then the problem is limited to the circuit alone).
traefik
Website:
License: MIT
A proxy that was originally created to work with request routing for microservices and their dynamic environment. Hence, many useful features: updating the configuration without rebooting at all, support for a large number of balancing methods, web interface, metrics forwarding, support for various protocols, REST API, canary releases, and much more. Another nice feature is support for Let's Encrypt certificates out of the box. The disadvantage is that in order to organize high availability (HA), the controller will need to install and connect its own KV storage.
HAProxy
Website:
License: Apache 2.0
HAProxy has long been known as a proxy and traffic balancer. As part of a Kubernetes cluster, it offers a “soft” configuration update (without loss of traffic), service discovery based on DNS, dynamic configuration using API. It can be attractive to completely customize the config template by replacing the CM, as well as the ability to use Sprig library functions in it. In general, the main emphasis of the solution is on high speed, its optimization and efficiency in consumed resources. The advantage of the controller is the support of a record number of different balancing methods.
Travel
Website:
License: Apache 2.0
Based on HAproxy controller, which is positioned as a universal solution that supports a wide range of features on a large number of providers. An opportunity is offered for balancing traffic on L7 and L4, and balancing TCP L4 traffic as a whole can be called one of the key features of the solution.
Contour
Website:
License: Apache 2.0
This solution is not only based on Envoy: it was developed by jointly with the authors of this popular proxy. An important feature is the ability to separate control of Ingress resources using IngressRoute CRD resources. For organizations with many development teams using the same cluster, this helps to maximize the security of working with traffic in neighboring loops and protect them from errors when changing Ingress resources.
It also offers an extended set of balancing methods (there is request mirroring, auto-repeat, request rate limiting, and much more), detailed monitoring of traffic flow and failures. Perhaps for someone it will be a significant drawback the lack of support for sticky sessions (although the work ).
Istio Ingress
Website:
License: Apache 2.0
A comprehensive service mesh solution that is not only an Ingress controller that manages incoming traffic from the outside, but also controls all traffic within the cluster. Under the hood, Envoy is used as a sidecar proxy for each service. In essence, this is a big combine that “can do anything”, and its main idea is maximum manageability, extensibility, security and transparency. With it, you can fine-tune traffic routing, access authorization between services, balancing, monitoring, canary releases, and much more. Read more about Istio in the series of articles "».
Ambassador
Website:
License: Apache 2.0
Another solution based on Envoy. It has free and commercial versions. It is positioned as "fully native to Kubernetes", which brings the corresponding advantages (tight integration with the methods and entities of the K8s cluster).
Comparison table
So, the culmination of the article is this huge table:
It is clickable for a closer view, and is also available in the format .
Conclusion
The purpose of this article is to provide a more complete understanding (however, by no means exhaustive!) of what choice to make in your particular case. As usual, each controller has its own advantages and disadvantages…
The classic Ingress from Kubernetes is good for its availability and provenness, rich enough features - in the general case, it should be “enough for the eyes”. However, if there are increased requirements for stability, the level of features and development, you should pay attention to Ingress with NGINX Plus and a paid subscription. Kong has the richest set of plug-ins (and, accordingly, the opportunities they provide), and in the paid version there are even more of them. It has ample opportunities to work as an API Gateway, dynamic configuration based on CRD resources, as well as basic Kubernetes services.
With increased requirements for balancing and authorization methods, take a look at Traefik and HAProxy. These are Open Source projects, proven over the years, very stable and actively developing. Contour has been out for a couple of years now, but it still looks too young and has only basic features added on top of Envoy. If there are requirements for the presence / embedding of WAF in front of the application, you should pay attention to the same Ingress from Kubernetes or HAProxy.
And the richest in terms of features are products built on top of Envoy, especially Istio. It seems to be a comprehensive solution that “can do anything”, which, however, also means a significantly higher entry threshold for configuration / launch / administration than other solutions.
We have chosen and still use Ingress from Kubernetes as a standard controller, which covers 80-90% of needs. It is quite reliable, easy to configure and expand. In general, in the absence of specific requirements, it should suit most clusters / applications. Of the same universal and relatively simple products, Traefik and HAProxy can be recommended.
PS
Read also on our blog:
- "Back to microservices with Istio": , , ;
- «";
- «».
Source: habr.com