When launching a Kubernetes cluster for a specific application, it is important to understand the requirements imposed on this resource by the application itself, the business, and the developers. With this information in hand, one can proceed to make architectural decisions and, in particular, choose a specific Ingress controller, of which there are already many today. To provide a basic understanding of the available options without having to sift through numerous articles/documentation, we have prepared this overview, including the main (production-ready) Ingress controllers.
We hope this will assist colleagues in making architectural decisions — at the very least, it will serve as a starting point for obtaining more detailed information and practical experiments. Preliminary research of similar materials online revealed, surprisingly, that there wasn't a single reasonably complete, and most importantly — structured — overview. Thus, let's fill this gap!
Criteria
To effectively compare and obtain any useful results, it’s necessary to understand not just the subject area but also to have a specific list of criteria that will guide the research direction. Without claiming to analyze all possible use cases of Ingress/Kubernetes, we have tried to highlight the most common requirements for controllers — be prepared that the specifics will still need to be researched separately.
But I'll start with the characteristics that have become so commonplace that they are implemented in all solutions and are not debated:
- dynamic service discovery;
- SSL termination;
- support for websockets.
Now, about the comparison points:
Supported Protocols
One of the fundamental criteria for selection. Your software may not operate over standard HTTP or may require operation over multiple protocols simultaneously. If your case is non-standard, be sure to factor this in to avoid having to reconfigure the cluster later. The list of supported protocols varies among all controllers.
Software at its core
There are several application options on which the controller is based. Popular ones include nginx, traefik, haproxy, and envoy. Generally, it may not significantly affect how traffic is received and transmitted, but it is always useful to know the potential nuances and characteristics of what is "under the hood."
Traffic Routing
What factors can be used to decide the direction of traffic to a particular service? Generally, this is based on host and path, but there can also be additional options.
Namespace within a Cluster
A namespace is a way to logically divide resources in Kubernetes (for example, into stage, production, etc.). There are Ingress controllers that need to be installed separately in each namespace (and then it can route traffic only to the pods of that namespace). However, there are those (and the vast majority) that work globally across the entire cluster — in them, traffic is directed to any pod in the cluster, regardless of the namespace.
Health Checks for Upstreams
How is traffic directed to healthy instances of applications and services? There are options with active and passive checks, retries, circuit breakers (for more details, see, for example, the ), custom health checks, etc. This is a crucial parameter if you have high availability requirements and timely removal of failed services from load balancing.
Load Balancing Algorithms
There are many options here: from traditional to exotic ones like , as well as separate options like .
Authentication
What authorization schemes does the controller support? Basic, digest, oauth, external-auth — I believe these options should be familiar. This is an important criterion if multiple environments for developers (and/or simply closed ones) are used, with access provided through Ingress.
Traffic Distribution
Does the controller support commonly used mechanisms for traffic distribution, such as canary releases, A/B testing, and traffic mirroring/shadowing? This is a genuinely sensitive topic for applications that require careful and precise traffic management for production testing, debugging product errors without live traffic (or with minimal losses), traffic analysis, etc.
Paid subscription
Is there a paid option for the controller with enhanced functionalities and/or technical support?
Graphical interface (Web UI)
Is there any graphical interface for managing the controller's configuration? Primarily for convenience and/or for those who need to make changes to the Ingress configuration, as working with 'raw' templates can be cumbersome. It may be useful if developers want to experiment with traffic on the fly.
JWT validation
The presence of built-in validation for JSON web tokens for authorization and user validation in the end application.
Customization options for the config
Extendability of templates in terms of having mechanisms that allow the addition of custom directives, flags, etc. to standard configuration templates.
Basic DDoS protection mechanisms
Simple rate limit algorithms or more complex options for filtering traffic based on addresses, whitelists, countries, etc.
Request tracing
Monitoring, tracking, and debugging capabilities for requests from Ingresses to specific services/pods, ideally also between services/pods.
WAF
Support .
Ingress controllers
The list of controllers was formed based on and . Some of them were excluded from the review due to their specificity or low prevalence (early development stage). The remaining ones are discussed below. We will start with a general description of the solutions and continue with a summary table.
Kubernetes Ingress
Website:
License: Apache 2.0
This is the official controller for Kubernetes, developed by the community. As the name suggests, it is based on nginx and is enhanced with a variety of Lua plugins used to implement additional features. Due to the popularity of nginx itself and minimal modifications made when used as a controller, this option may be the simplest and most understandable for the average engineer (with web experience).
NGINX Inc Ingress
Website:
License: Apache 2.0
An official product from the nginx developers. It has a paid version based on The main idea is a high level of stability, constant backward compatibility, absence of any external modules, and a declared increased speed (compared to the official controller), achieved by eliminating Lua.
The free version is significantly limited, especially when compared to the official controller (due to the lack of those same Lua modules). The paid version, however, has a fairly wide range of additional features: real-time metrics, JWT validation, active health checks, and more. An important advantage over NGINX Ingress is full support for TCP/UDP traffic (even in the community version!). The downside is that features for traffic distribution, which, however, "has the highest priority for developers", but requires time for implementation.
Kong Ingress
Website:
License: Apache 2.0
A product developed by Kong Inc. in two versions: commercial and free. Based on nginx, whose capabilities are extended by a large number of Lua modules.
Initially aimed at processing and routing API requests, i.e., as an API Gateway, it has now become a full-fledged Ingress controller. Its main advantages include a multitude of additional modules (including those from third-party developers) that are easy to install and configure, and through which a wide range of additional capabilities can be implemented. However, the built-in functions already offer many features. Configuration is performed using CRD resources.
An important feature of the product is that operating within a single context (instead of cross-namespaced) is a contentious topic: some may see it as a drawback (as it requires creating entities for each context), while others may view it as a feature (providing ahigher level of isolation, as if one controller is broken, the problem is confined to that specific context).Traefik
github.com/containous/traefik
Website:
License: MIT
A proxy originally designed for handling request routing for microservices and their dynamic environments. This brings many useful features: configuration updates without restarts, support for numerous load balancing methods, a web interface, metric forwarding, support for various protocols, a REST API, canary releases, and much more. An additional pleasant feature is the support for Let's Encrypt certificates out of the box. A downside is that setting up and connecting a custom KV storage is required for high availability (HA) with the controller.
HAProxy
Website:
License: Apache 2.0
HAProxy has long been known as a proxy and traffic balancer. Within a Kubernetes cluster, it offers 'soft' configuration updates (without traffic loss), DNS-based service discovery, and dynamic configuration through the API. The full customization of config templates through the replacement of ConfigMaps can be attractive, as well as the ability to use functions from the Sprig library. Overall, the main focus of the solution is on high speed, optimization, and efficiency in resource consumption. The advantage of the controller is its support for a record number of different load balancing methods.
Voyager
Website:
License: Apache 2.0
A controller based on HAProxy that is positioned as a universal solution supporting wide capabilities across many providers. It offers the ability to balance traffic at L7 and L4, with L4 TCP traffic balancing being one of the key features of the solution.
Contour
Website:
License: Apache 2.0
This solution is not only based on Envoy: it is developed in collaboration with the creators of this popular proxy. An important feature is the ability to separate Ingress resource management through CRD resources IngressRoute. For organizations with multiple development teams using a single cluster, this helps to maximize traffic handling safety in adjacent contours and protects against errors when modifying Ingress resources.
A comprehensive set of load balancing methods is also offered (including request mirroring, automatic retries, rate limiting, and much more), along with detailed monitoring of traffic flow and failures. The absence of sticky session support may be a significant drawback for some (although work is ongoing) ).
Istio Ingress
Website:
License: Apache 2.0
A comprehensive service mesh solution that not only acts as an Ingress controller managing incoming external traffic, but also controls all traffic within the cluster. Under the hood, an Envoy sidecar proxy is used for each service. Essentially, it's a powerful tool that "can do it all," with its main idea being maximum manageability, scalability, security, and transparency. With it, you can finely tune traffic routing, service access authorization, load balancing, monitoring, canary releases, and much more. More about Istio can be found in a series of articles titled "».
Ambassador
Website:
License: Apache 2.0
Another Envoy-based solution. It has both free and commercial versions. It is marketed as "fully native to Kubernetes," bringing the corresponding benefits (tight integration with the methods and entities of the K8s cluster).
Comparison Table
So, the culmination of this article is this huge table:
It is clickable for detailed viewing and is also available in .
In Summary
The goal of this article is to provide a more complete understanding (though certainly not exhaustive!) of what choice to make in your specific case. As is often the case, each controller has its advantages and disadvantages...
The classic Ingress from Kubernetes is known for its availability and reliability, offering quite rich options — in most cases, it should be more than sufficient. However, if there are higher demands for stability, feature levels, and development, consider Ingress with NGINX Plus and a paid subscription. Kong has a rich set of plugins (and, accordingly, the capabilities they provide), with even more in the paid version. It has extensive capabilities for functioning as an API Gateway, dynamic configuration based on CRD resources, and basic Kubernetes services.
If you have increased requirements for load balancing and authorization methods, take a look at Traefik and HAProxy. These are Open Source projects, proven over the years, very stable, and actively evolving. Contour has been around for a couple of years now, but still appears too young and has only basic capabilities added on top of Envoy. If there's a need for a WAF in front of the application, consider using Ingress from Kubernetes or HAProxy.
The most feature-rich products are those built on Envoy, particularly Istio. It presents a comprehensive solution that can do 'everything', which, however, translates to a significantly higher entry threshold for configuration, deployment, and administration compared to other solutions.
We have chosen the Ingress from Kubernetes as our standard controller, which still meets 80–90% of our needs. It is quite reliable, easy to configure, and extend. In general, in the absence of specific requirements, it should suit most clusters/applications. Other universal and relatively simple products that can be recommended include Traefik and HAProxy.
P.S.
Also read in our blog:
- "Back to Microservices with Istio": , , ;
- «»;
- «».
Source: habr.com
