Girl on a scooter. Illustration , the Nomad logo from
Kubernetes is a 300kg container orchestration gorilla. It works in some of the largest container systems in the world, but it comes at a cost.
Especially expensive for small teams that have to spend a lot of time on support and a steep learning curve. For our team of four, this is too much overhead. Therefore, we began to look for alternatives - and fell in love with .
What do you want
Our team maintains a number of typical performance monitoring and analysis services: API endpoints for metrics written in Go, Prometheus exports, log parsers such as Logstash and , as well as databases such as InfluxDB or Elasticsearch. Each of these services runs in its own container. We need a simple system to keep it all running.
We started with a list of requirements for container orchestration:
- Running a set of services on many machines.
- Overview of running services.
- Relationships between services.
- Automatic restart if service crashes.
- Maintenance of infrastructure by a small team.
In addition, the following things would be nice, but not required additions:
- Marking machines by their capabilities (for example, marking machines with fast disks for heavy I/O services).
- The ability to run services independent of the orchestrator (for example, during development).
- A common place to share configurations and secrets.
- End point for metrics and logs.
Why Kubernetes is not for us
While prototyping with Kubernetes, we noticed that we began to add more and more complex layers of logic, which we implicitly relied on.
As an example, Kubernetes supports built-in service configurations via . You can get confused quickly, especially when merging multiple configuration files or adding additional services to a pod. Kubernetes (or in this case) allows you to dynamically inject external configurations for separation of concerns. But this leads to a hard and dark connection between your project and Kubernetes. However, Helm and ConfigMaps are optional, so you don't have to use them. You can just copy the configuration to the Docker image. However, it's tempting to go that route and build unnecessary abstractions that you might regret later.
In addition, the Kubernetes ecosystem is rapidly evolving. It takes a lot of time and energy to stay up to date with best practices and the latest tools. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list goes on and on. Not all of these tools are needed to get started, but you don't know what you'll need, so you need to be aware of everything. Because of this, the learning curve is quite steep.
When to Use Kubernetes
Many people in our company use Kubernetes and are quite happy with it. These instances are managed by Google or Amazon, who have sufficient support resources.
Kubernetes comes with , which make large-scale container orchestration more manageable:
- detailed .
- add logic to the cluster. They are just programs that talk to the Kubernetes API.
- ! Kubernetes is able to scale services on demand using service metrics without requiring manual intervention.
The question is, do you really need all these features. You can't just rely on abstractions; .
Our team provides most of the services remotely (because of the close connection to the core infrastructure), so we didn't want to raise our own Kubernetes cluster. We just wanted to provide services.
Batteries not included
Nomad is 20% orchestration that gives 80% of what is needed. All it does is manage deployments. Nomad takes care of deployments, restarts containers in case of errors... and that's it.
The whole point of Nomad is what it does. minimum: no fine-grained rights management or , so specially conceived. These components are provided externally or not provided at all.
I think Nomad has found the perfect compromise between ease of use and usefulness. It is good for small, independent services. If you need more control, you will have to raise them yourself or use a different approach. nomad is just orchestrator.
The best thing about Nomad is that it's easy substitute. There is practically no vendor lock-in, since its functions are easily integrated into any other system that manages services. It works just like a regular binary on every machine in the cluster, that's all!
Loosely coupled Nomad ecosystem
The real strength of Nomad is in its ecosystem. It integrates very well with other - completely optional - products such as (key-value store) or (handling secrets). Inside the Nomad file, there are sections for extracting data from these services:
template {
data = <<EOH
LOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"
API_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}"
EOH
destination = "secrets/file.env"
env = true
}
Here we read the key service/geo-api/log-verbosity from Consul and in the course of work we represent it with an environment variable LOG_LEVEL. We also present the key secret/geo-api-key from Vault as API_KEY. Simple but powerful!
Due to its simplicity, Nomad is easily extensible with other services via an API. For example, job tags are supported. We tag all services with metrics with the tag trv-metrics. This way Prometheus easily finds these services through Consul and checks the endpoint periodically /metrics for new data. The same can be done, for example, for logs, using .
There are many other examples of extensibility:
- Run a Jenkins job with a hook, and Consul keeps track of the Nomad job being redeployed when the service configuration changes.
- Ceph adds a distributed file system to Nomad.
- for load balancing.
All this allows without special reference to the vendor.
Fair Warning
No system is perfect. I do not advise you to immediately introduce the newest features into production. Sure, there are bugs and missing features, but the same applies to Kubernetes.
Compared to Kubernetes, the Nomad community is not that big. Kubernetes already has about 75 commits and 000 contributors, while Nomad has about 2000 commits and 14 contributors. It will be difficult for Nomad to keep up with Kubernetes in speed, but it may not be necessary! It's a more specialized system, and the smaller community also means your pull request is more likely to be noticed and accepted than Kubernetes.
Summary
Takeaway: Don't use Kubernetes just because everyone else is doing it. Carefully evaluate your requirements and check which tool is more profitable.
If you are planning to deploy a lot of homogeneous services on a large-scale infrastructure, then Kubernetes is a good option. Just be aware of the added complexity and running costs. Some costs can be avoided by using a managed Kubernetes environment such as or .
If you're just looking for a solid orchestrator that's easy to maintain and extensible, why not give Nomad a try? You may be surprised how far this will take you.
If Kubernetes is like a car, Nomad is a scooter. Sometimes you need one and sometimes the other. Both have the right to exist.
Source: habr.com