🥇Tools for developers of applications running on Kubernetes

A modern approach to operation solves many pressing business problems. Containers and orchestrators make it easy to scale projects of any complexity, simplify releases of new versions, make them more reliable, but at the same time create additional problems for developers. The programmer, first of all, cares about his code: architecture, quality, performance, elegance, and not how he will go to Kubernetes and how to test and debug it after making even minimal changes. Therefore, it is quite natural that tools for Kubernetes are actively developing, helping to solve the problems of even the most “archaic” developers and allowing them to focus on the main thing.

This overview provides brief information about some of the tools that make life easier for a programmer whose code is spinning in the pod'ax of a Kubernetes cluster.

Simple helpers

kubectl-debug

The bottom line: add your container to Pod and see what happens in it.
GitHub.
Brief GH stats: 715 stars, 54 commits, 9 contributors.
Language: Go.
License: Apache License 2.0.

This plugin for kubectl allows you to create an additional container inside the pod of interest, which will share the process namespace with other containers. In it, you can debug the work of the pod: check the network, listen to network traffic, strace the process of interest, etc.

You can also switch to the process container by executing chroot /proc/PID/root - this can be very convenient when you need to get a root shell in a container for which securityContext.runAs.

The tool is simple and effective, so it can be useful for every developer. We wrote more about it in separate article.

telepresence

The bottom line: transfer the application to your computer. Develop and debug locally.
Site; GitHub.
Brief GH stats: 2131 stars, 2712 commits, 33 contributors.
Language: Python.
License: Apache License 2.0.

The idea of this snap-in is to run a container with an application on the local user's computer and proxy all traffic from the cluster to it and back. This approach allows you to develop locally by simply modifying files in your favorite IDE: the results will be available immediately.

The advantages of local launch are the convenience of editing and instant results, the ability to debug the application in the usual way. Of the minuses is the demands on the connection speed, which is especially noticeable when you have to work with an application with a fairly high RPS and traffic. Also, Telepresence has issues with volume mounts on Windows, which can be a major limitation for developers used to this OS.

We have already shared our experience of using Telepresence here.

ksync

The bottom line: almost instant code synchronization with the container in the cluster.
GitHub.
Brief GH stats: 555 stars, 362 commits, 11 contributors.
Language: Go.
License: Apache License 2.0.

The utility allows you to synchronize the contents of the local directory with the directory of the container running in the cluster. Such a tool is perfect for developers in scripting programming languages, whose main problem is to deliver the code to a running container. Ksync aims to relieve this headache.

With a single initialization by the command ksync init a DaemonSet is created in the cluster, which is used to track the state of the file system of the selected container. On his local computer, the developer runs the command ksync watch, which monitors configurations and runs syncthing, which directly synchronizes files with the cluster.

It remains to instruct ksync what to synchronize with what. For example, this command:

ksync create --name=myproject --namespace=test --selector=app=backend --container=php --reload=false /home/user/myproject/ /var/www/myproject/

... will create a watcher named myproject, which will look for a pod with the label app=backend and try to synchronize the local directory /home/user/myproject/ with catalog /var/www/myproject/ in a container called php.

Issues and notes on ksync from our experience:

Kubernetes cluster nodes should use overlay2 as a storage driver for Docker. It will not work with any other utility.
When using Windows as a client OS, the file system watcher may not work correctly. This bug was noticed when working with large directories - with a large number of nested files and directories. We created corresponding issue in the syncthing project, but so far (since the beginning of July) there has been no progress on it.
Use File .stignore to specify paths or file patterns that do not need to be synchronized (for example, directories app/cache и .git).
By default, ksync will reload the container whenever files change. For Node.js, this is convenient, but for PHP it is completely redundant. It's better to disable opcache and use the flag --reload=false.
The configuration can always be corrected in $HOME/.ksync/ksync.yaml.

Squash

The bottom line: debug processes directly in the cluster.
GitHub.
Brief GH stats: 1154 stars, 279 commits, 23 contributors.
Language: Go.
License: Apache License 2.0.

This tool is intended for debugging processes directly in pods. The utility is simple and interactively allows you to select the desired debugger (see below) and namespace + pod, the process of which needs to be interfered with. Currently supported:

delve - for Go applications;
GDB - via target remote + port forwarding;
JDWP port forwarding for debugging Java applications.

From the IDE side, support is only available in VScode (using expansion), but the plans for the current (2019) year include Eclipse and Intellij.

To debug processes, Squash runs a privileged container on the cluster nodes, so you need to first familiarize yourself with the possibilities safe mode to avoid security issues.

Complex solutions

Let's move on to heavy artillery - more "large-scale" projects designed to immediately cover many of the needs of developers.

NB: In this list, of course, there is a place for our Open Source utility yard (formerly known as dapp). However, we have already written and talked about it more than once, and therefore decided not to include it in the review. For those who want to get acquainted with its capabilities, we recommend reading / listening to the report “werf is our tool for CI/CD in Kubernetes».

DevSpace

The bottom line: for those who want to start working in Kubernetes, but do not want to get deep into its wilds.
GitHub.
Brief GH stats: 630 stars, 1912 commits, 13 contributors.
Language: Go.
License: Apache License 2.0.

A solution from a company of the same name that provides managed clusters with Kubernetes for team development. The utility was created for commercial clusters, but works great with any other.

When running the command devspace init in the directory with the project you will be offered (in interactive mode):

select a working Kubernetes cluster,
use existing Dockerfile (or generate a new one) to create a container based on it,
select a repository for storing container images, etc.

After all these preparatory steps, you can start development by running the command devspace dev. It will build the container, upload it to the repository, roll out the deployment to the cluster, and start port forwarding and synchronization of the container with the local directory.

Optionally, you will be prompted to switch the terminal to the container. You should not refuse, because in reality the container starts with the sleep command, and for real testing, the application needs to be launched manually.

Finally the command devspace deploy rolls out the application and its associated infrastructure to the cluster, after which everything starts to function in combat mode.

All project configuration is stored in a file devspace.yaml. In addition to the development environment settings, you can also find a description of the infrastructure in it, similar to the standard Kubernetes manifests, only greatly simplified.

Architecture and main stages of working with DevSpace

In addition, it is easy to add a predefined component (for example, MySQL DBMS) or a Helm chart to the project. Read more in documentation - She's easy.

Skaffold

Site; GitHub.
Brief GH stats: 7423 stars, 4173 commits, 136 contributors.
Language: Go.
License: Apache License 2.0.

This utility from Google claims to cover all the needs of a developer whose code will somehow run in a Kubernetes cluster. Starting to use it is not as easy as devspace: no interactivity, language detection and auto-creation Dockerfile here you will not be offered.

However, if it doesn’t scare you, here’s what Skaffold allows you to do:

Track changes to source code.
Synchronize it with the pod's container if it doesn't need to be built.
Build containers with code if the language is interpreted, or compile artifacts and pack them into containers.
The resulting images are automatically checked using container-structure-test.
Tag and upload images to the Docker Registry.
Deploy an application on a cluster using kubectl, Helm, or kustomize.
Do port forwarding.
Debug applications written in Java, Node.js, Python.

Workflow in various variations is declaratively described in the file skaffold.yaml. You can also define several profiles for a project, in which you can partially or completely change the build and deployment stages. For example, for development, specify a base image convenient for the developer, and for staging and production - the minimum one (+ use securityContext containers or override the cluster in which the application will be deployed).

Docker containers can be built locally or remotely: Google Cloud Build or in a cluster with Kaniko. Bazel and Jib Maven/Gradle are also supported. For tagging, Skaffold supports many strategies: by git commit hash, date/time, sha256 sum of sources, etc.

Separately, it is worth noting the possibility of testing containers. The already mentioned container-structure-test framework offers the following test methods:

Execution of commands in the context of a container with tracking exit statuses and checking the text output of the command.
Checking for the presence of files in the container and matching attributes to the specified ones.
File content control by regular expressions.
Verification of image metadata (ENV, ENTRYPOINT, VOLUMES etc.).
Checking license compatibility.

Synchronization of files with the container is not carried out in the most optimal way: Skaffold simply creates an archive with sources, copies it and unpacks it in a container (tar must be installed). Therefore, if your main task is code synchronization, it is better to look towards a specialized solution (ksync).

Main stages of Skaffold work

In general, the tool does not allow you to abstract from Kubernetes manifests and does not have any interactivity, so it may seem difficult to master. But this is also its plus - greater freedom of action.

Garden

Site; GitHub.
Brief GH stats: 1063 stars, 1927 commits, 17 contributors.
Language: TypeScript (it is planned to split the project into several components, some of which will be in Go, and also make an SDK for creating add-ons in TypeScript/JavaScript and Go).
License: Apache License 2.0.

Like Skaffold, Garden aims to automate the delivery of application code to the K8s cluster. To do this, you first need to describe the project structure in a YAML file, and then run the command garden dev. She will do all the magic:

Will collect containers with various parts of the project.
Conduct integration and unit tests, if any.
Rolls out all project components to the cluster.
In case of a change in the source code, it will restart the entire pipeline.

The focus of this tool is on sharing a remote cluster with a development team. In this case, if some build and test steps have already been done, this will greatly speed up the whole process, since the Garden will be able to use the cached results.

A project module can be a container, a Maven container, a Helm chart, a manifest for kubectl apply or even an OpenFaaS feature. Moreover, any of the modules can be pulled from a remote Git repository. A module may or may not define services, tasks, and tests. Services and tasks can have dependencies, thanks to which you can determine the sequence of deployment of a particular service, streamline the launch of tasks and tests.

Garden provides the user with a beautiful dashboard (while in experimental state), which displays the project graph: components, assembly sequence, execution of tasks and tests, their relationships and dependencies. Right in the browser, you can also view the logs of all project components, check what a particular component produces via HTTP (if, of course, an ingress resource is declared for it).

Panel for Garden

This tool also has a hot-reload mode, which simply synchronizes script changes with the container in the cluster, greatly speeding up the debugging process of the application. Garden has a good documentation and not bad set of examplesallowing you to quickly get used to it and start using it. By the way, we recently published article translation from its authors.

Conclusion

Of course, the tools for developing and debugging applications in Kubernetes are not limited to this list. There are many more very useful and practical utilities worthy, if not a separate article, then at least a mention. Tell us what you use, what problems you have encountered and how you solved them!

Tools for developers of applications that run on Kubernetes