Hi all! A few months ago, we launched our new open-source project into production - Grafana-plugin for monitoring kubernetes, which we called . Plugin source code is available at . And in this article, we want to share with you the story of how we created the plugin, what tools we used, and what pitfalls we encountered in the development process. Let's go!
Part 0 - introductory: how did we get to this point?
The idea to write our own plug-in for Grafan was born quite by accident. Our company has been monitoring web-projects of various levels of complexity for more than 10 years. During this time, we have gained a lot of expertise, interesting cases, and experience in using various monitoring systems. And at some point we asked ourselves: “Is there a magic tool for monitoring Kubernetes, so that, as they say, “set it and forget it”?” And as ready-made solutions for this stack, there is a large set of various kinds of tools: prometheus-operator, a set of kubernetes-mixin dashboards, grafana-kubernetes-app.
The grafana-kubernetes-app plugin seemed to be the most interesting option for us, but it has not been supported for more than a year and, moreover, does not know how to work with new versions of node-exporter and kube-state-metrics. And at some point we decided: “Shouldn’t we make our own decision?”
What ideas we decided to implement in our plugin:
- visualization of the “application map”: convenient presentation of applications in a cluster, grouped by namespaces, deployments…;
- visualization of links like "deployment - service (+ports)".
- visualization of the distribution of cluster applications by cluster nodes.
- collection of metrics and information from several sources: Prometheus and k8s api server.
- monitoring both the infrastructure part (the use of processor time, memory, disk subsystem, network) and the application logic - the health-status of pods, the number of available replicas, information about the passage of liveness/readyness-probes.
Part 1: What is a "plugin for Grafana"?
From a technical point of view, a plugin for Grafana is an angular controller that is stored in Grafana's data directory (/var/grafana/plugins/ /dist/module.js) and can be loaded as a SystemJS module. Also in this directory there should be a plugin.json file containing all the meta information about your plugin: name, version, plugin type, links to the repository / site / license, dependencies, and so on.
module.ts
plugin.json
As you can see in the screenshot, we have specified plugin.type = app. For plugins for Grafana can be of three types:
panel: the most common type of plugins - is a panel for visualizing any metrics, used to build various dashboards.
data source: plug-in connector to any data source (for example, Prometheus-datasource, ClickHouse-datasource, ElasticSearch-datasource).
app: A plugin that allows you to build your own front-end application inside Grafana, create your own html pages, and manually access the datasource to render various data. Plugins of other types (datasource, panel) and various dashboards can also be used as dependencies.
Example plugin dependencies with type = app.
As a programming language, you can use both JavaScript and TypeScript (we chose it). You can create templates for hello-world plugins of any type : this repository contains a large number of starter-packs (there is even an experimental example of a React plugin) with pre-installed and configured builders.
Part 2: preparing the local environment
To work on the plugin, of course, we need a kubernetes cluster with all the pre-installed tools: prometheus, node-exporter, kube-state-metrics, grafana. The environment should be set up quickly, easily and naturally, and to ensure hot-reload the Grafana data directory should be mounted directly from the developer's machine.
The most convenient, in our opinion, way to work locally with kubernetes is . The next step is to install the Prometheus + Grafana bundle using the prometheus-operator. IN the process of installing prometheus-operator on minikube is described in detail. To enable persistence, you must set the parameter persistence: true in the charts/grafana/values.yaml file, add your own PV and PVC and specify them in the persistence.existingClaim parameter
The final minikube startup script looks like this:
minikube start --kubernetes-version=v1.13.4 --memory=4096 --bootstrapper=kubeadm --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0
minikube mount
/home/sergeisporyshev/Projects/Grafana:/var/grafana --gid=472 --uid=472 --9p-version=9p2000.LPart 3: direct development
Object Model
As a preparation for the implementation of the plugin, we decided to describe all the basic Kubernetes entities that we will work with in the form of TypeScript classes: pod, deployment, daemonset, statefulset, job, cronjob, service, node, namespace. Each of these classes inherits from the common BaseModel class, which describes the constructor, destructor, methods for updating and switching visibility. Each of the classes describes nested relationships with other entities, for example, a list of pods for an entity of deployment type.
import {Pod} from "./pod";
import {Service} from "./service";
import {BaseModel} from './traits/baseModel';
export class Deployment extends BaseModel{
pods: Array<Pod>;
services: Array<Service>;
constructor(data: any){
super(data);
this.pods = [];
this.services = [];
}
}With the help of getters and setters, we can display or set the entity metrics we need in a convenient and readable form. For example, the formatted output of allocatable cpu nods:
get cpuAllocatableFormatted(){
let cpu = this.data.status.allocatable.cpu;
if(cpu.indexOf('m') > -1){
cpu = parseInt(cpu)/1000;
}
return cpu;
}Pages
The list of all pages of our plugin is initially described in our pluing.json in the dependencies section:
In the block for each page, we must specify the PAGE NAME (it will then be converted into a slug by which this page will be available); the name of the component responsible for the operation of this page (the list of components is exported to module.ts); specifying the user role for which work with this page is available and navigation settings for the sidebar.
In the component responsible for the page, we must set the templateUrl by passing the path to the html file with the markup there. Inside the controller, through dependency injection, we can access up to 2 important angular services:
- backendSrv - a service that provides interaction with the grafana api-server;
- datasourceSrv - a service that provides local interaction with all datasources installed in your Grafana (for example, the .getAll() method - returns a list of all installed datasources; .get( ) - returns an instance object of a specific datasource.
Part 4: data source
From the point of view of Grafana, datasource is exactly the same plugin as all the others: it has its own module.js entry point, there is a plugin.json file with meta information. When developing a plugin with type = app, we can interact with both existing datasources (for example, prometheus-datasource), and our own, which we can store directly in the plugin directory (dist/datasource/*) or install as a dependency. In our case, the datasource comes with the plugin code. It is also necessary to have a config.html template and a ConfigCtrl controller that will be used for the datasource instance configuration page and a Datasource controller that implements your datasource's logic.
In the KubeGraf plugin, in terms of user interface, datasource is an instance of a kubernetes cluster that implements the following features (source code is available ):
- fetching data from k8s api-server (getting a list of namespaces, deployments…)
- proxying requests to prometheus-datasource (which is selected in the plugin settings for each specific cluster) and formatting responses to use data both in static pages and in dashboards.
- updating data on static pages of the plugin (with a set refresh rate time).
- processing queries to form a template sheet in grafana-dashboards (method .metriFindQuery())
- connection test with the final k8s cluster.
testDatasource(){
let url = '/api/v1/namespaces';
let _url = this.url;
if(this.accessViaToken)
_url += '/__proxy';
_url += url;
return this.backendSrv.datasourceRequest({
url: _url,
method: "GET",
headers: {"Content-Type": 'application/json'}
})
.then(response => {
if (response.status === 200) {
return {status: "success", message: "Data source is OK", title: "Success"};
}else{
return {status: "error", message: "Data source is not OK", title: "Error"};
}
}, error => {
return {status: "error", message: "Data source is not OK", title: "Error"};
})
}A separate interesting point, in our opinion, is the implementation of the authentication and authorization mechanism for the datasource. As a rule, out of the box, to configure access to the final data source, we can use the built-in Grafana component - datasourceHttpSettings. Using this component, we can configure access to the http data source by specifying the url and basic authentication/authorization settings: login-password, or client-cert/client-key. In order to implement the ability to configure access using a bearer token (the de facto standard for k8s), I had to do a bit of “cheating”.
To solve this problem, you can use the built-in Grafana "Plugin Routes" mechanism (more on ). In the settings of our datasource, we can declare a set of routing rules that will be processed by the grafana proxy server. For example, for each individual endpoint, there is the possibility of setting headers or url with the possibility of templating, the data for which can be taken from the jsonData and secureJsonData fields (for storing passwords or tokens in encrypted form). In our example, queries like /__proxy/api/v1/namespaces will be proxied to the url of the form
/api/v8/namespaces with header Authorization: Bearer .
Naturally, to work with the k8s api server, we need a user with readonly access, the manifests for creating which you can also find in .
Part 5: release
Once you've written your own Grafana plugin, you'll naturally want to make it publicly available. In Grafana, this is a plugin library available at the link
In order for your plugin to be available in the official store, you need to make a PR in , adding the following content to the repo.json file:
where version is the version of your plugin, url is a link to the repository, and commit is the hash of the commit that will make the specific plugin version available.
And at the output you will see a wonderful picture of the form:
The data for it will be automatically robbed from your Readme.md, Changelog.md and the plugin.json file with the plugin description.
Part 6: instead of conclusions
We didn't stop developing our plugin after the release. And now we are working on the correct monitoring of the use of resources of the cluster nodes, the introduction of new features to improve UX, and we are also raking in a large amount of feedback received after installing the plugin both by our clients and from ishshui on github (if you leave your issue or pull request, I I will be very happy 🙂).
We hope that this article will help you understand such a great tool as Grafana and, perhaps, write your own plugin.
Thank you!)
Source: habr.com