I offer you to review the transcript of Roman Khavronenko's report "ExtendedPromQL"
A brief introduction about myself. My name is Roman. I work at CloudFlare, I live in London. However, I am also a maintainer of VictoriaMetrics.
And I am the author of for Grafana and – a small proxy for ClickHouse.
We will start with the first part, titled 'Translation Challenges', in which I will talk about how any language, or even just a means of communication, is very important. Because it is how you convey your thoughts to another person or system and how you formulate a request. People on the internet debate which language is better – Java or some other. For myself, I decided that it should be chosen based on the task because it is all specific.
Let's start from the very beginning. What is PromQL? PromQL is the Prometheus Query Language. It is how we form queries in Prometheus to retrieve time series data.
What is time series data? Literally, it consists of three parameters.
They are:
- What we are looking at.
- When we are looking at it.
- And what value it shows.
If we look at this chart (this chart is from my phone, showing my step statistics), we can quickly answer these questions.
We are looking at steps. We see the value and the time when we are looking at it. That is, by looking at this chart, it is easy to say that on Sunday I took about 15,000 steps. This is time series data.
Now let's 'break down' (transform) them into another data model in the form of a table. Here we also have what we are looking at. I added some additional data here, which we will refer to as metadata, meaning that it wasn't just me, but two people, say, Jay and Silent Bob. This is what we are observing; what it shows and when it shows that value.
Now let's try to save all this data in a database. As an example, I used the ClickHouse syntax. And here we are creating a table called 'Steps', that is, what we are looking at. Here we have the time when we are observing it; what it shows and some metadata where we will store who it is: Jay and Silent Bob.
And in order to attempt to visualize all this, we will use Grafana, because, firstly, it is beautiful.
We will also use this plugin for two reasons. First, because I wrote it. And I know exactly how difficult it is to pull time series data from ClickHouse to display in Grafana.
We will display it in the Graph Panel. This is the most popular panel in Grafana, which shows the relationship between values over time, so we only need two parameters.
Let’s write the simplest query to show step statistics in Grafana, storing this data in ClickHouse, in the table we created. Here’s a simple query. We are selecting from steps. We select the value and the timestamp of these values, i.e., the same three parameters we mentioned.
As a result, we will get a chart like this. Does anyone know why it looks so strange?
That's right, we need to sort by time.
And in the end, we will get a better chart, but it still looks strange. Does anyone know why? That's right, there are two participants, and we are sending two time series in Grafana, because if we revisit the data model, each time series is a unique combination of name and all key-value labels.
So we need to select a specific person. We’re selecting Jay.
And let’s plot it again. Now the chart looks accurate. Now it's a normal chart, and everything works well.
And you probably know how to do something similar in Prometheus using PromQL. It looks something like this. A bit easier. We are still looking at Steps and filtering by Jay. Here, we don’t specify that we need to get a value and we don’t select the time.
Now let’s try to calculate Jay's or Silent Bob's speed. In ClickHouse, we will need to use runningDifference, which means calculating the difference between pairs of points and dividing by time to get the exact speed. The query will look something like this.
And it will show values like this, meaning that Silent Bob or Jay takes about 1.8 steps per second.
And in Prometheus, you know how to do this as well. It's much simpler than it was before.
To make it just as easy in Grafana, I added this wrapper that looks very similar to PromQL. It's called Rate Macros, or whatever you want to call it. In Grafana, you simply write "rate," but deep down it transforms into a large query like this. You don’t even need to look at it; it’s there in the background, saving you a lot of time because writing such large SQL queries is always time-consuming. You can easily make a mistake and then take a long time to understand what’s going on.
And here’s a query that didn’t even fit on one slide, so I had to split it into two columns. This is also a query in ClickHouse that does the same rate but for both time series: Silent Bob and Jay, so we have two time series on the panel. And this is already very complicated, in my opinion.
For Prometheus, it will be sum(rate). For ClickHouse, I created a separate macro called RateColumns that looks like a query in Prometheus.
We looked at it, and PromQL seems great, but of course, it has some limitations.
They are:
- Limited SELECT.
- Boundary JOIN.
- No support for HAVING.
And if you've worked with it extensively, you know that sometimes it's very difficult to do something in PromQL, while you can almost do anything in SQL because all these variants we just discussed could be implemented in SQL. But would it be convenient to use? This leads me to think that not always the most powerful language is the most convenient.
Therefore, sometimes you need to choose a language based on the tasks. It’s like the battle between Batman and Superman. Clearly, Superman is stronger, but Batman was able to defeat him because he was more practical and knew exactly what he was doing.
The next part is Extending PromQL.
Once again about VictoriaMetrics. What is VictoriaMetrics? It’s a time series database, open-source, and we distribute both single and cluster versions. According to our benchmarks, it is the fastest currently available on the market and similarly efficient in compression, with real users reporting compression rates of about 0.4 bytes per point, compared to Prometheus, which is 1.2-1.4.
We support not only Prometheus. We also support InfluxDB, Graphite, and OpenTSDB.
You can 'write' to us, meaning you can transfer old data.
And we work perfectly with Prometheus and Grafana, meaning we support the PromQL engine. In Grafana, you can simply change the Prometheus endpoint to VictoriaMetrics, and all your dashboards will function just as they did.
But you can also use additional features provided by VictoriaMetrics.
We will quickly go through the functions that we have added.
Omit interval param – you can skip the interval parameters in Grafana. When you want to avoid strange graphs during zooming in/out on the panel, it is recommended to use the variable $__interval. This is an internal Grafana variable that chooses the data range itself. VictoriaMetrics also understands what this range should be. You won't need to update all your queries, making it much simpler.
The second feature is interval referencing. You can use this interval in your expressions. You can multiply, divide, pass, and reference it.
Next is the family of rollup functions. A rollup function transforms any of your time series into three separate time series: min, max, and avg. I find this very convenient because it can sometimes reveal outliers and inaccuracies.
And if you're just using irate or rate, you might miss some cases where the time series behaves differently than expected. With this function, it's much easier to see, for example, when max is significantly higher than avg.
Next is the default variable. Default specifies what value should be drawn in Grafana when there is no time series available at that moment. When does this happen? For example, if you’re exporting some error metrics and you have such a great application that when you start, you have no errors, not even for the next three hours or even a day. If you have dashboards showing the ratio of success to error, they will display nothing because you have no error metric. However, in default, you can specify any value.
Keep_last_Value – retains the last value of the metric if it disappears. If Prometheus does not find it during the next scrape for 5 minutes, we will remember its last value, and your graphs won’t break again.
Scrape_interval – shows how often Prometheus collects data on your metric, at what frequency. Here, you can see gaps, for example.
Label replace – a popular function. However, we consider it somewhat complex because it takes several arguments. You need to not only remember 5 arguments but also recall their sequence.
So, why not simplify them? That is, break them down into smaller functions with clear syntax.
And now the most interesting part. Why do we consider this extended PromQL? Because we support Common Table Expressions. You can scan the QR code (), check links with examples, and play around where you can execute queries directly in VictoriaMetrics without installing it, simply in your browser.
So, what is this all about? That query above is quite a popular one. I believe you use the same filter across many dashboards in various companies. That's usually how it goes. But when you need to add a new filter, you end up updating each panel, or downloading the dashboard, opening it in JSON, and doing a find and replace, which also takes time. Why not store that value in a variable and reuse it? It seems much easier and clearer, in my opinion.
For instance, when I need to update filters in Grafana across all queries, and the dashboard might be huge or there might even be several dashboards. How would I like to solve this problem in Grafana?
I solve this issue like this: I create a commonFilter and define this filter in it, and then I reuse it in the queries. But if you do the same now, it won't work because Grafana doesn’t allow you to use variables within query variables. And that's a bit strange.
And that's why I created a variant that allows doing this. And if you're interested or want this feature, please support or dislike it if you don’t like the idea.
Next, about PromQL extended. Here, we not only define a variable but a whole function. We call it ru (resource usage). This function takes free resources, resource limits, and a filter. The syntax seems simple overall. It's very easy to use this function and calculate the percentage of free memory we have. That is, how much memory we have, what the limit is, and how to filter. It would be much more convenient if you wrote all this by reusing the same filters, as that would turn into a large, large query.
Here is an example of such a big request. It comes from the official NodeExporter dashboard for Grafana. However, I have a hard time understanding what is happening here. I mean, of course I can see it if I look closely, but the number of brackets can immediately decrease the motivation to figure out what’s going on. So why not make it simpler and clearer?
For instance, like this, highlighting significant things or parts into variables. Then we can perform our basic mathematics. This already looks more like programming, which is what I would like to see in the future in Grafana.
Here is a second example of how we could make it even simpler if we already had this function available, which it is in VictoriaMetrics. Then you simply pass the cached value that you declared in the CTE.
I have already mentioned how important it is to use the right programming language. And probably, in every company, something unique is happening in Grafana. Also, you might grant access to Grafana to your developers and they do their own things. They all do it somewhat differently. It would be nice to have some uniformity, bringing it to a common standard.
Let’s say you have not just system engineers; maybe you even have experts, DevOps, or SREs. Perhaps you have specialists who understand what monitoring is, know what Grafana is, meaning they have been working with it for years and know exactly how to do it right. They’ve already written this a hundred times and explained it to everyone, but for some reason, no one listens.
What if they could directly embed this knowledge into Grafana so that other users could reuse the functions? And if they needed to calculate the percentage of free memory, they would simply apply the function. What if the creators of exporters, along with their product, also provided a set of functions for working with their metrics because they know exactly what those metrics are and how to calculate them correctly?
This idea actually doesn’t exist yet. I came up with this myself. This is about library support in Grafana. Let’s say the guys who created NodeExporter did what I mentioned and also provided a set of functions.
It looks something like this. You integrate this library into Grafana, you enter edit mode, and it’s very simply outlined in JSON how to work with this metric. That is, there’s a set of functions, their descriptions, and how they unfold.
I think this could be useful because then in Grafana you would write just like that. And Grafana "tells" you that there is such a function from this library – let’s use it. It seems to me that this would be very great.
A bit about VictoriaMetrics. We are doing a lot of interesting things. Check out our articles on compression, our competitions with other time series data applications, our explanations on how to work with PromQL, because there are still many beginners, as well as about vertical scalability and the competition with Thanos.
Questions:
I’ll start my question with a simple life story. When I first started using Grafana, I wrote a very convincing query that was 5 lines long. As a result, it produced a very convincing graph. That graph almost went into production. But upon closer inspection, it turned out that this graph showed complete nonsense, having nothing to do with reality, although the numbers fell within the range we expected to see. And my question is: we have libraries, we have functions, but how do we write tests for Grafana? You’ve written a complex query that the business decision relies on – whether to order actual server containers or not. And how do we know this function, which draws the graph, resembles the truth? Thank you.
Thank you for the question. There are two parts to this. The first – based on my experience, I have the impression that most users, when looking at their graphs, don’t understand what they are showing. For some reason, people are very good at coming up with excuses for any anomalies that occur on the graphs, even if it’s an error within the function. And the second part – it seems to me that using such functions would be much better suited to solving your problem, instead of each of your developers making their own capacity planning and being wrong with some probability.
How to check?
How to check? Probably, there’s no way.
In the form of a test in Grafana.
What does Grafana have to do with it? Grafana translates this query directly to the DataSource.
By adding a bit to the parameters.
No, nothing is added in Grafana. There can be GET parameters, like step for instance. It's not explicitly stated, but you can override it or leave it as is, and it gets added automatically. You can't write tests here. I think it's unwise to rely on Grafana as a source of truth.
Thank you for the presentation! Thanks for the compression! You mentioned mapping variables in the chart, that you can't use a variable within a variable in Grafana. Do you understand what I mean?
Yes.
This was initially a headache when I wanted to create an alert in Grafana. You need to create an alert for each host individually. Does what you created work for alerts in Grafana?
If Grafana doesn’t handle variables differently, then yes, it will work. But my advice is not to use alerting in Grafana at all; it's better to use alertmanager.
Yes, I use it, but it just seemed easier to set up in Grafana, so thanks for the advice!
Source: habr.com
