{"id":75585,"date":"2020-03-27T07:42:28","date_gmt":"2020-03-27T05:42:28","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko"},"modified":"2020-03-27T07:42:28","modified_gmt":"2020-03-27T05:42:28","slug":"extendedpromql-rasshifrovka-doklada-romana-havronenko","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko","title":{"rendered":"\"ExtendedPromQL\" \u2014 decoding the report by Roman Khavronenko","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><strong>I invite you to review the interpretation of Roman Khavronenko's report \"ExtendedPromQL\"<\/strong><\/p>\n<p>\n<center><div class=\"youtube-placeholder\" data-id=\"lWYaiagvmfw\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/lWYaiagvmfw\/hqdefault.jpg\" alt=\"Play video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/54a175d9c74c90d9d313f8ff71476638.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A brief introduction about myself. My name is Roman. I work at CloudFlare, I live in London. However, I am also a maintainer of VictoriaMetrics.<br \/>\nAnd I am the author of <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Vertamedia\/clickhouse-grafana\">the ClickHouse plugin<\/a><\/noindex> for Grafana and <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/Vertamedia\/chproxy\">ClickHouse-proxy<\/a><\/noindex> \u2013 a small proxy for ClickHouse.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/fc1aa56b144def5db9fcee844d5016f1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We will start with the first part, titled 'Translation Challenges', in which I will talk about how any language, or even just a means of communication, is very important. Because it is how you convey your thoughts to another person or system and how you formulate a request. People on the internet debate which language is better \u2013 Java or some other. For myself, I decided that it should be chosen based on the task because it is all specific.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/ea21f52ba5c2d3507e0106715c4ab96d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let's start from the very beginning. What is PromQL? PromQL is the Prometheus Query Language. It is how we form queries in Prometheus to retrieve time series data.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/56b1ef4cc04defdffeea8b2666fdc354.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>What is time series data? Literally, it consists of three parameters. <\/p>\n<p><\/p>\n<p>They are:<\/p>\n<p><\/p>\n<ul>\n<li>What we are looking at. <\/li>\n<li>When we are looking at it.<\/li>\n<li>And what value it shows. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/358a193e889204d28debe1b74b93df3a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>If we look at this chart (this chart is from my phone, showing my step statistics), we can quickly answer these questions. <\/p>\n<p><\/p>\n<p>We are looking at steps. We see the value and the time when we are looking at it. That is, by looking at this chart, it is easy to say that on Sunday I took about 15,000 steps. This is time series data. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/859299ee9d25a3419c869b3ed6db6e72.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Now let's \"break down\" (transform) them into another data model in the form of a table. Here we also have what we are looking at. I've added some extra data that we will refer to as metadata, meaning that it wasn't just me, but two people, let's say, Jay and Silent Bob. This is what we are looking at; what it shows and when it displays this value. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/fd7b48ea3c25b57c13605e84c4a52a2c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nNow let's try to save all this data in a database. As an example, I used the ClickHouse syntax. And here we are creating a table called 'Steps', that is, what we are looking at. Here we have the time when we are observing it; what it shows and some metadata where we will store who it is: Jay and Silent Bob.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/bd01f9dc0b29d34aacbb8f012710a94f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And in order to attempt to visualize all this, we will use Grafana, because, firstly, it is beautiful.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/9f573241e9ae32e31d6d0b879115970b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We will also use this plugin for two reasons. First, because I wrote it. And I know exactly how difficult it is to pull time series data from ClickHouse to display in Grafana. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/29f2d72f9995afb54b23bb7b4230b840.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We will display it in the Graph Panel. This is the most popular panel in Grafana, which shows the relationship between values over time, so we only need two parameters.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/d820f63944ce0112660337b44a6c1f36.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nLet\u2019s write the simplest query to show step statistics in Grafana, storing this data in ClickHouse, in the table we created. Here\u2019s a simple query. We are selecting from steps. We select the value and the timestamp of these values, i.e., the same three parameters we mentioned.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/ad7895105050178de5bc8a154da7e6e1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>As a result, we will get a chart like this. Does anyone know why it looks so strange?<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/4a8108737e834cf35b7834be7b4db719.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>That's right, we need to sort by time.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/d37343c7a6a811e2012f1cbae4b9a5d5.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And in the end, we will get a better chart, but it still looks strange. Does anyone know why? That's right, there are two participants, and we are sending two time series in Grafana, because if we revisit the data model, each time series is a unique combination of name and all key-value labels.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/f19bf1a9f43ae8de22af6bf7b494902f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>So we need to select a specific person. We\u2019re selecting Jay.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/b6a714d5d6ce4d4b27634da9414aa6f6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And let\u2019s plot it again. Now the chart looks accurate. Now it's a normal chart, and everything works well.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/2a93b4cd548239c9314c3283c5b9f488.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And you probably know how to do something similar in Prometheus using PromQL. It looks something like this. A bit easier. We are still looking at Steps and filtering by Jay. Here, we don\u2019t specify that we need to get a value and we don\u2019t select the time. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/5e080a1b88c156522af21bf00f3cd641.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Now let\u2019s try to calculate Jay's or Silent Bob's speed. In ClickHouse, we will need to use runningDifference, which means calculating the difference between pairs of points and dividing by time to get the exact speed. The query will look something like this.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/13102b04ba4cf0ff9c955c36801044d0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And it will show values like this, meaning that Silent Bob or Jay takes about 1.8 steps per second.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/1c328f99da6f160bdca23ab5f461bda8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And in Prometheus, you know how to do this as well. It's much simpler than it was before.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/a7145785dee9fafcc8ab573e920cb88e.jpeg\" style=\"display:block;margin: 0 auto;\" \/>To make it just as easy in Grafana, I added this wrapper that looks very similar to PromQL. It's called Rate Macros, or whatever you want to call it. In Grafana, you simply write \"rate,\" but deep down it transforms into a large query like this. You don\u2019t even need to look at it; it\u2019s there in the background, saving you a lot of time because writing such large SQL queries is always time-consuming. You can easily make a mistake and then take a long time to understand what\u2019s going on.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/511bb35469567b07412a10bb7c0b7312.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And here\u2019s a query that didn\u2019t even fit on one slide, so I had to split it into two columns. This is also a query in ClickHouse that does the same rate but for both time series: Silent Bob and Jay, so we have two time series on the panel. And this is already very complicated, in my opinion. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/c9c1e6b58672069370153a151dceef3d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>For Prometheus, it will be sum(rate). For ClickHouse, I created a separate macro called RateColumns that looks like a query in Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/9743c6a4b3de4aab7cfb5bd3fd50fcfd.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We looked at it, and PromQL seems great, but of course, it has some limitations.<\/p>\n<p><\/p>\n<p>They are:<\/p>\n<p><\/p>\n<ul>\n<li>Limited SELECT.<\/li>\n<li>Boundary JOIN.<\/li>\n<li>No support for HAVING.<\/li>\n<\/ul>\n<p><\/p>\n<p>And if you've worked with it extensively, you know that sometimes it's very difficult to do something in PromQL, while you can almost do anything in SQL because all these variants we just discussed could be implemented in SQL. But would it be convenient to use? This leads me to think that not always the most powerful language is the most convenient.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/ec8786f1d72a1f8652fa631cdffc9c25.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Therefore, sometimes you need to choose a language based on the tasks. It\u2019s like the battle between Batman and Superman. Clearly, Superman is stronger, but Batman was able to defeat him because he was more practical and knew exactly what he was doing.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/59e5cb7e54ee14705a9e8a36db6f5541.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The next part is Extending PromQL.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/b806acf0e6c7c416614d3fd49004b4be.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Once again about VictoriaMetrics. What is VictoriaMetrics? It\u2019s a time series database, open-source, and we distribute both single and cluster versions. According to our benchmarks, it is the fastest currently available on the market and similarly efficient in compression, with real users reporting compression rates of about 0.4 bytes per point, compared to Prometheus, which is 1.2-1.4. <\/p>\n<p><\/p>\n<p>We support not only Prometheus. We also support InfluxDB, Graphite, and OpenTSDB.<\/p>\n<p><\/p>\n<p>We can \"write\" to it, meaning we can transfer old data. <\/p>\n<p><\/p>\n<p>And we work perfectly with Prometheus and Grafana, meaning we support the PromQL engine. In Grafana, you can simply change the Prometheus endpoint to VictoriaMetrics, and all your dashboards will function just as they did. <\/p>\n<p><\/p>\n<p>But you can also use additional features provided by VictoriaMetrics.<\/p>\n<p><\/p>\n<p>We will quickly go through the functions that we have added.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/78e9c0ce91fad48ab859fc2cf06a0689.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Omit interval param \u2013 you can skip the interval parameters in Grafana. When you want to avoid strange graphs during zooming in\/out on the panel, it is recommended to use the variable <code>$__interval<\/code>. This is an internal Grafana variable that chooses the data range itself. VictoriaMetrics also understands what this range should be. You won't need to update all your queries, making it much simpler.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/4094c7b8b5e6eef6134b5aa344ef7409.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The second feature is interval referencing. You can use this interval in your expressions. You can multiply, divide, pass, and reference it.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/967ca6166e588987bd69c1e1953dca31.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Next is the family of rollup functions. A rollup function transforms any of your time series into three separate time series: min, max, and avg. I find this very convenient because it can sometimes reveal outliers and inaccuracies. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/d5ff7d6f6ae8f7550434a3dbbca09b20.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And if you're just using irate or rate, you might miss some cases where the time series behaves differently than expected. With this function, it's much easier to see, for example, when max is significantly higher than avg.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/a4e9f0d246fa33459597f7140e09538c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Next is the default variable. Default specifies what value should be drawn in Grafana when there is no time series available at that moment. When does this happen? For example, if you\u2019re exporting some error metrics and you have such a great application that when you start, you have no errors, not even for the next three hours or even a day. If you have dashboards showing the ratio of success to error, they will display nothing because you have no error metric. However, in default, you can specify any value.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/cb4e23199194006186ac6907d2335cf6.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Keep_last_Value \u2013 retains the last value of the metric if it disappears. If Prometheus does not find it during the next scrape for 5 minutes, we will remember its last value, and your graphs won\u2019t break again.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/ca202f8a70f071cbd85b33b98cd28bb5.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Scrape_interval \u2013 shows how often Prometheus collects data on your metric, at what frequency. Here, you can see gaps, for example. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/9e9f43c7e9ef9a24dcc0b43673558046.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nLabel replace \u2013 a popular function. However, we consider it somewhat complex because it takes several arguments. You need to not only remember 5 arguments but also recall their sequence.<br \/>\n<img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/adfee335e9329f6b03d6f8a42ad8e7b8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nSo, why not simplify them? That is, break them down into smaller functions with clear syntax.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/e928549205018e7405cc2a81970a8514.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And now the most interesting part. Why do we consider this extended PromQL? Because we support Common Table Expressions. You can scan the QR code (<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/wiki\/ExtendedPromQL\">https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/wiki\/ExtendedPromQL<\/a><\/noindex>), check links with examples, and play around where you can execute queries directly in VictoriaMetrics without installing it, simply in your browser.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/33259a4bcd1a92f2cab6e943b73502ad.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>So, what is this all about? That query above is quite a popular one. I believe you use the same filter across many dashboards in various companies. That's usually how it goes. But when you need to add a new filter, you end up updating each panel, or downloading the dashboard, opening it in JSON, and doing a find and replace, which also takes time. Why not store that value in a variable and reuse it? It seems much easier and clearer, in my opinion.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/60aea73cf49934ef7d9a7928a8e2f8e7.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>For instance, when I need to update filters in Grafana across all queries, and the dashboard might be huge or there might even be several dashboards. How would I like to solve this problem in Grafana?<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/7cf9eb13219bbd9a5d3240bdda1839a2.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>I solve this issue like this: I create a commonFilter and define this filter in it, and then I reuse it in the queries. But if you do the same now, it won't work because Grafana doesn\u2019t allow you to use variables within query variables. And that's a bit strange.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/0b7a40976cb3eeda370518a242f78708.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And that's why I created a variant that allows doing this. And if you're interested or want this feature, please support or dislike it if you don\u2019t like the idea. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/grafana\/grafana\/pull\/16694\">https:\/\/github.com\/grafana\/grafana\/pull\/16694<\/a><\/noindex><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/68e3a408ff2b554cdfd61a2da4c75bc0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Next, about PromQL extended. Here, we not only define a variable but a whole function. We call it ru (resource usage). This function takes free resources, resource limits, and a filter. The syntax seems simple overall. It's very easy to use this function and calculate the percentage of free memory we have. That is, how much memory we have, what the limit is, and how to filter. It would be much more convenient if you wrote all this by reusing the same filters, as that would turn into a large, large query.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/4ca66e18ac665fe1038493709530b82b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Here is an example of such a big request. It comes from the official NodeExporter dashboard for Grafana. However, I have a hard time understanding what is happening here. I mean, of course I can see it if I look closely, but the number of brackets can immediately decrease the motivation to figure out what\u2019s going on. So why not make it simpler and clearer?<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/50e01e7f0c1d03965b028266130413f0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>For instance, like this, highlighting significant things or parts into variables. Then we can perform our basic mathematics. This already looks more like programming, which is what I would like to see in the future in Grafana.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/81f957d89f97ebf686abdf969982d3ba.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Here is a second example of how we could make it even simpler if we already had this function available, which it is in VictoriaMetrics. Then you simply pass the cached value that you declared in the CTE.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/415302741e14232228353db6ec57a2e1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>I have already mentioned how important it is to use the right programming language. And probably, in every company, something unique is happening in Grafana. Also, you might grant access to Grafana to your developers and they do their own things. They all do it somewhat differently. It would be nice to have some uniformity, bringing it to a common standard.<\/p>\n<p><\/p>\n<p>Let\u2019s say you have not just system engineers; maybe you even have experts, DevOps, or SREs. Perhaps you have specialists who understand what monitoring is, know what Grafana is, meaning they have been working with it for years and know exactly how to do it right. They\u2019ve already written this a hundred times and explained it to everyone, but for some reason, no one listens.<\/p>\n<p><\/p>\n<p>What if they could directly embed this knowledge into Grafana so that other users could reuse the functions? And if they needed to calculate the percentage of free memory, they would simply apply the function. What if the creators of exporters, along with their product, also provided a set of functions for working with their metrics because they know exactly what those metrics are and how to calculate them correctly?<\/p>\n<p><\/p>\n<p>This idea actually doesn\u2019t exist yet. I came up with this myself. This is about library support in Grafana. Let\u2019s say the guys who created NodeExporter did what I mentioned and also provided a set of functions. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/755391ee41fd2abf4cb74fad17cecaf0.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>It looks something like this. You integrate this library into Grafana, you enter edit mode, and it\u2019s very simply outlined in JSON how to work with this metric. That is, there\u2019s a set of functions, their descriptions, and how they unfold.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/78a85dd219a421757807be895c173a39.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>I think this could be useful because then in Grafana you would write just like that. And Grafana \"tells\" you that there is a certain function from a specific library \u2013 let\u2019s use it. I believe this would be really cool.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/6e6b2b459c574d95d56251cd5fae5586.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A bit about VictoriaMetrics. We are doing a lot of interesting things. Check out our articles on compression, our competitions with other time series data applications, our explanations on how to work with PromQL, because there are still many beginners, as well as about vertical scalability and the competition with Thanos.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"&quot;ExtendedPromQL&quot; \u2014 decoding the report by Roman Khavronenko\" src=\"\/wp-content\/uploads\/2020\/03\/562829a25ee6131c7eaed8c561c6927c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Questions:<\/p>\n<p><\/p>\n<p><em>I\u2019ll start my question with a simple life story. When I first started using Grafana, I wrote a very convincing query that was 5 lines long. As a result, it produced a very convincing graph. That graph almost went into production. But upon closer inspection, it turned out that this graph showed complete nonsense, having nothing to do with reality, although the numbers fell within the range we expected to see. And my question is: we have libraries, we have functions, but how do we write tests for Grafana? You\u2019ve written a complex query that the business decision relies on \u2013 whether to order actual server containers or not. And how do we know this function, which draws the graph, resembles the truth? Thank you.<\/em><\/p>\n<p><\/p>\n<p>Thank you for the question. There are two parts to this. The first \u2013 based on my experience, I have the impression that most users, when looking at their graphs, don\u2019t understand what they are showing. For some reason, people are very good at coming up with excuses for any anomalies that occur on the graphs, even if it\u2019s an error within the function. And the second part \u2013 it seems to me that using such functions would be much better suited to solving your problem, instead of each of your developers making their own capacity planning and being wrong with some probability.<\/p>\n<p><\/p>\n<p><em>How to check?<\/em><\/p>\n<p><\/p>\n<p>How to check? Probably, there\u2019s no way. <\/p>\n<p><\/p>\n<p><em>In the form of a test in Grafana.<\/em><\/p>\n<p><\/p>\n<p>What does Grafana have to do with it? Grafana translates this query directly to the DataSource.<\/p>\n<p><\/p>\n<p><em>By adding a bit to the parameters.<\/em><\/p>\n<p><\/p>\n<p>No, nothing is added in Grafana. There can be GET parameters, like step for instance. It's not explicitly stated, but you can override it or leave it as is, and it gets added automatically. You can't write tests here. I think it's unwise to rely on Grafana as a source of truth.<\/p>\n<p><\/p>\n<p><em>Thank you for the presentation! Thanks for the compression! You mentioned mapping variables in the chart, that you can't use a variable within a variable in Grafana. Do you understand what I mean?<\/em><\/p>\n<p><\/p>\n<p>Yes.<\/p>\n<p><\/p>\n<p><em>This was initially a headache when I wanted to create an alert in Grafana. You need to create an alert for each host individually. Does what you created work for alerts in Grafana?<\/em><\/p>\n<p><\/p>\n<p>If Grafana doesn\u2019t handle variables differently, then yes, it will work. But my advice is not to use alerting in Grafana at all; it's better to use alertmanager.<\/p>\n<p><\/p>\n<p><em>Yes, I use it, but it just seemed easier to set up in Grafana, so thanks for the advice!<\/em><\/p>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/494034\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e &quot;ExtendedPromQL&quot; \u041a\u0440\u0430\u0442\u043a\u043e \u043e\u0431\u043e \u043c\u043d\u0435. \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0420\u043e\u043c\u0430\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 CloudFlare, \u0436\u0438\u0432\u0443 \u0432 \u041b\u043e\u043d\u0434\u043e\u043d\u0435. \u041d\u043e \u0442\u0430\u043a\u0436\u0435 \u044f \u043c\u0435\u0439\u0442\u0435\u043d\u0435\u0440 VictoriaMetrics. \u0418 \u044f \u0430\u0432\u0442\u043e\u0440 ClickHouse \u043f\u043b\u0430\u0433\u0438\u043d\u0430 \u0434\u043b\u044f Grafana \u0438 ClickHouse-proxy \u2013 \u044d\u0442\u043e \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0439 \u043f\u0440\u043e\u043a\u0441\u0438 \u0434\u043b\u044f ClickHouse. \u041c\u044b \u043d\u0430\u0447\u043d\u0435\u043c \u0441 \u043f\u0435\u0440\u0432\u043e\u0439 \u0447\u0430\u0441\u0442\u0438, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043d\u0430\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u00ab\u0421\u043b\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u0430\u00bb \u0438 \u0432 \u043d\u0435\u0439 \u044f \u0431\u0443\u0434\u0443 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":75586,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-75585","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e &quot;ExtendedPromQL&quot;\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u00abExtendedPromQL\u00bb \u2014 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u0430 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e &quot;ExtendedPromQL&quot;\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-27T05:42:28+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-27T05:42:28+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47\"ExtendedPromQL\" \u2014 the presentation by Roman Khavronenko | ProHoster","description":"I offer you to review the transcript of Roman Khavronenko's report \"ExtendedPromQL\"","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u00abExtendedPromQL\u00bb \u2014 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u0430 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0420\u043e\u043c\u0430\u043d\u0430 \u0425\u0430\u0432\u0440\u043e\u043d\u0435\u043d\u043a\u043e &quot;ExtendedPromQL&quot;","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/extendedpromql-rasshifrovka-doklada-romana-havronenko","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-27T05:42:28+00:00","article:modified_time":"2020-03-27T05:42:28+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"75585","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:54:47","updated":"2022-09-28 07:46:53","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/75585","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=75585"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/75585\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/75586"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=75585"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=75585"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=75585"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}