{"id":54601,"date":"2019-12-30T00:00:00","date_gmt":"2019-12-29T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics"},"modified":"2020-02-18T14:02:38","modified_gmt":"2020-02-18T11:02:38","slug":"vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics","title":{"rendered":"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Hello everyone. Below is the transcript <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=HyOXAdQE0Pk\">of the report from Big Monitoring Meetup 4<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><strong><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/prometheus\">Prometheus<\/a><\/noindex><\/strong> \u2013 a monitoring system for various systems and services, which allows system administrators to collect information about the current parameters of systems and set up alerts to receive notifications about deviations in system performance.<\/p>\n<p><\/p>\n<p>The report will compare <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\">Thanos<\/a><\/noindex> and <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">VictoriaMetrics<\/a><\/noindex> \u2014 projects for long-term storage of Prometheus metrics.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/b65866353c1f816a8cb2a90af1ceac23.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>\n<center><div class=\"youtube-placeholder\" data-id=\"HyOXAdQE0Pk\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/HyOXAdQE0Pk\/hqdefault.jpg\" alt=\"Play video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/dc3c90711a8102d3e70b972ca47ec152.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>First, let me tell you about Prometheus. It's a monitoring system that collects metrics from specified targets and stores them in a local repository. Prometheus can write metrics to remote storage, generate alerts, and create recording rules. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/e9852597e74c675d2216d971f19857c2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h3 id=\"ogranicheniya-prometheus\">Limitations of Prometheus:<\/h3>\n<p><\/p>\n<ul>\n<li>It lacks a global query view. This is when you have multiple independent instances of Prometheus. They collect metrics, and you want to query across all these metrics collected from different Prometheus instances. Prometheus does not allow this.<\/li>\n<li>The performance of Prometheus is limited to a single server. Prometheus cannot automatically scale across multiple servers. You can only manually distribute your targets across several Prometheuses.<\/li>\n<li>The volume of metrics in Prometheus is limited to a single server for the same reason it cannot automatically scale to multiple servers.<\/li>\n<li>In Prometheus, organizing data retention is not straightforward.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/c140d41f1fe39d33e1fc84f5cc111481.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h3 id=\"resheniya-etih-problemzadach\">What are the solutions to these problems\/tasks?<\/h3>\n<p><\/p>\n<p>The solutions are:<\/p>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\">Thanos<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">VictoriaMetrics<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/m3db\">Uber M3<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/cortexproject\/cortex\">Cortex<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<p>All these solutions are for remote data storage collected by Prometheus. They address the remote storage issue from the previous slide in different ways. In this presentation, I will only cover the first two solutions: <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\">Thanos<\/a><\/noindex> and <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">VictoriaMetrics<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>The first information about <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\">Thanos<\/a><\/noindex> appeared in <noindex><a rel=\"nofollow\" href=\"https:\/\/improbable.io\/blog\/thanos-prometheus-at-scale\">this link<\/a><\/noindex>. It describes the architecture <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\">Thanos<\/a><\/noindex> and how it works.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/317db6bb4c3cd69d8742addb2bae16a4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos takes data that Prometheus has saved on the local disk and copies it to S3, to <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/storage\/\">GCS<\/a><\/noindex> or another object storage.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/7cf5f99dc5ce78f22c5956373812c0aa.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thus, Thanos provides a global query view. You can query data stored in object storage from multiple Prometheus instances.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/286cf441fe18fb756c2708086872d8d8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos supports PromQL and <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/api\/\">Prometheus querying API<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6a750070d76414370501a8153cd48007.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos uses Prometheus code for data storage.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/ba0ca17154f9407900efc336f213ba07.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos is developed by the same developers as Prometheus.<\/p>\n<p><\/p>\n<p>About <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">VictoriaMetrics<\/a><\/noindex>. Here\u2019s <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/faun\/victoriametrics-creating-the-best-remote-storage-for-prometheus-5d92d66787ac\">link<\/a><\/noindex>, where we first talked about <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">VictoriaMetrics<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a901d016e6e58c40c6c7c96b1cd5a8c9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics retrieves data from multiple Prometheuses via <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/docs\/practices\/remote_write\/\">remote write API<\/a><\/noindex> protocol supported by Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/24798414102cc63a1a1062c5ae327f4c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics provides a global query view, allowing multiple Prometheus instances to write data to a single VictoriaMetrics. Consequently, you can query all this data.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a717855dbdf1df859927a75557ee2573.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics also supports PromQL and the Prometheus querying API, just like Thanos.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/84daf1b7e160e6f9d8a53a83329f2436.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Unlike Thanos, the source code of VictoriaMetrics has been written from the ground up and is optimized for speed and resource consumption. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/371dee8d4705438f560245c4e5e7444f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Unlike Thanos, VictoriaMetrics scales both vertically and horizontally. There is <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/blob\/master\/README.md\">a single-node version<\/a><\/noindex>, which scales vertically. You can start with one processor and 1 GB of memory and gradually scale up to hundreds of processors and 1 TB of memory. VictoriaMetrics can utilize all these resources. Its performance can increase approximately 100 times compared to a single-core system. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/697e77546cda492f8512e7e761c0300f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The history of Thanos began in November 2017 with the first public commit. Before that, Thanos was developed internally within the company <noindex><a rel=\"nofollow\" href=\"https:\/\/improbable.io\">improbable.io<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/610a81cfb98082bb61b7ae28b00d2c78.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In June 2019, a landmark release 0.5.0 was made, in which <noindex><a rel=\"nofollow\" href=\"https:\/\/thanos.io\/proposals\/201809_gossip-removal.md\/\">the<\/a><\/noindex> <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Gossip_protocol\">gossip<\/a><\/noindex> protocol was removed. It was removed from Thanos because it did not perform well. Often, the Thanos cluster operated incorrectly, and nodes connected to it improperly due to the gossip protocol. Therefore, it was decided to remove it. I believe this was the right decision. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/d7ecf0105d46464aadc4c46147edf9dc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In the same June 2019, they submitted application number <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/cncf\/toc\/pull\/256\">256<\/a><\/noindex> downward API support (simultaneously with this in <noindex><a rel=\"nofollow\" href=\"https:\/\/www.cncf.io\/\">Cloud Native Computing Foundation<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a8685a616bbdc619009c570e28c9f15b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>And after a couple of months, Thanos was accepted into <noindex><a rel=\"nofollow\" href=\"https:\/\/www.cncf.io\/\">Cloud Native Computing Foundation<\/a><\/noindex>, which includes Prometheus, Kubernetes, and other popular projects.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/90c64c519aa283b13ce64b96da5ac945.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In January 2018, the development of VictoriaMetrics began.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/c2243e3e0a8d00715644d7c84627c5cb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In September 2018, I first publicly mentioned VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/66feecad1783c36651cdf2a19e6d0d70.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In December 2018, the single-node version was released.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/aacf5d683c7d639df7a9f05e297f08c4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>In May 2019, <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.usejournal.com\/open-sourcing-victoriametrics-f31e34485c2b\">the source code for both the single-node and cluster versions were published.<\/a><\/noindex> In June 2019, just like Thanos, we submitted an application to the CNCF foundation under the number <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6da160c13c87133b1994ef32f86db077.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>. We submitted our application one day earlier than Thanos. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/cncf\/toc\/pull\/255\">255<\/a><\/noindex>Unfortunately, we have not yet been accepted there. We need community support. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/12c281eae67b3cec5a66fab41aee1fdb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let's look at the most important slides that demonstrate the architecture of Thanos and VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/ab7b20661c3cfa01209cd742eb71a898.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>We will start with Thanos. The yellow components are the components of Prometheus. Everything else consists of Thanos components. Let's begin with the most important component. Thanos Sidecar is a component that is installed next to each Prometheus. It handles loading Prometheus data from local storage to S3 or another Object Storage.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a268b600bcd0cebfd9c5069d2d911a9d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let's start with Thanos. The yellow components are Prometheus components. Everything else consists of Thanos components. We begin with the most important component. The Thanos Sidecar is a component that is installed alongside each Prometheus. It is responsible for uploading Prometheus data from local storage to S3 or another Object Storage. <\/p>\n<p><\/p>\n<p>There is also a component called Thanos Store Gateway, which can read data from Object Storage upon incoming requests from Thanos Query. Thanos Query implements PromQL and the Prometheus API. Therefore, it appears externally as Prometheus. It accepts PromQL queries, sends them to Thanos Store Gateway, which retrieves the necessary data from Object Storage and sends it back.<\/p>\n<p><\/p>\n<p>However, we have data in Object Storage that doesn't include the last two hours due to the specifics of Thanos Sidecar's implementation, which cannot upload the last two hours to Object Storage S3 because Prometheus has not yet created files in local storage for this period.<\/p>\n<p><\/p>\n<p>How was this issue resolved? Thanos Query, in addition to sending requests to Thanos Store Gateway, concurrently sends requests to each Thanos Sidecar located near Prometheus. <\/p>\n<p><\/p>\n<p>Thanos Sidecar, in turn, proxies the requests further to Prometheus and retrieves the data for the last two hours.<\/p>\n<p><\/p>\n<p>In addition to these components, there is an optional component without which Thanos would struggle to function properly. This is Thanos Compact, which merges small files in Object Storage into larger files that have been uploaded by Thanos Sidecars. Thanos Sidecar uploads data files with a two-hour time window. If these files are not merged into larger files, their number can grow significantly. The more such files there are, the more memory is needed for Thanos Store Gateway, and the more resources are required for data transfer and metadata. The operation of Thanos Store Gateway becomes inefficient. Therefore, it is essential to run Thanos Compact, which merges small files into larger ones to reduce their quantity and minimize overhead on Thanos Store Gateway.<\/p>\n<p><\/p>\n<p>There is also a component called Thanos Ruler. It executes Prometheus alerting rules and can compute Prometheus recording rules to write data back to Object Storage. However, this component is not recommended for use, as it <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\/blob\/master\/docs\/components\/rule.md#risk\">is prone to returning incomplete data.<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>This is the simple scheme of Thanos. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/3f7cd77a1c056bf7ac245c774d05ea34.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Now let's compare it with the scheme of VictoriaMetrics.<\/p>\n<p><\/p>\n<p>VictoriaMetrics has two versions: Single-node and clustered. The Single-node runs on a single computer. It does not have these components; just one binary. This binary appears in the slide as this square. Everything inside the square is the content of the binary file for the Single-node version. You do not need to know about it. Just run the binary\u2014and everything works. <\/p>\n<p><\/p>\n<p>The cluster version is more complex. It contains three different components: vmselect, vminsert, and vmstorage. Their names indicate what each of them does. The Insert component accepts data in various formats: from the Prometheus remote write API, Influx line protocol, Graphite protocol, and from the OpenTSDB protocol. The Insert component accepts them, parses, and distributes them among the available storage components, where the data is finally saved. <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/@valyala\/promql-tutorial-for-beginners-9ab455142085\">PromQL<\/a><\/noindex>, as well as the Prometheus querying API, and it can be used as a replacement for Prometheus in Grafana or other Prometheus API clients. The Select component accepts PromQL queries, parses them, reads the necessary data to execute this query from the storage nodes, processes this data, and returns an answer.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/5b4f1ec2504c1bea423e58ced26570bd.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let's compare the complexity of installing Thanos and VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/39b293fc911b2987a191cc5d98b1c9f7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Starting with Thanos. Before you begin working with Thanos, you need to create a bucket in an Object Storage service like S3 or GCS so that the Thanos Sidecar can write data there. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/7a59c36c4453f1747067c79d9a9b8e2a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Then, for each Prometheus, you need to install the Thanos Sidecar. Before that, do not forget to disable data compaction in Prometheus. Data compaction periodically compresses the data in Prometheus's local storage to reduce resource consumption.<\/p>\n<p><\/p>\n<p>When you install Thanos Sidecar on your Prometheuses, you must disable this data compaction because Thanos Sidecar does not work properly with data compaction enabled. This means that your Prometheus starts saving data in blocks of two hours and stops merging those blocks into larger ones. Consequently, if you make queries that exceed the duration of the last two hours, they will not perform as efficiently compared to how they could if data compaction were enabled.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/825b70ede9edd1de6658576e2b824cd1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Therefore, Thanos recommends reducing the data retention time in local storage to 6-8 hours to minimize the overhead of a large number of small blocks.<\/p>\n<p><\/p>\n<p>After you have installed the Thanos Sidecar, you need to install two components for each Object Storage Bucket. These are the Thanos Compactor and Thanos Store Gateway. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/77575d65a75bfd9af8d7dedf0cc5c922.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>After that, you need to install Thanos Query and configure it to connect to all the Thanos Store Gateways that you have, as well as to connect to all the Thanos Sidecars.<\/p>\n<p><\/p>\n<p>There may be a small issue here. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6eddee956c58991ccee52891b368f10f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>You need to configure a reliable and secure connection from Thanos Query to these components. If your Prometheuses are located in different data centers or in different VPCs, external connections to them are prohibited. However, to work with Thanos Query, you need to establish a connection there, and you must find a way to do so.<\/p>\n<p><\/p>\n<p>If you have multiple data centers, the reliability of the entire system decreases accordingly. Thanos Query needs to maintain constant connections to all Thanos Sidecars located in different data centers. With each incoming request, it will send requests to all Thanos Sidecars. If the connection is interrupted, you will either receive an incomplete dataset or an error message stating, \"the cluster is not operational.\" <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/13c3765871568ac0db4014964cb5d752.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>With VictoriaMetrics, things are a bit simpler. For the Single-node version, you only need to run a single binary, and everything works. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/1edf992a6404f5dea1ae2bd4f4d2d8fd.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>For the clustered version, you simply need to run all three types of components mentioned above in any quantity you require, or use <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/helm-charts\/\">helm chart<\/a><\/noindex> to automate the deployment of components in Kubernetes. We also plan to create a Kubernetes operator. The Helm chart does not cover certain cases and can cause issues. For example, it allows you to reduce the number of storage nodes, which will lead to data loss.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/ad4249222cb5e591d7e7bd0b7f1708c0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>After you have started a single binary or the clustered version, you just need to add to the Prometheus configuration <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/blob\/master\/README.md#prometheus-setup\">the setting for the remote write URL<\/a><\/noindex>, so it starts writing data in parallel to local storage and remote storage. As you've noticed, this configuration should work much more reliably compared to Thanos configuration. We don\u2019t need to maintain connections from VictoriaMetrics to all Prometheus instances because the Prometheus servers connect to VictoriaMetrics themselves and transmit data. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/465ffc6cd77b2ca022b2541851a7ddbc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let's consider the maintenance of Thanos and VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6043f0d3d99cb3e49837f03d3e6bbb9b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos needs to monitor the Sidecar to ensure they continue uploading data to Object Storage. They may stop uploading data due to upload errors, such as a temporary network interruption with Object Storage or if Object Storage becomes temporarily unavailable. At this moment, Thanos Sidecar will notice this, report an error, may crash, and subsequently stop functioning. If you do not monitor it, data will no longer be transmitted to Object Storage. If the retention period (recommended 6-8 hours) passes, you will lose data that hasn't reached Object Storage.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/7c6556fecdee4ca4986931c2eebf9895.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos compactor may stop functioning due to <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\/issues\/1919\">races with Sidecar<\/a><\/noindex>. Compactors take data from Object Storage and merge it into larger chunks. Since the compactors are not synchronized with the Sidecars, the following situation can occur: a Sidecar has not finished writing a block yet, and the Compactor decides the block is fully written. The Compactor begins to read it. It reads the block incompletely and stops functioning. See details. <noindex><a rel=\"nofollow\" href=\"https:\/\/thanos.io\/proposals\/201901-read-write-operations-bucket.md\/\">here<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/7fd00e243009763781b54043dec1e94b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The Store Gateway may return inconsistent data due to races between the Compactor and the Sidecars. The same issue arises here because the Store Gateway is not synchronized with the Compactors and the Sidecars. Accordingly, race conditions may occur where the Store Gateway does not see part of the data or sees extra data.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/85fcc811173dda1a42ac8d4e3b212f5e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The Query component in Thanos by default returns a partial result if some Sidecars or Store Gateway are unavailable at the moment. You will receive some data and won\u2019t even know that you haven't received all the data. This is how it works by default. In a similar situation, VictoriaMetrics returns marked data as partial. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a9e5a70bdcafae9b98545514ca7a7ff3.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Unlike Thanos, VictoriaMetrics rarely loses data. Even if the connection from Prometheus to VictoriaMetrics is interrupted, it's not a problem since Prometheus continues to write incoming new data to the Write Ahead Log, which spans 2 hours. If you restore the connection to VictoriaMetrics within two hours, the data will not be lost. Prometheus <noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/blog\/2019\/03\/25\/whats-new-in-prometheus-2.8-wal-based-remote-write\/\">is capable of appending data after restoring the connection to VictoriaMetrics.<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/fa4be02056f7338416029bfd8f9d0214.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Unlike Thanos, which writes data to object storage only after two hours, Prometheus automatically replicates data via the remote write protocol to remote storage, such as VictoriaMetrics. You need not worry about losing local storage in Prometheus. If it unexpectedly loses local storage, in the worst case, you will lose only the last few seconds of data that haven't been written to remote storage.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/a6e9b2322eb9b3e145603a6b9db1002d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kubernetes automatically manages the cluster, unlike Thanos. All Thanos components are difficult to fit into a single Kubernetes cluster, unlike the clustered components of VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/5b3d1586fe461a0c6f34de09fdd12845.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Updating VictoriaMetrics to a new version is very straightforward. You just stop VictoriaMetrics, update the binaries, and restart it. When stopped via the SIGINT signal, all VictoriaMetrics binaries perform a graceful shutdown. They properly save the necessary data and correctly close incoming connections to avoid any loss. Therefore, you will not lose anything during the update. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/96a52e2f0696172159919536efb68ad3.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Expanding a cluster with VictoriaMetrics is very simple. You just add the necessary components and continue working. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/5f5e6ef78b322bb278e618a526f05e26.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>On the pitfalls of Thanos and VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/17808bc76a16e571b7e4fa1655964201.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos has the following pitfalls. Prometheus must store data for the last two hours. If they are lost, you will lose them completely, as they have not yet been written to Object Storage, such as S3. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/81ca9135d2d97b95e9731dd1f5501859.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The Store Gateway component and the compactor component may require a lot of memory to work with large Object Storage if it stores many small files. The more files there are and the larger their size, the more RAM is needed by the Store Gateway and the compactor to store metadata. Thanos has many issues regarding this. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\/issues\/448\">The Store Gateway and the compactor crash with moderate data volumes.<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/cf75315535439843c9bb32cf1175391b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos advertises that it can scale infinitely with the number of your Prometheus instances. In reality, this is not true. Since all queries go through the Query component, which must poll all Store Gateway components and all Sidecar components in parallel to extract data and then preprocess it. Obviously, the speed of queries is limited by the slowest weak link, whether it be the slowest Store Gateway or the slowest Sidecar. <\/p>\n<p><\/p>\n<p>These components can be unevenly loaded. For example, you have Prometheus collecting millions of metrics per second. And then there's a Prometheus that collects thousands of metrics per second. The Prometheus collecting millions of metrics per second puts a significantly higher load on the server it runs on. Consequently, the Sidecar operates slower there. Everything runs slowly in that environment, and the Query component will retrieve data from there very slowly. As a result, the performance of your entire cluster will be limited by this slow Sidecar.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/68070f629821a54bfd8ed934d67344fd.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>By default, Thanos returns partial data if some Sidecars or the Store Gateway are unavailable. For example, if your Sidecars are spread around the world in different data centers, the likelihood of connection interruptions and component unavailability significantly increases. Consequently, in most cases, you will receive partial data without even realizing it.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6a1dea50e739314c23b4370c569b44a4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics also has its pitfalls. The first pitfall is an option that limits the amount of memory used for the VictoriaMetrics cache. By default, it is set to 60% of the RAM on the machine where VictoriaMetrics is running, or 60% of the pod's RAM in Kubernetes.<\/p>\n<p><\/p>\n<p>If this value is incorrectly configured, it can severely impact the performance of VictoriaMetrics. For example, setting it too low may cause data to no longer fit into the VictoriaMetrics cache. This forces it to perform unnecessary work, placing additional load on the CPU and the disk. Conversely, setting this option too high increases, firstly, the chance that VictoriaMetrics will crash with an out-of-memory error, and secondly, it leaves very little RAM available for the file cache in the operating system. VictoriaMetrics relies on the file cache for performance. If it's insufficient, disk load can increase significantly. Therefore, the advice is: do not change this parameter without extreme necessity.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/f145833dcbe42c2b62545358cfb204e5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The second option is retentionPeriod \u2014 a duration which is set to 1 month by default. This is the time period during which VictoriaMetrics retains data. After this period, VictoriaMetrics deletes the data.<\/p>\n<p><\/p>\n<p>Many launch VictoriaMetrics without this parameter, recording data for a month. Later, they ask: why did the data disappear for the previous month? Because the retentionPeriod defaults to 1 month. Therefore, it is essential to know and set the correct retentionPeriod. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/377e4e19b7dc5648dd16bff39f4f1651.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let\u2019s go over the unique features.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/bca5529a95e94ada650e67dba063ada6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos has a feature called downsampling: 5-minute and hourly intervals that often <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+is%3Aopen+downsampling\">do not work correctly<\/a><\/noindex>. If you search on GitHub, you'll find many issues related to this downsampling, as it sometimes doesn't function as expected by users.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/84f116767826d1174286b45ccf60dfe7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos has data deduplication for Prometheus HA pairs. When two Prometheus instances collect the same metrics from the same targets, Thanos aggregates them in Object Storage. Thanos can correctly deduplicate this data, unlike VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/efd797cafcb339a4900f4d2cddd04638.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos includes an alert component, which was shown in the Thanos diagram. However, it is <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/thanos-io\/thanos\/blob\/master\/docs\/components\/rule.md\">not recommended for production use<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/87d12ff7b3c33dae9a8eda7a2a95bce1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thanos has the advantage that the code for Thanos and Prometheus is shared. Thanos and Prometheus are developed by the same team. When improvements are made in either Thanos or Prometheus, both benefit.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/92435f15d58e83eb4d548a8bf5256c30.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The main feature of VictoriaMetrics is MetricsQL. This is the extension of VictoriaMetrics for PromQL, which I discussed at the last big monitoring meetup.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/43a1618a397badc3df9d4eded19be1ec.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics supports data ingestion via multiple protocols. VictoriaMetrics can not only accept data from Prometheus but also via Influx, OpenTSDB, and Graphite protocols.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/b4bca4b255990f1af11a6ceaf42f46ef.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics data typically takes much less space compared to Thanos and Prometheus. <\/p>\n<p><\/p>\n<p>When recording actual data, users report a 2-5 times reduction in disk space compared to Prometheus and Thanos.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/6aeab3c42b45dbd620d328fc43ce4bec.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Another advantage of VictoriaMetrics is that it is optimized for speed. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/89010a913f0d7b1388b6c62007c7835b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Let\u2019s discuss infrastructure costs. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/65bd322bb2a64ac4188e29b79e54763a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>One advantage of Thanos is that it stores data in object storage, which is relatively inexpensive.<\/p>\n<p><\/p>\n<p>When you save data in object storage, you have to pay for read and write operations ($10 per million operations). When you write data to object storage, you incur costs for your hosting services to upload data to the Internet, unless your cluster is on AWS \u2014 in which case it is free. When you read data, you pay between $10 to $230 per 1TB. This can add up significantly if you frequently request historical data from the Thanos cluster.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/7f1b32c75d49efa082201fa55b43f6bb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>For a Thanos cluster, servers must be paid for the Compact, Store Gateway, and Query components, which require a lot of memory and CPU for large data volumes.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/19c90f30760e77f57a658e67507f4f98.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>The expenses for VictoriaMetrics are as follows. If storing data on GCE HDD disks, it totals $40 per 1TB. VictoriaMetrics only requires standard HDD disks, without any need for SSDs, which are five times more expensive. VictoriaMetrics is optimized for HDD.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/c7e922b4d97f353218d6edad7cdea584.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics requires servers for components: either Single-node or for cluster components, which, unlike Thanos components, require significantly less CPU and RAM \u2014 thus being more affordable. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/aaf63cad75238ba19efb825376cf356c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Implementation examples. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/53ce0a68724c339d88ad769c18a2e5f9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>An implementation example for Thanos is Gitlab. Gitlab operates entirely on Thanos. However, not everything is smooth. If you look at their <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/infrastructure\/issues\/8647\">issues<\/a><\/noindex>, you can see that they constantly encounter some <noindex><a rel=\"nofollow\" href=\"https:\/\/gitlab.com\/gitlab-com\/gl-infra\/infrastructure\/issues\/8413\">operational issues with Thanos<\/a><\/noindex>: they lack memory for Store Gateway or Query components. They continually have to increase memory capacity. <\/p>\n<p><\/p>\n<p>This leads to increased costs for resolving these issues. <\/p>\n<p><\/p>\n<p>The second implementation, which may be more successful, is the company Improbable, which started the development of Thanos. They published the Thanos source code. Improbable is a company that develops game engines.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/3a1279cae3cfc7e94c2eccfc0972f06e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Public examples of VictoriaMetrics implementations include:<\/p>\n<p><\/p>\n<ul>\n<li>wix.com website builder<\/li>\n<li>Adidas implements VictoriaMetrics and even presented a report at the last PromCon 2019<\/li>\n<li>TrafficStars \u2014 ad network<\/li>\n<li>Seznam.cz \u2014 a popular Czech search engine.<\/li>\n<\/ul>\n<p><\/p>\n<p>Then there are some unknown companies that I cannot name right now. They did not give consent. <\/p>\n<p><\/p>\n<ul>\n<li>One large game developer. Bigger than Improbable.<\/li>\n<li>A large graphics software developer.<\/li>\n<li>A major Russian bank.<\/li>\n<li>A European wind turbine manufacturer that successfully tested VictoriaMetrics. This manufacturer implements VictoriaMetrics for monitoring data obtained from wind turbines at a rate of 50 samples per second for each sensor. Each wind turbine has several hundred sensors. They have several hundred wind turbines.<\/li>\n<li>Russian airlines that want to implement VictoriaMetrics but are yet to do so. We are in the contract stage with them.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/040affc4d56712783db61ee3b0ec509e.jpg\" style=\"display:block;margin: 0 auto;\" \/>Conclusions.<\/p>\n<p><\/p>\n<p>VictoriaMetrics and Thanos address similar tasks but in different ways:<\/p>\n<p><\/p>\n<ul>\n<li>Global query view<\/li>\n<li>horizontal scaling <\/li>\n<li>arbitrary retention<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/af6285e8f6f68f373da1ebc552104f51.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Thank you.<\/p>\n<p><\/p>\n<p>We look forward to seeing you on our <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/VictoriaMetrics_ru1\">telegram channel<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Choosing a data store for Prometheus: Thanos vs VictoriaMetrics\" src=\"\/wp-content\/uploads\/2019\/12\/c45b2a3b6e541e7e6cd580629e418c35.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p class=\"for_users_only_msg\">Only registered users can participate in the survey. <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/auth\/login\/\">Please log in<\/a><\/noindex>, please.<\/p>\n<h2 class=\"default-block__polling-title\">What do you use as long-term storage for Prometheus?<\/h2>\n<ul class=\"poll-result\">\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent\">35,3%<\/strong>Thanos<\/p>\n<\/li>\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent\">0,0%<\/strong>Cortex<\/p>\n<\/li>\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent\">0,0%<\/strong>M3DB<\/p>\n<\/li>\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent  poll-result__data-percent_winner\">41,2%<\/strong>VictoriaMetrics<\/p>\n<\/li>\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent\">23,5%<\/strong>other<\/p>\n<\/li>\n<\/ul>\n<p>    17 users voted. 16 users abstained.<br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/482272\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442. \u041d\u0438\u0436\u0435 \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u0435\u043d\u0430 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u0430 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u0441 Big Monitoring Meetup 4. Prometheus \u2013 \u0441\u0438\u0441\u0442\u0435\u043c\u0430 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430 \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0438 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432, \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u0442\u043e\u0440\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u0435 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u044b \u043c\u043e\u0433\u0443\u0442 \u0441\u043e\u0431\u0438\u0440\u0430\u0442\u044c \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044e \u043e \u0442\u0435\u043a\u0443\u0449\u0438\u0445 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u0430\u0445 \u0441\u0438\u0441\u0442\u0435\u043c \u0438 \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0442\u044c \u043e\u043f\u043e\u0432\u0435\u0449\u0435\u043d\u0438\u044f \u0434\u043b\u044f \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u044f \u0443\u0432\u0435\u0434\u043e\u043c\u043b\u0435\u043d\u0438\u0439 \u043e\u0431 \u043e\u0442\u043a\u043b\u043e\u043d\u0435\u043d\u0438\u044f\u0445 \u0432 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441\u0438\u0441\u0442\u0435\u043c. \u0412 \u0434\u043e\u043a\u043b\u0430\u0434\u0435 \u0431\u0443\u0434\u0435\u0442 \u0441\u0440\u0430\u0432\u043d\u0435\u043d\u0438\u0435 Thanos \u0438 VictoriaMetrics \u2014 \u043f\u0440\u043e\u0435\u043a\u0442\u043e\u0432 \u0434\u043b\u044f \u0434\u043e\u043b\u0433\u043e\u0441\u0440\u043e\u0447\u043d\u043e\u0433\u043e \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043c\u0435\u0442\u0440\u0438\u043a [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-54601","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0412\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 \u0434\u043b\u044f Prometheus: Thanos vs VictoriaMetrics | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-12-29T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:02:38+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Choosing a data storage solution for Prometheus: Thanos vs VictoriaMetrics | ProHoster","description":"Hello everyone.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0412\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 \u0434\u043b\u044f Prometheus: Thanos vs VictoriaMetrics | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vybiraem-hranilishhe-dannyh-dlya-prometheus-thanos-vs-victoriametrics","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-12-29T21:00:00+00:00","article:modified_time":"2020-02-18T11:02:38+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"54601","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 12:01:24","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:02:45","updated":"2026-01-24 12:01:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/54601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=54601"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/54601\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=54601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=54601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=54601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}