{"id":52113,"date":"2019-11-01T00:00:00","date_gmt":"2019-10-31T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik"},"modified":"2020-02-18T13:59:47","modified_gmt":"2020-02-18T10:59:47","slug":"bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/bioyino-raspredelyonnyj-masshtabiruemyj-agregator-metrik","title":{"rendered":"Bioyino \u2014 a distributed, scalable metrics aggregator","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

So, you are collecting metrics. Just like us. We are also collecting metrics. Of course, the ones needed for business. Today we will talk about the very first link in our monitoring system \u2014 a statsd-compatible aggregation server. bioyino<\/a><\/noindex>, why we wrote it and why we moved away from brubeck. <\/p>\n

<\/p>\n

\"Bioyino<\/p>\n

<\/a><\/noindex><\/p>\n

From our previous articles (1<\/a><\/noindex>, 2<\/a><\/noindex>) you can learn that for some time we collected tags using brubeck.<\/a><\/noindex>It's written in C. From a coding perspective \u2014 it's as simple as a cork (this is important when you want to contribute) and, most importantly, it handles our volumes of 2 million metrics per second (MPS) at peak without any significant issues. The documentation claims support for 4 million MPS with asterisks. This means you will achieve the stated figure if you configure the network correctly on Linux. (We do not know how many MPS can be obtained if the network is left as is). Despite these advantages, we had several serious complaints about brubeck.<\/p>\n

<\/p>\n

Complaint 1.<\/em> Github \u2014 the project's developer \u2014 stopped supporting it: publishing patches and fixes, accepting our and (not just our) PRs. Activity resumed in the last few months (around February-March 2018), but there was almost 2 years of complete silence before that. Additionally, the project is developed for Github's internal needs<\/a><\/noindex>, which can become a serious obstacle to implementing new features.<\/p>\n

<\/p>\n

Complaint 2.<\/em> Accuracy of calculations. Brubeck aggregates only 65536 values. In our case, for some metrics during the aggregation period (30 seconds), there can be much more values (1,527,392 at peak). As a result of such sampling, the maximum and minimum values appear useless. For example, like this: <\/p>\n

<\/p>\n

\"Bioyino
\nHow it was<\/em><\/p>\n

<\/p>\n

\"Bioyino
\nHow it should have been<\/em><\/p>\n

<\/p>\n

For the same reason, sums are calculated incorrectly. Add to this the bug with the overflow of a 32-bit float, which sends the server into a segfault upon receiving what seems to be an innocuous metric, and it gets even better. The bug, by the way, still hasn\u2019t been fixed.<\/p>\n

<\/p>\n

And finally, Complaint X<\/em>At the time of this writing, we are ready to present it to all 14 more or less functional implementations of statsd that we could find. Let's imagine that some particular infrastructure has grown to the point where handling 4 million MPS is no longer sufficient. Or perhaps it hasn't grown yet, but metrics are already important enough for you that even short, 2-3 minute outages on the graphs can become critical and trigger bouts of unbearable depression among managers. Since treating depression is a thankless task, technical solutions are necessary.<\/p>\n

<\/p>\n

Firstly, fault tolerance, so that a sudden issue on the server doesn't cause a psychiatric zombie apocalypse in the office. Secondly, scalability, to be able to handle more than 4 million MPS while not delving into the depths of the Linux networking stack and comfortably growing 'horizontally' to the required sizes.<\/p>\n

<\/p>\n

Since we had headroom in scaling, we decided to start with fault tolerance. 'Oh! Fault tolerance! This is simple, we can do this,' we thought and launched 2 servers, deploying a copy of brubeck on each. To accomplish this, we had to duplicate the traffic with metrics to both servers and even write a little utility for that. utility<\/a><\/noindex>. We solved the fault tolerance issue with this approach, but\u2026 not very well. At first, everything seemed to be working fine: each brubeck collects its own aggregation variant, writes data to Graphite every 30 seconds, overwriting the old interval (this is done on the Graphite side). If one server fails, we always have the second with its own copy of aggregated data. However, the problem is that if a server fails, a 'sawtooth' pattern appears on the graphs. This is related to the fact that the 30-second intervals on brubeck are not synchronized, and at the moment of failure, one of them does not get overwritten. The same happens when the second server starts. It\u2019s quite bearable, but we want better! The scalability issue hasn\u2019t gone away either. All metrics are still 'flying' to a single server, so we are still constrained by the same 2-4 million MPS depending on the network upgrade.<\/p>\n

<\/p>\n

If you think a little about the problem and simultaneously dig through the snow with a shovel, an obvious idea might come to mind: we need a statsd that can operate in distributed mode. That is, one where synchronization between nodes is implemented based on time and metrics. \"Of course, such a solution must already exist,\" we said and went Googling... and found nothing. After sifting through documentation on various statsd (https:\/\/github.com\/etsy\/statsd\/wiki#server-implementations<\/a><\/noindex> as of 11.12.2017), we found absolutely nothing. Apparently, neither developers nor users of these solutions have faced such an abundance of metrics yet, otherwise they would have certainly come up with something.<\/p>\n

<\/p>\n

And then we remembered the \"toy\" statsd \u2014 bioyino, which we wrote at a hackathon just for fun (the project name was generated by a script before the hackathon started) and realized that we urgently needed our own statsd. Why?<\/p>\n

<\/p>\n