{"id":55734,"date":"2020-01-27T00:00:00","date_gmt":"2020-01-26T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/highload-andrej-gushhin-zabbix-vysokaya-proizvoditelnost-i-nativnoe-partitsionirovanie"},"modified":"2020-02-18T14:03:52","modified_gmt":"2020-02-18T11:03:52","slug":"highload-andrej-gushhin-zabbix-vysokaya-proizvoditelnost-i-nativnoe-partitsionirovanie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/highload-andrej-gushhin-zabbix-vysokaya-proizvoditelnost-i-nativnoe-partitsionirovanie","title":{"rendered":"HighLoad++, Andrey Gushchin (Zabbix): High performance and native partitioning","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

We will discuss the operation of Zabbix with the TimescaleDB database as a backend. We will show how to set it up from scratch and how to migrate from PostgreSQL. We will also provide comparative performance tests of the two configurations.<\/p>\n

\"HighLoad++,
\n
\nHighLoad++ Siberia 2019. Hall \"Tomsk\". June 24, 16:00. Theses and presentation<\/a><\/noindex>. The next HighLoad++ conference will take place on April 6 and 7, 2020, in St. Petersburg. Details and tickets at the link<\/a><\/noindex>.<\/p>\n

Andrey Gushchin (hereafter \u2013 AG):<\/b> \u2013 I am a technical support engineer for ZABBIX (hereafter \u2013 \"Zabbix\"), a trainer. I have been working in technical support for over 6 years and have directly encountered performance issues. Today, I will talk about the performance that TimescaleDB can deliver compared to regular PostgreSQL 10. I will also provide some introductory information on how this works in general.<\/a><\/noindex><\/p>\n

The main performance challenges: from data collection to data cleanup<\/h3>\n

\nLet\u2019s start with the fact that there are certain performance challenges that every monitoring system faces. The first performance challenge is the rapid collection and processing of data.<\/p>\n

\"HighLoad++,
\n
\nA good monitoring system should promptly and timely receive all data, process it according to trigger expressions, which means processing based on certain criteria (this varies in different systems), and store it in a database for future use.<\/p>\n

\"HighLoad++,
\n
\nThe second performance challenge is historical data storage. Storing metrics in the database and having quick and easy access to these metrics collected over a period of time. Most importantly, the data should be conveniently accessible for use in reports, graphs, triggers, threshold values, alerts, etc.<\/p>\n

\"HighLoad++,
\n
\nThe third performance challenge is data cleanup, that is, when you reach a point where you no longer need to store certain detailed metrics that have been collected over five years (or even months or two months). Some network nodes have been removed, or certain hosts are no longer needed because they are outdated and have stopped being collected. All this needs to be cleaned up to prevent your database from growing too large. In fact, data cleanup is often a serious test for storage and heavily impacts performance.<\/p>\n

How to solve caching problems?<\/h3>\n

\nI am now going to specifically talk about 'Zabbix'. In 'Zabbix', the first and second calls are resolved through caching.<\/p>\n

\"HighLoad++,
\n
\nData collection and processing \u2013 we use RAM to store all this data. More details about this data will be provided now.<\/p>\n

There is also certain caching on the database side for main queries \u2013 for graphs and other items.<\/p>\n

Caching on the Zabbix server side: we have ConfigurationCache, ValueCache, HistoryCache, TrendsCache. What is this?<\/p>\n

\"HighLoad++,
\n
\nConfigurationCache is the main cache where we store metrics, hosts, data items, triggers; everything needed for preprocessing, data collection, what hosts to collect from, and how often. All of this is stored in ConfigurationCache to avoid unnecessary database queries. After the server starts, we update this cache and periodically refresh it (depending on configuration settings).<\/p>\n

\"HighLoad++,
\n<\/p>\n

Caching in Zabbix. Data collection<\/h3>\n

\nThe scheme here is quite large:<\/p>\n

\"HighLoad++,
\n
\nThe main elements in the scheme are these collectors:<\/p>\n

\"HighLoad++,
\n
\nThese are the assembly processes themselves, various 'pollers' that are responsible for different types of collections. They gather data via ICMP, IPMI, various protocols, and transmit it all for preprocessing.<\/p>\n

PreProcessing HistoryCache<\/h3>\n

\nAlso, if we have calculated data items (those familiar with 'Zabbix' will know), meaning derived, aggregated data items, we pull them directly from ValueCache. I will explain how it gets filled later. All these collectors use ConfigurationCache to obtain their tasks and then pass them on for preprocessing.<\/p>\n

\"HighLoad++,
\n
\nPreprocessing also uses ConfigurationCache to get preprocessing steps and processes this data in various ways. Starting from version 4.2, it has been offloaded to proxies. This is very convenient because preprocessing is quite a heavy operation. If you have a very large 'Zabbix', with a significant number of data items and a high collection frequency, this greatly eases the workload.<\/p>\n

Accordingly, after we have processed this data in some way using preprocessing, we save it in HistoryCache for further processing. This concludes the data collection phase. We move on to the main process.<\/p>\n

History syncer operation<\/h3>\n

\n\"HighLoad++,
\n
\nThe main process in Zabbix (since it is a monolithic architecture) is the History syncer. This is the primary process responsible for atomic processing of each data element, meaning each value:<\/p>\n