In this article, I will explain how to use Pinba alongside ClickHouse and Grafana instead of pinba_engine and pinboard.
On a PHP project, Pinba is probably the only reliable way to understand what is happening with performance. However, it is usually implemented only when problems are observed and it's unclear where to dig.
Often, no one has any idea how many times per second/minute a particular script is called, and optimization starts 'by touch', beginning with the areas that seem more logical.
Some analyze nginx logs, while others look at slow database queries.
Of course, Pinba wouldn't be unnecessary, but there are several reasons why it is far from being present on every project.
And the first reason is installation.
To get some meaningful output from implementing Pinba, it's very desirable to see metrics not only for the last few minutes but also over a longer period (from days to months).
For this, you need to:
- install the extension for PHP (and you might want a module for nginx)
- compile the extension for MySQL
- install Pinboard and set up cron
Due to the limited information about Pinba, many have the impression that it only worked on PHP5 and has long been consigned to the past, but as we will see further, this is not the case.
The first step is the simplest; you just need to execute the command:
apt install php-pinbaThis extension is available in repositories up to PHP 7.3 inclusive, and you don't need to compile anything.
After executing the installation command, we instantly get a working extension that collects and sends metrics for each script (execution duration, memory, etc.) in the format via UDP to 127.0.0.1:30002.
These UDP packets are currently not being captured or processed by anyone, but this does not negatively affect the speed or stability of your PHP scripts.
Until recently, the only application that could capture and process these UDP packets was . The description is 'The installation process makes you never want to read or dive into it again. In lengthy dependency lists are both package names and program names, along with links to separate pages for installation, which in turn have their own links to other dependencies. No one has the time or desire to deal with this nonsense.
Installation Process did not become .
Perhaps one day, pinba10 can be installed with just one or two commands without having to read a ton of material to understand how to do it, but for now, that is not the case.
If you have installed pinba_engine, that is only half the job. Without , you will be limited to data for only the last few minutes or will have to aggregate, save, and visualize the data yourself. Fortunately, pinboard is quite simple to .
It might seem frustrating, because all metrics from PHP are already sent to the UDP port in protobuf format, and all that is needed is to write an application to catch them and store them somewhere. Apparently, those developers who had this thought immediately sat down to write their own solutions, some of which ended up on GitHub.
Next is a review of four open-source projects that save metrics in storages from which this data can be easily retrieved and visualized, for example, using Grafana.
(November 2017)
a UDP server in Go that saves metrics in OpenTSDB. If OpenTSDB is already being used in your project, this solution might work for you; otherwise, I recommend passing it by.
(June 2018)
a UDP server in Go from the same , which this time saves metrics in InfluxDB. Many projects are already using InfluxDB for monitoring, so this solution could work well for them.
Pros:
- InfluxDB aggregate the received metrics while deleting the original after a specified time.
Cons:
- this solution does not save information based on timers.
- InfluxDB will save page addresses as tags, and if you have many unique page addresses, it will lead to At a certain point, it will "«. ()
(January 2019)
UDP server in Go that saves metrics to ClickHouse. This solution is from my friend. After getting acquainted with it, I decided it was time to tackle Pinba and ClickHouse.
Pros:
- ClickHouse is perfect for such tasks; it allows compressing data to the extent that you can store all raw data even without aggregations.
- If needed, you can easily aggregate the received metrics.
- Ready template for Grafana.
- Saves information by timers.
Cons:
- There is no configuration in which you can set the database name and tables, server address, and port.
- When saving raw data, a helper dictionary table is used to store page addresses and domains, which complicates queries later on.
- Other minor issues stemming from the first drawback.
(April 2019)
UDP server in PHP that saves metrics to ClickHouse. This is my solution, resulting from my acquaintance with Pinba, ClickHouse, and Protobuf. While figuring out this entire system, I wrote a 'proof of concept' that unexpectedly consumed minimal resources (30 MB of RAM and less than 1% of one of the eight CPU cores), so I decided to share it with the public.
Pros — the same as the previous solution; I also used familiar names from the original pinba_engine. I added a configuration that allows launching multiple instances of the Pinba server to save metrics in different tables — this is useful if you want to collect data not only from PHP but also from Nginx.
Cons — 'fatal flaw' and those minor issues that may not suit you personally, but my solution is 'as simple as a slipper' and consists of only about 100 lines of code, so any PHP developer can change what they don't like in a couple of minutes.
Operating principle
Listening on UDP port 30002. All incoming packets are decoded according to the Protobuf schema and aggregated. Once a minute, a batch is inserted into ClickHouse in the pinba.requests table. (All parameters are configured in the )
A bit about ClickHouse
ClickHouse supports various data storage engines. The most frequently used is MergeTree.
If at any point you decide to store aggregated data over time while keeping only raw data from the last period, you can create a materialized view with grouping and periodically clean the main table pinba.requests, while all data will remain in the materialized view. Moreover, when creating the pinba.requests table, you can specify "engine = Null," so the raw data will not be saved to disk at all, yet it will still be included in the materialized view and stored aggregated. I use this scheme for nginx metrics because I have 50 times more requests on nginx than on php.
So, you have come a long way, and I wouldn’t want to leave you halfway, so what follows is a detailed description of the installation and configuration of my solution and everything you will need, as well as the pitfalls that have sunk more than one ship. The entire installation process is described for Ubuntu 18.04 LTS and Centos 7; the process may vary slightly on other distributions and versions.
Installation
I have compiled all the necessary commands in to facilitate the reproducibility of the instructions. Below, only the pitfalls will be described.
php-pinba
After installation, ensure that all options are uncommented in the file /etc/php/7.2/fpm/conf.d/20-pinba.ini. In some distributions (for example, Centos), they may be commented out.
extension=pinba.so
pinba.enabled=1
pinba.server=127.0.0.1:30002clickhouse
During the installation, ClickHouse will prompt you to set a password for the default user. By default, this user is accessible from all IPs, so if you do not have a firewall on the server, be sure to set a password for it. This can also be done after installation in the file /etc/clickhouse-server/users.xml.
It's also worth noting that ClickHouse uses several ports, including 9000. This port is also used for PHP-FPM in some distributions (for example, Centos). If this port is already in use, you can change it to another one in the file /etc/clickhouse-server/config.xml.
grafana with clickhouse plugin
After installing Grafana, use the login admin and the password admin. Upon the first login, Grafana will prompt you to set a new password.
Next, go to the menu "+" -> import and specify the dashboard number for import . This dashboard I prepared and uploaded so you wouldn't have to create it yourself again.
Grafana supports working with ClickHouse via a third-party plugin, but alerts for third-party plugins do not work in Grafana (this ticket has been open for several years).
pinba-server
Installing protobuf and libevent is not mandatory, but it improves the performance of pinba-server. If you install pinba-server in a folder other than /opt, you will also need to tweak the file.
pinba module for nginx
To compile the module, you need the source code of the same version of nginx that is already installed on your server, as well as the same compilation options, otherwise the build will be successful but connecting the module will yield an error stating that "the module is not binary compatible." You can view the compilation options using the command nginx -V.
Life Hacks
All my websites operate exclusively over https. The schema field becomes meaningless, so I use it to separate web/console.
In scripts that are accessible from the web, I use:
if (ini_get('pinba.enabled')) {
pinba_schema_set('web');
}
And in console scripts (e.g., cron scripts):
if (ini_get('pinba.enabled')) {
pinba_schema_set('console');
}In my Grafana dashboard, there is a toggle for web/console to view statistics separately.
Tags can also be passed in pinba, for example:
pinba_tag_set('country', $countryCode);That's all.
Please respond to the surveys below the article.
I traditionally warn that I do not consult or help through personal messages on Habra or social networks.
Open a ticket on GitHub.
Also, please support with likes of this article .
Only registered users can participate in the survey. , please.
Which OS are you using on the server?
Ubuntu
CentOS
Debian
Gentoo
Red Hat
Alpine
OpenSUSE
SuSE
Unix
Windows
other
114 users voted. 11 users abstained.
Which version of PHP are you using on the server?
7.3
7.2
7.1
7.0
5
other
105 users voted. 17 users abstained.
Have you ever used pinba?
yes
No, but I would like to.
No, and I wouldn't want to.
No, and I haven't heard of it.
100 users voted. 14 users abstained.
Which version of the pinba server would you like to try?
pinba_engine (mysql engine)
pinba2 (mysql engine)
pinboard (php + mysql)
olegfedoseev/pinba-server (go + OpenTSDB)
olegfedoseev/pinba-influxdb (go + influxdb)
pinba-server/pinba-server (go + clickhouse)
pinba-server/pinba-server (php + clickhouse)
I will write my own.
other
39 users voted. 47 users abstained.
Source: habr.com
