{"id":95783,"date":"2020-10-03T13:42:02","date_gmt":"2020-10-03T11:42:02","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/nastrojka-yadra-linux-dlya-glusterfs"},"modified":"2020-10-03T13:42:02","modified_gmt":"2020-10-03T11:42:02","slug":"nastrojka-yadra-linux-dlya-glusterfs","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/nastrojka-yadra-linux-dlya-glusterfs","title":{"rendered":"Configuring the Linux kernel for GlusterFS","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

This article's translation was prepared in anticipation of the course launch \u00abLinux Administrator. Professional\u00bb<\/a><\/noindex>.<\/i><\/b><\/p>\n

\"Configuring<\/p>\n

From time to time, questions arise regarding Gluster's recommendations for kernel configuration and whether it is necessary.<\/p>\n

Such necessity rarely arises. In most workloads, the kernel performs very well. However, there is a downside. Historically, the Linux kernel tends to consume a lot of memory if given the opportunity, including for caching as a primary means of boosting performance.<\/a><\/noindex><\/p>\n

In most cases, this works excellently, but under heavy load, it can lead to issues. <\/p>\n

We have extensive experience with memory-intensive systems, such as CAD, EDA, and similar applications, which began to slow down under high loads. Occasionally, we encountered issues with Gluster. After careful observation of memory usage and disk wait times over several days, we experienced overloads, substantial iowait, kernel oops errors, and freezes, among others. <\/p>\n

This article is the result of many experiments with configuration parameters performed in various situations. Thanks to these parameters, not only has responsiveness improved overall, but cluster stability has significantly increased.<\/p>\n

When it comes to memory configuration, the first area to examine is the virtual memory (VM) subsystem, which has a multitude of options that can be quite confusing.<\/p>\n

vm.swappiness<\/h4>\n

\nParameter vm.swappiness<\/code> determines how much the kernel uses swapping in comparison to RAM. In the source code, it is also defined as \"tendency to steal mapped memory.\" A high swappiness value means that the kernel will be more inclined to unload mapped pages. A low swappiness value means the opposite: the kernel will unload pages from memory less frequently. In other words, the higher the value vm.swappiness<\/code>, the more the system will utilize swap.<\/p>\n

Excessive swapping is undesirable, as large blocks of data are loaded into and out of RAM. Many argue that the swappiness value should be high, but in my experience, setting it to \"0\" leads to better performance.<\/p>\n

You can read more about it here \u2014 lwn.net\/Articles\/100978<\/a><\/noindex> <\/p>\n

However, these settings should be applied cautiously and only after testing the specific application. For high-load streaming applications, this parameter should be set to '0'. Changing it to '0' improves the system's responsiveness.<\/p>\n

vm.vfs_cache_pressure<\/h4>\n

\nThis parameter controls the memory consumed by the kernel for caching directory objects and index descriptors (dentry and inode).<\/p>\n

With a default value of 100, the kernel will attempt to free dentry and inode caches 'fairly' relative to the pagecache and swapcache. Reducing vfs_cache_pressure causes the kernel to retain dentry and inode caches. When the value is '0', the kernel will never clear the dentry and inode cache due to memory pressure, which can easily lead to an out-of-memory error. Increasing vfs_cache_pressure above 100 prioritizes the kernel to unload dentry and inode.<\/p>\n

When using GlusterFS, multiple users with large data volumes and many small files can easily consume significant amounts of RAM on the server due to inode\/dentry caching, which may lead to performance degradation as the kernel processes data structures in a system with 40 GB of memory. Setting this parameter above 100 has helped many users achieve fairer caching and improve the kernel's responsiveness.<\/p>\n

vm.dirty_background_ratio and vm.dirty_ratio<\/h4>\n

\nThe first parameter (vm.dirty_background_ratio<\/code>) defines the percentage of memory with dirty pages that should be reached before starting a background flush of dirty pages to disk. Until this percentage is reached, pages are not flushed to the disk. And when the flush begins, it is done in the background without interrupting running processes. <\/p>\n

The second parameter (vm.dirty_ratio<\/code>) determines the percentage of memory that can be occupied by dirty pages before a forced flush begins. When this threshold is reached, all processes become synchronous (blocked), and they are not allowed to continue until the I\/O operation they requested is actually completed and the data is on disk. Under high I\/O loads, this causes issues since data caching is absent, and all processes performing I\/O are blocked waiting for I\/O. This leads to a large number of stuck processes, high load, unstable system operation, and poor performance.<\/p>\n

Reducing these parameter values causes data to be flushed to disk more frequently, instead of being held in RAM. This can help systems with a large amount of memory, where it is normal to flush a page cache of 45-90 GB to disk, resulting in huge wait times for frontend applications, decreasing overall responsiveness and interactivity.<\/p>\n

\"1\" > \/proc\/sys\/vm\/pagecache<\/h4>\n

\nThe page cache is a cache that stores the data of files and executable programs, meaning these are pages containing the actual content of files or block devices. This cache is used to reduce the number of reads from disk. A value of \"1\" means that 1% of RAM is used for caching, resulting in more read operations from disk than from RAM. It is not necessary to modify this parameter, but if you are paranoid about controlling page cache, you can utilize it.<\/p>\n

\"deadline\" > \/sys\/block\/sdc\/queue\/scheduler<\/h4>\n

\nThe I\/O scheduler is a component of the Linux kernel that handles read and write queues. Theoretically, for an intelligent RAID controller, it is better to use \"noop\" because Linux knows nothing about the physical geometry of the disk, so it is more efficient to let the controller, which knows the disk's geometry well, process requests as quickly as possible. However, it seems that \"deadline\" improves performance. More about schedulers can be read in the Linux kernel source documentation: linux\/Documentation\/block\/*osched.txt<\/code>. And I also observed an increase in read throughput during mixed operations (many write operations).<\/p>\n

\"256\" > \/sys\/block\/sdc\/queue\/nr_requests<\/h4>\n

\nThe number of I\/O requests in the buffer before they are sent to the scheduler. The internal queue size of some controllers (queue_depth) is larger than the nr_requests of the I\/O scheduler, making it difficult for the I\/O scheduler to properly prioritize and merge requests. For deadline and CFQ schedulers, it is better when nr_requests is twice the internal queue size of the controller. Merging and reordering requests helps the scheduler to be more responsive under high load.<\/p>\n

echo \"16\" > \/proc\/sys\/vm\/page-cluster<\/h4>\n

\nThe page-cluster parameter controls the number of pages that are swapped at one time. In the example above, the value is set to \"16\" according to the stripe size of the RAID at 64 KB. This setting does not make sense with swappiness = 0, but if you set swappiness to 10 or 20, using this value will help when your RAID stripe size is 64 KB.<\/p>\n

blockdev --setra 4096 \/dev\/<<\/code>devname><\/code> (-sdb, hdc, or dev_mapper)<\/h4>\n

\nThe default block device settings for many RAID controllers often lead to terrible performance. Adding the above option configures read-ahead for 4096 * 512-byte sectors. At least for streaming operations, the speed is increased by filling the disk's built-in cache through read-ahead during the period used by the kernel to prepare I\/O. Data that will be requested in the next read can be cached. Excessive read-ahead can severely impact random I\/O for large files if it occupies potentially useful disk time or loads data beyond the cache.<\/p>\n

Here are a few more recommendations at the file system level. However, they have not yet been tested. Ensure that your file system is aware of the stripe size and the number of disks in the array. For example, this could be a RAID5 array with a stripe size of 64K across six disks (actually five, as one disk is used for parity). These recommendations are based on theoretical assumptions and compiled from various blogs\/articles by RAID experts.<\/p>\n

-> ext4 fs, 5 disks, 64K stripe, units in 4K blocks\nmkfs -text4 -E stride=$((64\/4))\n-> xfs, 5 disks, 64K stripe, units in 512-byte sectors\nmkfs -txfs -d sunit=$((64*2)) -d swidth=$((5*64*2))<\/code><\/pre>\n
\nFor large files, consider increasing the stripe sizes mentioned above.<\/p>\n
WARNING!<\/b> Everything described above is highly subjective for certain types of applications. This article does not guarantee any improvements without prior testing of the relevant applications by the user. It should only be applied if there is a need to improve overall system responsiveness or if it addresses current issues.<\/p>\n
Additional materials:<\/h4>\n
<\/p>\n