One of the goals of the hosting provider is to maximize the utilization of existing hardware to deliver quality service to end users. The resources of the final servers are always limited, but the number of client services deployed, and in our case referring to VPS, can vary significantly. Read more about how to fit everything into the tree and eat a burger.
Consolidating VPS on the node in such a way that clients do not feel it at all greatly helps improve the economic indicators of any hosting provider. Undoubtedly, the node should not be bursting at the seams if it is packed full of containers, as any surge in load is immediately felt by all clients.
The number of VPS that can be hosted on one node depends on many factors, such as:
1. The hardware specifications of the node
2. The size of the VPS
3. The nature of the load on the VPS
4. Software technologies that help optimize density
In this case, we will share our experience using the Pfcache technology for Virtuozzo.
We use the 6th branch, but everything mentioned is also true for the 7th.
– a mechanism of Virtuozzo that helps deduplicate IOPS and RAM in containers by allocating identical files in containers to a separate common area.
It essentially consists of:
1. Kernel code
2. User-space daemon
3. User-space utility
On the node side, we allocate a whole partition where the files that will be directly used by all VPS on the node will be created. A block device ploop is mounted to this partition. Then, upon starting a container, it receives a reference to this partition:
[root@pcs13 ~]# cat /proc/mounts
...
/dev/ploop62124p1 /vz/pfcache ext4 rw,relatime,barrier=1,data=ordered,balloon_ino=12 0 0
...
/dev/ploop22927p1 /vz/root/418 ext4 rw,relatime,barrier=1,data=ordered,balloon_ino=12,pfcache_csum,pfcache=/vz/pfcache 0 0
/dev/ploop29642p1 /vz/root/264 ext4 rw,relatime,barrier=1,data=ordered,balloon_ino=12,pfcache_csum,pfcache=/vz/pfcache 0 0
...
Here is an approximate statistic of the number of files on one of our nodes:
[root@pcs13 ~]# find /vz/pfcache -type f | wc -l
45851
[root@pcs13 ~]# du -sck -h /vz/pfcache
2.4G /vz/pfcache
2.4G total
The principle of operation of pfcache is as follows:
• The user-space daemon Pfcached writes the sha-1 hash of the file into the xattr attribute of that file. Not all files are processed, only those in the directories /usr, /bin, /usr/sbin, /sbin, /lib, /lib64
• It is most likely that files in these directories will be 'common' and will be used by multiple containers;
• Pfcached periodically collects file read statistics from the kernel, analyzes them, and adds files to the cache if they are frequently accessed;
• The directories' data can be different and is configured in the configuration files.
• When reading a file, it checks whether it contains the specified hash in the extended attributes xattr. If it does, it opens a 'common' file instead of the container file. This substitution occurs unnoticed by the container code and is hidden in the kernel;
• When writing to a file, the hash is invalidated. Thus, when reopening, the actual container file will be accessed directly, not its cache.
By holding common files from /vz/pfcache in the page cache, we achieve savings in the cache itself as well as IOPS savings. Instead of reading ten files from disk, we read one file that goes directly into the page cache.
struct inode {
...
struct file *i_peer_file;
...
};
struct address_space {
...
struct list_head i_peer_list;
...
}
The VMA list for the file remains unified (deduplicates memory) and we read from disk less frequently (saving IOPS). Our 'common' storage is on SSD – providing an additional speed gain.
Example for caching the file /bin/bash:
[root@pcs13 ~]# ls -li /vz/root/2388/bin/bash
524650 -rwxr-xr-x 1 root root 1021112 Oct 7 2018 /vz/root/2388/bin/bash
[root@pcs13 ~]# pfcache dump /vz/root/2388 | grep 524650
8e3aa19fdc42e87659746f6dc8ea3af74ab30362 i:524650 g:1357611108 f:CP
[root@pcs13 ~]# sha1sum /vz/root/2388/bin/bash
8e3aa19fdc42e87659746f6dc8ea3af74ab30362 /vz/root/2388/bin/bash
[root@pcs13 ~]# getfattr -ntrusted.pfcache /vz/root/2388/bin/bash
# file: vz/root/2388/bin/bash
trusted.pfcache="8e3aa19fdc42e87659746f6dc8ea3af74ab30362"
[root@pcs13 ~]# sha1sum /vz/pfcache/8e/3aa19fdc42e87659746f6dc8ea3af74ab30362
8e3aa19fdc42e87659746f6dc8ea3af74ab30362 /vz/pfcache/8e/3aa19fdc42e87659746f6dc8ea3af74ab30362
The efficiency of usage is calculated .
This script goes through all containers on the node, calculating the cached files for each container.
[root@pcs16 ~]# /pcs/distr/pfcache-examine.pl
...
Pfcache cache uses 831 MB of memory
Total use of pfcached files in containers is 39837 MB of memory
Pfcache effectiveness: 39006 MB
Thus, we save about 40 gigabytes of files in containers in terms of memory, and they will be loaded from the cache.
To make this mechanism work even better, it is necessary to deploy 'similar' VPS on the node. For example, those for which the user does not have root access and where an environment is set up from a deployed image.
You can tune the operation of pfcache through the configuration file
/etc/vz/pfcache.conf
MINSIZE, MAXSIZE – minimum/maximum file size for caching
TIMEOUT – timeout between caching attempts
You can review the full list of parameters. .
Source: habr.com
