Setting up Out-Of-Memory Killer in Linux for PostgreSQL

When in Linux The database server terminates unexpectedly, and the cause needs to be determined. There could be several reasons. For example, SIGSEGV — failure due to a bug in the backend server. But this is rare. Most often, you simply run out of disk space or memory. If you run out of disk space, there is only one way out - to free up space and restart the database.

Out Of Memory Killer

When Server or the process runs out of memory, Linux The Out-Of-Memory Killer offers two solutions: crash the entire system or terminate the process (application) that's eating up memory. It's better, of course, to terminate the process and save the OS from crashing. In short, the Out-Of-Memory Killer is a process that terminates an application to save the kernel from crashing. It sacrifices the application to keep the OS running. Let's first discuss how OOM works and how to control it, and then look at how the OOM Killer decides which applications to terminate.

One of the main tasks Linux — Allocate memory to processes when they request it. Typically, a process or application requests memory from the OS but doesn't use it completely. If the OS allocates memory to everyone who requests it but doesn't plan to use it, the system will quickly run out of memory, and the system will crash. To prevent this, the OS reserves memory for a process but doesn't actually allocate it. Memory is allocated only when a process actually intends to use it. Sometimes the OS doesn't have free memory, but it allocates memory to a process, and when the process needs it, the OS allocates it if it can. The downside is that sometimes the OS reserves memory, but when needed, there's no free memory, causing a system crash. OOM plays a key role in this scenario, terminating processes to prevent the kernel from panicking. When the PostgreSQL process is forcibly terminated, the following message appears in the log:

Out of Memory: Killed process 12345 (postgres).

If there is not enough memory in the system and it is impossible to free it, the function is called out_of_memory. At this stage, she has only one thing left - to complete one or more processes. Should OOM-killer kill the process right away or can you wait? Obviously, when out_of_memory is called, it is either waiting for an I/O operation or paging to disk. Therefore, the OOM-killer must first perform checks and, based on them, decide that the process needs to be terminated. If all of the checks below are positive, OOM will end the process.

Process selection

When memory runs out, a function is called out_of_memory(). It has a function select_bad_process(), which is evaluated by the function badness(). The most “bad” process will fall under the distribution. Function badness() selects a process according to certain rules.

1. The kernel needs some minimum memory for itself.
2. You need to free up a lot of memory.
3. No need to terminate processes that use little memory.
4. It is necessary to complete a minimum of processes.
5. Sophisticated algorithms that increase the chances of completion for those processes that the user himself wants to complete.

After doing all these checks, OOM learns the estimate (oom_score). OOM appoints oom_score per process, and then multiplies this value by the amount of memory. Processes with higher values ​​are more likely to fall victim to the OOM Killer. Processes associated with a root user have a lower score and are less likely to be forced to terminate.

postgres=# SELECT pg_backend_pid();
pg_backend_pid 
----------------
    3813
(1 row)

The Postgres process ID is 3813, so another shell can get the estimate using this kernel option oom_score:

vagrant@vagrant:~$ sudo cat /proc/3813/oom_score
2

If you don't want OOM-Killer to end the process at all, there is another kernel option: oom_score_adj. Add a large negative value to reduce the chances of terminating a process you love.

sudo echo -100 > /proc/3813/oom_score_adj

To set a value oom_score_adj, set OOMScoreAdjust in the service block:

[Service]
OOMScoreAdjust=-1000

Or use oomprotect in a team rcctl.

rcctl set <i>servicename</i> oomprotect -1000

Force termination of a process

When one or more processes are already selected, OOM-Killer calls the function oom_kill_task(). This function sends a termination signal to the process. In case of low memory oom_kill() calls this function to send a SIGKILL signal to the process. A message is written to the kernel log.

Out of Memory: Killed process [pid] [name].

How to control OOM-Killer

В Linux You can enable or disable OOM-Killer (although the latter is not recommended). To enable or disable it, use the parameter vm.oom-kill. To enable OOM-Killer at runtime, run the command sysctl.

sudo -s sysctl -w vm.oom-kill = 1

To disable OOM-Killer, specify the value 0 in the same command:

sudo -s sysctl -w vm.oom-kill = 0

The result of this command will not be saved forever, but only until the first reboot. If you need more persistence, add this line to the file /etc/sysctl.conf:

echo vm.oom-kill = 1 >>/etc/sysctl.conf

Another way to enable and disable is to write a variable panic_on_oom. The value can always be checked in /proc.

$ cat /proc/sys/vm/panic_on_oom
0

If you set the value to 0, then when the memory runs out, there will be no kernel panic.

$ echo 0 > /proc/sys/vm/panic_on_oom

If you set the value to 1, then when the memory runs out, a kernel panic will occur.

echo 1 > /proc/sys/vm/panic_on_oom

OOM-Killer can be turned on and off, as we've already mentioned. Linux may reserve more memory for processes than is available, but not actually allocate it, and this behavior is controlled by a kernel parameter LinuxThe variable is responsible for this. vm.overcommit_memory.

You can specify the following values ​​for it:

0: The kernel itself decides whether to reserve too much memory. This is the default value in most versions. Linux.
1: the kernel will always reserve extra memory. This is risky, because memory can run out, because, most likely, one day the processes will demand what they have to.
2: the kernel will not reserve more memory than specified in the parameter overcommit_ratio.

In this parameter, you specify the percentage of memory for which over-reserving is allowed. If there is no room for it, no memory is allocated and the reservation will be denied. This is the safest option recommended for PostgreSQL. OOM-Killer is affected by another element - the ability to swap, which is controlled by the variable cat /proc/sys/vm/swappiness. These values ​​tell the kernel how to handle paging. The larger the value, the less likely it is that OOM will terminate the process, but due to I/O operations, this will negatively impact the database. And vice versa - the lower the value, the higher the likelihood of OOM-Killer interference, but the database performance is also higher. The default value is 60, but if the entire database fits in memory, it's better to set the value to 1.

Conclusion

Don't let the "killer" in OOM-Killer scare you. In this case, the killer will be the savior of your system. It "kills" the worst processes and saves the system from crashing. To avoid having to use OOM-Killer to terminate PostgreSQL, set to vm.overcommit_memory a value of 2. This does not guarantee that the OOM-Killer will not have to intervene, but it will reduce the chance of forcibly terminating the PostgreSQL process.

Source: habr.com