Configuring Out-Of-Memory Killer in Linux for PostgreSQL

When a Linux database server unexpectedly crashes, it's necessary to find the cause. There can be several reasons. For example, SIGSEGV — a crash due to a bug in the backend server. But this is rare. More often, it simply runs out of disk space or memory. If disk space runs out, the only option is to free up space and restart the database.

Out-Of-Memory Killer

When a server or process runs out of memory, Linux offers 2 solutions: crash the entire system or terminate the process (application) that is consuming memory. Ideally, it's better to terminate the process and save the OS from crashing. In brief, the Out-Of-Memory Killer is a process that terminates an application to save the kernel from failure. It sacrifices the application to keep the OS running. Let’s first discuss how OOM works and how to control it, then we'll see how the OOM Killer decides which application to terminate.

One of the main tasks of Linux is to allocate memory to processes when they request it. Typically, a process or application requests memory from the OS, and it doesn't always use it fully. If the OS were to grant memory to everyone who requests it but doesn't plan to use it, very soon the memory would run out and the system would fail. To avoid this, the OS reserves memory for a process but does not actually allocate it. Memory is allocated only when the process is truly about to use it. Sometimes, the OS has no free memory, but it reserves memory for a process, and when the process needs it, the OS allocates it if possible. The downside is that sometimes the OS reserves memory, but at the critical moment, there is no free memory, and a system failure occurs. OOM plays a crucial role in this scenario and terminates processes to save the kernel from panic. When the PostgreSQL process is forcibly terminated, the log contains the message:

Out of Memory: Killed process 12345 (postgres).

If the system runs low on memory and it cannot be freed, the function is called out_of_memoryAt this stage, there is only one thing left for her to do — terminate one or more processes. Should the OOM killer terminate the process immediately, or can it wait? Clearly, when out_of_memory is triggered, this is related to waiting for an I/O operation or paging a page to disk. Therefore, the OOM killer should first perform checks and decide based on them which process to terminate. If all the checks below are positive, OOM will terminate the process.

Process selection

When memory runs out, the function is called out_of_memory(). It includes a function select_bad_process(), which receives a rating from the function badness(). The 'worst' process will be targeted. The function badness() selects a process based on certain rules.

1. The kernel needs a minimum amount of memory for itself.
2. A lot of memory needs to be freed.
3. Processes that use little memory shouldn't be terminated.
4. At least a minimum number of processes must be terminated.
5. Complex algorithms that increase the chances of termination for those processes that the user wants to terminate themselves.

After all these checks, OOM examines the score (oom_score). OOM assigns a score to each process and then multiplies this value by the amount of memory. Processes with higher scores have a greater chance of becoming victims of the OOM Killer. Processes associated with a privileged user have a lower score and are less likely to be forcibly terminated. oom_score postgres=# SELECT pg_backend_pid(); pg_backend_pid ---------------- 3813 (1 row)

The Postgres process ID is 3813, so in another shell, you can get the score by using this kernel parameter

vagrant@vagrant:~$ sudo cat /proc/3813/oom_score 2 oom_score:

If you absolutely do not want the OOM Killer to terminate the process, there is another kernel parameter:

oom_score_adj . Add a large negative value to reduce the chances of terminating the process you care about.sudo echo -100 > /proc/3813/oom_score_adj

To set the value

, configure OOMScoreAdjust in the service block: . Add a large negative value to reduce the chances of terminating the process you care about.[Service] OOMScoreAdjust=-1000

Or use

oomprotect in the command rcctl rcctl set servicename oomprotect -1000.

Forced termination of the process

Once one or more processes have been selected, the OOM Killer calls the function

oom_kill_task(). This function sends a termination signal to the process. In case of memory shortage,oom_kill() calls this function to send the SIGKILL signal to the process. A message is recorded in the kernel log. calls this function to send a SIGKILL signal to the process. A message is logged in the kernel log.

Out of Memory: Killed process [pid] [name].

How to Control the OOM-Killer

In Linux, you can enable and disable the OOM-Killer (although disabling it is not recommended). To enable or disable it, use the parameter vm.oom-kill. To enable the OOM-Killer at runtime, execute the command sysctl.

sudo -s sysctl -w vm.oom-kill=1

To disable the OOM-Killer, set the value to 0 in the same command:

sudo -s sysctl -w vm.oom-kill=0

The result of this command will be saved temporarily, only until the first reboot. If you need more persistence, add this line to the file /etc/sysctl.conf:

echo vm.oom-kill=1 >> /etc/sysctl.conf

Another way to enable and disable this is to write the variable panic_on_oom. The value can always be checked in /proc.

$ cat /proc/sys/vm/panic_on_oom
0

If you set the value to 0, the kernel will not panic when the memory runs out.

$ echo 0 > /proc/sys/vm/panic_on_oom

If you set the value to 1, a kernel panic will occur when the memory runs out.

echo 1 > /proc/sys/vm/panic_on_oom

The OOM-Killer can not only be enabled or disabled. We've already mentioned that Linux can reserve more memory for processes than it has but not actually allocate it, and this behavior is controlled by a kernel parameter. This is dictated by the variable vm.overcommit_memory.

You can set the following values for it:

0: the kernel itself decides whether to reserve too much memory. This is the default value in most versions of Linux.
1: the kernel will always reserve extra memory. This is risky, as memory can run out because eventually, processes will likely request their due.
2: the kernel will not reserve more memory than specified by the parameter overcommit_ratio.

In this parameter, you specify the percentage of memory that is allowed to be overcommitted. If there is no space for it, memory will not be allocated, and the overcommitment will be denied. This is the safest option, recommended for PostgreSQL. Another factor affecting the OOM-Killer is the swap behavior, which is controlled by the variable cat /proc/sys/vm/swappinessThese values indicate to the kernel how to handle paging memory. The higher the value, the less likely the OOM will terminate the process, but due to I/O operations, this negatively impacts the database. Conversely, the lower the value, the higher the likelihood of OOM-Killer intervention, but database performance also improves. The default value is 60, but if the entire database fits in memory, it's better to set the value to 1.

Summary

Don't be alarmed by the 'killer' in OOM-Killer. In this case, the killer will be the savior of your system. It 'kills' the most troublesome processes and saves the system from crashing. To avoid having to use OOM-Killer to terminate PostgreSQL, set the value to vm.overcommit_memory 2. This does not guarantee that OOM-Killer will not have to intervene, but it will reduce the likelihood of forced termination of the PostgreSQL process.

Source: habr.com