Kafa Out-Of-Memory Killer a Linux don PostgreSQL

Lokacin da uwar garken bayanai ya daina ba zato ba tsammani a cikin Linux, kuna buƙatar nemo dalilin. Wataƙila akwai dalilai da yawa. Misali, SIGSEGV - gazawar saboda bug a cikin uwar garken baya. Amma wannan ba kasafai ba ne. Mafi yawan lokuta, kawai kuna ƙarewa daga sarari ko ƙwaƙwalwar ajiya. Idan sararin diski ya ƙare, akwai hanya ɗaya kawai - fitar da sarari kuma sake kunna bayanan.

Kisan Wajen Tunawa

Lokacin da uwar garken ko tsari ya ƙare daga ƙwaƙwalwar ajiya, Linux yana ba da mafita 2: rushe tsarin gaba ɗaya ko ƙare tsarin ( aikace-aikacen ) wanda ke cinye ƙwaƙwalwar ajiya. Yana da kyau, ba shakka, don ƙare tsarin kuma ajiye OS daga faɗuwa. A taƙaice, Killer Out-Of-Memory tsari ne da ke kashe aikace-aikacen don ceton kwaya daga faɗuwa. Yana sadaukar da aikace-aikacen don ci gaba da gudanar da OS. Bari mu fara tattauna yadda OOM ke aiki da yadda za a sarrafa shi, sannan mu ga yadda OOM Killer ke yanke shawarar wace aikace-aikacen da zai ƙare.

Ɗaya daga cikin manyan ayyuka na Linux shine rarraba ƙwaƙwalwar ajiya zuwa matakai lokacin da suka nemi shi. Yawanci, tsari ko aikace-aikacen yana buƙatar ƙwaƙwalwar ajiya daga OS, amma ba ya cika amfani da shi. Idan OS ya ba da ƙwaƙwalwar ajiya ga duk wanda ya nema amma ba shi da shirin yin amfani da shi, nan da nan memorin zai ƙare kuma tsarin zai lalace. Don guje wa wannan, OS yana adana ƙwaƙwalwar ajiya don aiwatarwa, amma ba a zahiri ya sake shi ba. Ana keɓance ƙwaƙwalwar ajiya ne kawai lokacin da ainihin tsari zai yi amfani da shi. Yana faruwa cewa OS ba shi da ƙwaƙwalwar ajiyar kyauta, amma yana ba da ƙwaƙwalwar ajiya ga tsari, kuma lokacin da tsari ke buƙatar shi, OS yana rarraba shi idan zai iya. Abin da ya rage shi ne cewa wani lokaci OS yana adana ƙwaƙwalwar ajiya, amma a lokacin da ya dace babu ƙwaƙwalwar ajiyar kyauta, kuma tsarin ya rushe. OOM yana taka muhimmiyar rawa a cikin wannan yanayin kuma yana ƙare matakai don hana kwaya daga firgita. Lokacin da aka tilasta tsarin PostgreSQL ya ƙare, saƙo yana bayyana a cikin log ɗin:

Out of Memory: Killed process 12345 (postgres).

Idan tsarin yana da ƙananan ƙwaƙwalwar ajiya kuma ba za a iya 'yantar da shi ba, ana kiran aikin out_of_memory. A wannan mataki, abu ɗaya kawai ya rage ta yi - kammala tsari ɗaya ko fiye. Shin ya kamata OOM-killer ya ƙare aikin nan da nan ko zai iya jira? A bayyane yake, lokacin da aka kira out_of_memory, yana faruwa ne saboda jiran aikin I/O ko yin amfani da faifai. Don haka, mai kashe OOM dole ne ya fara yin cak kuma, bisa ga su, yanke shawarar cewa ana buƙatar dakatar da tsarin. Idan duk binciken da ke ƙasa yana da inganci, OOM zai ƙare aikin.

Zaɓin tsari

Lokacin da ƙwaƙwalwar ajiya ta ƙare, ana kiran aikin out_of_memory(). Yana da aiki select_bad_process(), wanda ke karɓar kimantawa daga aikin badness(). Hanyar "mafi muni" za a yi niyya. Aiki badness() yana zaɓar tsari bisa ga wasu dokoki.

1. Kwayar tana buƙatar ɗan ƙaramin ƙwaƙwalwar ajiya don kanta.
2. Kuna buƙatar yantar da ƙwaƙwalwar ajiya mai yawa.
3. Babu buƙatar ƙare hanyoyin da ke amfani da ƙananan ƙwaƙwalwar ajiya.
4. Ana buƙatar kammala mafi ƙarancin matakai.
5. Hadaddiyar algorithms waɗanda ke haɓaka damar kammalawa ga waɗannan hanyoyin da mai amfani da kansa ke son kammalawa.

Bayan kammala duk waɗannan cak, OOM yayi nazarin maki (oom_score). OOM ya sanya oom_score kowane tsari, sannan ya ninka wannan darajar ta adadin ƙwaƙwalwar ajiya. Tsari tare da ƙima mafi girma suna da yuwuwar faɗawa ga mai kashe OOM. Hanyoyin da ke da alaƙa da tushen mai amfani suna da ƙarancin ƙima kuma ba su da yuwuwar tilastawa su ƙare.

postgres=# SELECT pg_backend_pid();
pg_backend_pid 
----------------
    3813
(1 row)

ID ɗin tsari na Postgres shine 3813, don haka a cikin wani harsashi yana yiwuwa a sami maki ta amfani da wannan sigar kwaya. oom_score:

vagrant@vagrant:~$ sudo cat /proc/3813/oom_score
2

Idan ba kwa son OOM-Killer ya kashe tsarin kwata-kwata, akwai wani zaɓi na kwaya: oom_score_adj. Ƙara babbar ƙima mara kyau don rage damar kammala aikin da kuke ƙima.

sudo echo -100 > /proc/3813/oom_score_adj

Don saita ƙima oom_score_adj, saita OOMScoreAdaidaita a cikin toshe sabis:

[Service]
OOMScoreAdjust=-1000

Ko amfani oomprotect a cikin tawaga rcctl.

rcctl set <i>servicename</i> oomprotect -1000

Ƙaddamar da aiki

Lokacin da aka riga aka zaɓi ɗaya ko fiye matakai, OOM-Killer yana kiran aikin oom_kill_task(). Wannan aikin yana aika siginar ƙarewa zuwa tsari. Idan akwai karancin ƙwaƙwalwar ajiya oom_kill() Ya kira wannan aikin don aika siginar SIGKILL zuwa aikin. Ana rubuta saƙo zuwa gunkin kwaya.

Out of Memory: Killed process [pid] [name].

Yadda ake sarrafa OOM-Killer

A Linux, zaku iya kunna ko kashe OOM-Killer (kodayake ba a bada shawarar karshen ba). Don kunna ko kashe amfani da siga vm.oom-kill. Don kunna OOM-Killer a lokacin aiki, gudanar da umarni sysctl.

sudo -s sysctl -w vm.oom-kill = 1

Don kashe OOM-Killer, saka ƙimar 0 a cikin wannan umarni:

sudo -s sysctl -w vm.oom-kill = 0

Sakamakon wannan umarni ba zai sami ceto har abada ba, amma sai an sake yi na farko. Idan kuna buƙatar ƙarin dagewa, ƙara wannan layin a cikin fayil ɗin /etc/sysctl.conf:

echo vm.oom-kill = 1 >>/etc/sysctl.conf

Wata hanya don kunnawa da kashewa ita ce rubuta m panic_on_oom. Ana iya bincika ƙimar koyaushe a ciki /proc.

$ cat /proc/sys/vm/panic_on_oom
0

Idan ka saita ƙimar zuwa 0, to lokacin da ƙwaƙwalwar ajiya ta ƙare, ba za a sami firgita kernel ba.

$ echo 0 > /proc/sys/vm/panic_on_oom

Idan ka saita ƙimar zuwa 1, to lokacin da ƙwaƙwalwar ajiya ta ƙare, tsoro na kernel zai faru.

echo 1 > /proc/sys/vm/panic_on_oom

OOM-Killer ba kawai za a iya kunna da kashewa ba. Mun riga mun faɗi cewa Linux na iya adana ƙarin ƙwaƙwalwar ajiya don matakai fiye da yadda ake samu ba tare da an ware shi a zahiri ba, kuma ana sarrafa wannan ɗabi'ar ta hanyar sigar kernel na Linux. Mai canzawa yana da alhakin wannan vm.overcommit_memory.

Kuna iya ƙididdige ƙididdiga masu zuwa gare shi:

0: Kwayar da kanta tana yanke shawarar ko za a ajiye ƙwaƙwalwar ajiya da yawa. Wannan shine tsoho akan yawancin nau'ikan Linux.
1: Kwaya koyaushe zai tanadi ƙarin ƙwaƙwalwar ajiya. Wannan yana da haɗari, saboda ƙwaƙwalwar ajiya na iya ƙare, saboda, mafi mahimmanci, wata rana hanyoyin zasu buƙaci shi.
2: kwaya ba zai tanadi ƙarin ƙwaƙwalwar ajiya fiye da ƙayyadaddun bayanai a cikin siga ba overcommit_ratio.

Tare da wannan siga, kuna ƙididdige yawan adadin ƙwaƙwalwar ajiya da aka ba da izinin yin tanadi fiye da kima. Idan babu wani wuri don shi, ba a ware ƙwaƙwalwar ajiya ba, kuma za a hana ajiyar ajiyar. Wannan shine mafi aminci zaɓi da aka ba da shawarar don PostgreSQL. OOM-Killer yana shafar wani abu - ikon musanyawa, wanda mai canzawa ke sarrafa shi cat /proc/sys/vm/swappiness. Waɗannan dabi'u suna gaya wa kernel yadda ake sarrafa rubutun. Mafi girman ƙimar, ƙananan yuwuwar shine OOM zai ƙare aikin, amma saboda ayyukan I/O yana da mummunan tasiri akan bayanan. Kuma akasin haka - ƙananan ƙimar, mafi girman yiwuwar shiga tsakani OOM-Killer, amma aikin bayanai kuma ya fi girma. Tsohuwar ƙimar ita ce 60, amma idan duk bayanan bayanan sun dace da ƙwaƙwalwar ajiya, yana da kyau a saita ƙimar zuwa 1.

Sakamakon

Kada ka bari "kisan" a OOM-Killer ya tsoratar da kai. A wannan yanayin, mai kisan zai zama mai ceton tsarin ku. Yana "kashe" mafi munin matakai kuma ya ceci tsarin daga faduwa. Don guje wa amfani da OOM-Killer don ƙare PostgreSQL, saita zuwa vm.overcommit_memory darajar 2. Wannan baya bada garantin cewa OOM-Killer ba zai shiga tsakani ba, amma zai rage yuwuwar tilasta tsarin PostgreSQL ya ƙare.

source: www.habr.com