Sata: wanda ke satar lokacin sarrafawa daga injunan kama-da-wane

Sata: wanda ke satar lokacin sarrafawa daga injunan kama-da-wane

Sannu! Ina so in gaya muku a cikin sauƙi game da injiniyoyin sata a cikin injina da kuma game da wasu kayan tarihi marasa ma'ana waɗanda muka yi nasarar ganowa yayin bincikensa, waɗanda dole ne in nutse a matsayin darektan fasaha na dandalin girgije. Mail.ru Cloud Solutions. Dandalin yana gudana akan KVM.

Lokacin satar CPU shine lokacin da injin kama-da-wane ba ya karɓar albarkatun sarrafawa don aiwatar da shi. Wannan lokacin ana ƙidaya shi ne kawai a cikin tsarin aiki na baƙo a cikin mahallin ƙirƙira. Dalilan inda mafi yawan albarkatun da aka ware, kamar a rayuwa, ba su da tabbas. Amma mun yanke shawarar gano shi, har ma da aiwatar da gwaje-gwaje da yawa. Ba wai yanzu mun san komai game da sata ba, amma za mu gaya muku wani abu mai ban sha'awa a yanzu.

1. Menene sata

Don haka, sata ma'auni ne wanda ke nuna ƙarancin lokacin sarrafawa don aiwatarwa a cikin injin kama-da-wane. Kamar yadda aka bayyana a cikin KVM kernel patchStealth shine lokacin da hypervisor ke aiwatar da wasu matakai akan OS mai watsa shiri ko da yake ya yi layi akan tsarin injin kama-da-wane don aiwatarwa. Wato ana ƙididdige sata a matsayin bambanci tsakanin lokacin da tsarin ke shirye don aiwatarwa da lokacin da aka keɓe lokacin aikin na'ura.

Kernel ɗin inji yana karɓar ma'aunin sata daga hypervisor. A lokaci guda, hypervisor ba ya ƙayyade ainihin abin da sauran hanyoyin da yake gudana ba, kawai yana cewa "yayin da nake aiki, ba zan iya ba ku lokaci ba." A kan KVM, an ƙara tallafi don lissafin sata faci. Akwai mahimman abubuwa guda biyu anan:

  • Injin kama-da-wane yana koya game da sata daga hypervisor. Wato, daga mahangar hasara, ga matakai akan na'urar kama-da-wane da kanta wannan ma'aunin kaikaice ne wanda zai iya fuskantar murdiya iri-iri.
  • Hypervisor ba ya raba bayanai tare da na'ura mai mahimmanci game da abin da yake yi - babban abu shi ne cewa baya ba da lokaci zuwa gare shi. Saboda haka, na'urar kama-da-wane da kanta ba zata iya gano murdiya a cikin alamar sata ba, wanda za'a iya tantance shi ta yanayin tsarin gasa.

2. Abin da ke shafar sata

2.1. Satar lissafi

Ainihin, ana ƙididdige sata kusan daidai da lokacin amfani da CPU na yau da kullun. Babu bayanai da yawa game da yadda ake la'akari da sake yin amfani da su. Wataƙila saboda yawancin mutane suna ɗaukar wannan tambayar a fili. Amma kuma a nan akwai ramuka. Don sanin kanku da wannan tsari, kuna iya karantawa labarin Brendan Gregg: za ku koyi game da nuances da yawa lokacin ƙididdige amfani da kuma game da yanayin lokacin da wannan lissafin zai yi kuskure saboda dalilai masu zuwa:

  • Mai sarrafawa yayi zafi sosai, yana haifar da hawan keke zuwa tsalle.
  • Kunna / kashe haɓakar turbo, wanda ke canza mitar agogo mai sarrafawa.
  • Canji a cikin tsayin yanki wanda ke faruwa lokacin amfani da fasahar ceton wutar lantarki kamar SpeedStep.
  • Matsala tare da ƙididdige matsakaici: ƙididdige amfani da minti ɗaya a 80% na iya ɓoye fashewar ɗan gajeren lokaci na 100%.
  • Kulle juzu'i yana sa mai sarrafawa ya dawo da shi, amma tsarin mai amfani baya ganin wani ci gaba a aiwatar da shi. Sakamakon haka, amfani da na'ura mai ƙididdigewa ta hanyar tsarin zai zama ɗari bisa ɗari, kodayake tsarin ba zai cinye lokacin sarrafawa ba.

Ban sami labarin da ke kwatanta lissafin irin wannan don sata ba (idan kun sani, raba shi a cikin sharhi). Amma, yin la'akari da lambar tushe, tsarin lissafin daidai yake da na sake amfani da shi. Kawai, ana ƙara wani ƙididdiga a cikin kwaya, kai tsaye don tsarin KVM (tsarin injin kama-da-wane), wanda ke ƙididdige tsawon lokacin aikin KVM yana jiran lokacin CPU. Ma'aunin yana ɗaukar bayanai game da na'ura mai sarrafawa daga ƙayyadaddun sa kuma yana bincika ko ana amfani da duk ticks ɗin sa ta hanyar injin kama-da-wane. Idan haka ne, to muna ɗauka cewa processor ɗin ya shagaltu da tsarin injin kama-da-wane. In ba haka ba, muna sanar da cewa processor yana yin wani abu dabam, sata ya bayyana.

Tsarin kirgawa na sata yana fuskantar matsaloli iri ɗaya kamar kirgawa na sake amfani da su na yau da kullun. Ba a ce irin waɗannan matsalolin suna bayyana sau da yawa ba, amma suna kallon sanyi.

2.2. Nau'in haɓakawa akan KVM

A faɗin magana, akwai nau'ikan haɓakawa iri uku, waɗanda KVM ke goyan bayansu. Hanyar faruwar sata na iya dogara ne akan nau'in haɓakawa.

Watsa shirye-shirye. A wannan yanayin, aikin na'ura mai kwakwalwa tare da na'urorin hypervisor na jiki yana faruwa kamar haka:

  1. Tsarin aiki na baƙo yana aika umarni zuwa na'urar baƙo.
  2. Direban na'urar baƙo yana karɓar umarni, ya haifar da buƙatun na'urar BIOS kuma aika shi zuwa hypervisor.
  3. Tsarin hypervisor yana fassara umarni zuwa umarni don na'urar zahiri, sanya shi, a tsakanin sauran abubuwa, mafi aminci.
  4. Direban na'urar jiki yana karɓar umarnin da aka gyara kuma ya aika zuwa na'urar ta zahiri kanta.
  5. Sakamakon aiwatar da umarni yana komawa ta hanya ɗaya.

Amfanin fassarar shine yana ba ku damar yin koyi da kowace na'ura kuma baya buƙatar shiri na musamman na kernel tsarin aiki. Amma dole ne ku biya wannan, da farko, cikin sauri.

Haɓaka Hardware. A wannan yanayin, na'urar a matakin hardware tana fahimtar umarni daga tsarin aiki. Wannan ita ce hanya mafi sauri kuma mafi kyau. Amma, abin takaici, ba a samun goyan bayan duk na'urorin jiki, hypervisors da tsarin aiki na baƙi. A halin yanzu, manyan na'urorin da ke goyan bayan sarrafa kayan aikin su ne na'urori masu sarrafawa.

Paravirtualization. Zaɓin da ya fi dacewa don ƙaƙƙarfan na'ura akan KVM kuma gabaɗaya yanayin haɓakawa na gama gari don tsarin aiki na baƙo. Bambancinsa shine yin aiki tare da wasu subsystems na hypervisor (misali, tare da hanyar sadarwa ko tari) ko rarraba shafukan ƙwaƙwalwar ajiya yana faruwa ta amfani da API hypervisor, ba tare da fassarar ƙananan umarni ba. Lalacewar wannan hanyar haɓakawa shine cewa dole ne a gyaggyara kernel ɗin baƙo don ta iya sadarwa tare da hypervisor ta amfani da wannan API. Amma yawanci ana warware wannan ta hanyar shigar da direbobi na musamman akan tsarin aiki na baƙi. A cikin KVM ana kiran wannan API API ɗin virtio.

Tare da paravirtualization, idan aka kwatanta da watsa shirye-shirye, hanyar zuwa na'urar ta jiki ta ragu sosai ta hanyar aika umarni kai tsaye daga na'ura mai mahimmanci zuwa tsarin hypervisor akan mai watsa shiri. Wannan yana ba ku damar hanzarta aiwatar da duk umarnin da ke cikin na'urar kama-da-wane. A cikin KVM, ana yin wannan ta vintio API, wanda ke aiki don wasu na'urori kawai, kamar hanyar sadarwa ko adaftar diski. Wannan shine dalilin da ya sa ake shigar da direbobin virtio a cikin injina.

Ƙarƙashin wannan haɓakar shi ne cewa ba duk hanyoyin da ke gudana a cikin injin kama-da-wane suka kasance a cikinsa ba. Wannan yana haifar da wasu sakamako na musamman waɗanda zasu iya haifar da haifuwa akan sata. Ina ba da shawarar fara cikakken nazarin wannan batu tare da API ɗin don kama-da-wane I/O: virtio.

2.3. "Tsarin tsari".

Injin kama-da-wane akan hypervisor, a zahiri, tsari ne na yau da kullun wanda ke biyayya ga dokokin tsarawa (rarrabuwar albarkatu tsakanin matakai) a cikin kernel na Linux, don haka bari mu ɗan duba shi.

Linux yana amfani da abin da ake kira CFS, Cikakken Jadawalin Daidaitawa, wanda ya zama mai tsarawa na asali tun kernel 2.6.23. Don fahimtar wannan algorithm, zaku iya karanta Linux Kernel Architecture ko lambar tushe. Mahimmancin CFS shine rarraba lokacin sarrafawa tsakanin matakai dangane da tsawon lokacin aiwatar da su. Yawancin lokacin CPU da tsari ke buƙata, ƙarancin lokacin CPU yana karɓa. Wannan yana tabbatar da cewa an aiwatar da dukkan matakai "a gaskiya" - don kada wani tsari ya mamaye dukkan na'urori masu sarrafawa, kuma sauran matakai na iya aiwatarwa.

Wani lokaci wannan yanayin yana haifar da abubuwa masu ban sha'awa. Masu amfani da Linux na dogon lokaci suna iya tuna daskarewa na editan rubutu na yau da kullun akan tebur yayin gudanar da aikace-aikace masu ƙarfi kamar mai tarawa. Wannan ya faru ne saboda ayyuka marasa ƙarfi a cikin aikace-aikacen tebur sun yi gasa tare da ayyuka masu ƙarfi, kamar mai tarawa. CFS na tunanin wannan rashin adalci ne, don haka lokaci-lokaci yana dakatar da editan rubutu kuma ya bar mai sarrafawa ya sarrafa ayyukan mai tarawa. An gyara wannan ta hanyar amfani da tsari sched_autogroup, amma da yawa wasu siffofi na rarraba lokacin sarrafawa tsakanin ayyuka sun kasance. A gaskiya, wannan ba labari ba ne game da yadda duk abin da ke cikin CFS ba shi da kyau, amma ƙoƙari na jawo hankali ga gaskiyar cewa "daidaitaccen" rarraba lokacin sarrafawa ba shine mafi ƙarancin aiki ba.

Wani muhimmin batu a cikin mai tsarawa shine preemption. Wannan wajibi ne don fitar da tsarin snickering daga processor kuma bari wasu suyi aiki. Ana kiran tsarin fitarwar mahallin. A wannan yanayin, ana kiyaye dukkanin mahallin aikin: yanayin tari, rajista, da dai sauransu, bayan haka an aika da tsari don jira, kuma wani ya ɗauki wurinsa. Wannan aiki ne mai tsada ga OS kuma ba kasafai ake amfani da shi ba, amma babu wani abu da ya dace da shi. Sauye-sauyen mahallin akai-akai na iya nuna matsala a cikin OS, amma yawanci yana ci gaba kuma baya nuna wani abu musamman.

Ana buƙatar irin wannan dogon labari don bayyana hujja ɗaya: yawancin albarkatun sarrafawa da tsari ke ƙoƙarin cinyewa a cikin mai tsara tsarin Linux na gaskiya, da sauri za a dakatar da shi ta yadda sauran hanyoyin suma su yi aiki. Ko wannan daidai ne ko a'a, tambaya ce mai sarƙaƙiya wacce za a iya warware ta daban a ƙarƙashin kaya daban-daban. A cikin Windows, har zuwa kwanan nan, mai tsara jadawalin ya mayar da hankali kan aiwatar da fifikon aikace-aikacen tebur, wanda zai iya haifar da tsarin bayanan baya don daskare. Sun Solaris yana da nau'o'in tsarawa daban-daban guda biyar. Lokacin da muka ƙaddamar da tsarin aiki, mun ƙara na shida, Mai tsara rabo mai adalci, saboda biyar da suka gabata ba su yi aiki yadda ya kamata ba tare da karkatar da yankunan Solaris. Ina ba da shawarar fara cikakken nazarin wannan batu tare da littattafai kamar Solaris Internals: Solaris 10 da OpenSolaris Kernel Architecture ko Fahimtar Linux Kernel.

2.4. Yadda ake saka idanu akan sata?

Kulawa da sata a cikin na'ura mai kama-da-wane, kamar kowane ma'aunin sarrafawa, abu ne mai sauƙi: kuna iya amfani da kowane kayan aikin awo na sarrafawa. Babban abu shine cewa injin kama-da-wane yana kan Linux. Don wasu dalilai Windows ba ya ba da wannan bayanin ga masu amfani da shi. 🙁

Sata: wanda ke satar lokacin sarrafawa daga injunan kama-da-wane
Fitowar babban umarni: cikakkun bayanai game da nauyin mai sarrafawa, a cikin ginshiƙi na dama - sata

Wahalar tana tasowa lokacin ƙoƙarin samun wannan bayanin daga hypervisor. Kuna iya ƙoƙarin yin hasashen sata akan na'ura mai ɗaukar hoto, alal misali, ta amfani da madaidaicin Load Average (LA) - matsakaicin ƙimar adadin hanyoyin da ake jira a cikin jerin gwanon kisa. Hanyar ƙididdige wannan sigar ba ta da sauƙi, amma gabaɗaya, idan LA ta daidaita ta adadin zaren sarrafawa ya wuce 1, wannan yana nuna cewa uwar garken Linux ya cika da wani abu.

Menene duk waɗannan matakai ke jira? Amsar a bayyane ita ce mai sarrafawa. Amma amsar ba daidai ba ce, saboda wani lokacin na'urar tana da kyauta, amma LA yana kashe sikelin. Ka tuna yadda NFS ya fadi da yadda LA ke girma. Hakanan zai iya faruwa tare da faifai da sauran na'urorin shigarwa/fitarwa. Amma a zahiri, matakai na iya jira ƙarshen kowane kulle, ko dai na zahiri, mai alaƙa da na'urar I/O, ko ma'ana, kamar mutex. Wannan kuma ya haɗa da kullewa a matakin kayan aiki (madaidaicin amsa daga diski), ko ma'ana (abin da ake kira kulle primitives, wanda ya haɗa da gungu na ƙungiyoyi, mutex adaptive da spin, semaphores, masu canjin yanayi, maƙallan rw, makullin ipc. ...).

Wani fasalin LA shine ana ɗaukarsa azaman matsakaicin tsarin aiki. Misali, matakai 100 suna fafatawa don fayil ɗaya, sannan LA=50. Irin wannan babban darajar zai zama alama yana nuna cewa tsarin aiki mara kyau. Amma ga sauran crookedly rubuta code, wannan na iya zama wani al'ada jihar, duk da cewa kawai shi ne mummuna, da kuma sauran matakai a cikin tsarin aiki ba su sha wahala.

Saboda wannan matsakaicin (kuma a cikin ƙasa da minti ɗaya), ƙaddamar da wani abu ta alamar LA ba shine mafi kyawun aiki ba, tare da sakamako mara tabbas a takamaiman lokuta. Idan kun yi ƙoƙarin gano shi, za ku ga cewa labaran kan Wikipedia da sauran albarkatun da ake da su suna bayyana kawai mafi sauƙi lokuta, ba tare da cikakken bayani game da tsarin ba. Ina aika duk mai sha'awar sake, nan zuwa Brendan Gregg  - bi hanyoyin da ke ƙasa. Wane ne ya yi kasala sosai don ya iya Turanci - fassarar sanannen labarinsa game da LA.

3. Tasiri na musamman

Yanzu bari mu dubi manyan lamuran sata da muka ci karo da su. Zan gaya muku yadda suke bi daga duk abubuwan da ke sama da kuma yadda suke da alaƙa da alamomi akan hypervisor.

Sake yin amfani da su. Mafi sauƙi kuma mafi na kowa: an sake amfani da hypervisor. Lallai, akwai injunan kama-da-wane da yawa, yawan amfani da na'ura mai sarrafawa a cikin su, gasa da yawa, amfani da LA ya fi 1 (al'ada ta zaren processor). Duk abin da ke cikin duk injunan kama-da-wane yana raguwa. Satar da aka watsa daga hypervisor kuma yana girma, ya zama dole don sake rarraba kaya ko kashe wani. Gabaɗaya, komai yana da ma'ana kuma ana iya fahimta.

Paravirtualization vs. Single Misalai. Na'ura mai kama-da-wane ne kawai akan hypervisor; yana cinye ɗan ƙaramin sashi, amma yana samar da babban nauyin I/O, misali akan faifai. Kuma daga wani wuri ƙaramin sata ya bayyana a cikinsa, har zuwa 10% (kamar yadda gwaje-gwaje da yawa suka nuna).

Lamarin yana da ban sha'awa. Sata yana bayyana a nan daidai saboda toshewa a matakin direbobin da ba su dace ba. An ƙirƙiri katsewa a cikin injin kama-da-wane, direban ya sarrafa shi kuma a aika zuwa hypervisor. Sakamakon katsewar da aka yi akan hypervisor, na'ura mai kama da kama yana kama da buƙatun da aka aiko, yana shirye don aiwatarwa kuma yana jiran mai sarrafawa, amma ba a ba shi lokacin sarrafawa ba. Yarinyar tana tunanin cewa an sace wannan lokacin.

Wannan yana faruwa a lokacin da aka aika buffer, yana shiga cikin kernel space na hypervisor, kuma mun fara jira. Kodayake, daga ra'ayi na na'ura mai mahimmanci, ya kamata ya dawo nan da nan. Saboda haka, bisa ga sata lissafin algorithm, wannan lokacin ana daukar sata. Mafi mahimmanci, a cikin wannan yanayin za a iya samun wasu hanyoyin (misali, sarrafa wasu kira na sys), amma kada su bambanta.

Mai tsara tsarawa tare da injunan kama-da-wane masu kayatarwa. Lokacin da injin kama-da-wane ɗaya ke fama da sata fiye da sauran, wannan ya faru ne saboda mai tsara tsarin. Da zarar tsari ya yi lodin na'ura, da zarar mai tsarawa zai fitar da shi ta yadda sauran su ma su yi aiki. Idan na'urar kama-da-wane ta cinye kadan, da kyar za ta ga sata: tsarin sa da gaske ya zauna yana jira, muna buƙatar ba shi ƙarin lokaci. Idan injin kama-da-wane yana samar da matsakaicin nauyi a kan dukkan nau'ikansa, galibi ana fitar da shi daga na'urar kuma suna ƙoƙarin kada su ba shi lokaci mai yawa.

Har ma ya fi muni lokacin da matakai a cikin injin kama-da-wane suke ƙoƙarin samun ƙarin processor saboda ba za su iya jurewa sarrafa bayanai ba. Sa'an nan kuma tsarin aiki a kan hypervisor, saboda ingantawa na gaskiya, zai samar da ƙarancin lokaci mai sarrafawa. Wannan tsari yana faruwa kamar dusar ƙanƙara, kuma yana satar tsalle-tsalle zuwa sararin sama, kodayake sauran injinan kama-da-wane ba za su iya lura da shi ba. Kuma mafi yawan muryoyin, mafi muni da abin ya shafa. A takaice, injunan kama-da-wane da aka ɗora da su tare da daɗaɗɗa da yawa sun fi shan wahala.

Low LA, amma akwai sata. Idan LA yana da kusan 0,7 (wato, ana ganin hypervisor ba a ɗora shi ba), amma ana lura da sata a cikin injunan kama-da-wane:

  • Zaɓin tare da paravirtualization riga aka bayyana a sama. Na'urar kama-da-wane na iya karɓar ma'auni masu nuna sata, kodayake hypervisor yana da kyau. Dangane da sakamakon gwaje-gwajenmu, wannan zaɓin sata bai wuce 10% ba kuma bai kamata ya yi tasiri sosai kan ayyukan aikace-aikace a cikin na'ura mai ƙima ba.
  • An ƙididdige sigar LA ba daidai ba. Fiye da daidai, a kowane takamaiman lokacin ana ƙididdige shi daidai, amma idan aka ƙididdige shi sama da minti ɗaya sai ya zama ba a ƙididdige shi ba. Misali, idan injin kama-da-wane daya bisa uku na hypervisor ya cinye duk na'urorin sarrafa shi na tsawon rabin minti daya, to LA a minti daya akan hypervisor zai zama 0,15; irin waɗannan injunan kama-da-wane guda huɗu da ke aiki a lokaci ɗaya za su ba da 0,6. Kuma gaskiyar cewa tsawon rabin minti daya akan kowannen su an yi satar daji a kashi 25% bisa ga alamar LA ba za a iya cirewa ba.
  • Bugu da ƙari, saboda mai tsara jadawalin wanda ya yanke shawarar cewa wani yana cin abinci da yawa kuma ya bar wani ya jira. A halin yanzu, zan canza mahallin, sarrafa katsewa da kula da wasu mahimman abubuwan tsarin. Sakamakon haka, wasu injunan kama-da-wane ba sa ganin wata matsala, yayin da wasu ke fuskantar mummunar lalacewar aiki.

4. Wasu murdiya

Akwai ƙarin dalilai miliyan guda don karkatar da kyakkyawar dawowar lokacin sarrafawa akan na'ura mai kama-da-wane. Misali, hyperthreading da NUMA suna gabatar da matsaloli cikin lissafi. Sun rikitar da zabin kwaya don aiwatar da tsarin, saboda mai tsarawa yana amfani da ƙididdiga - ma'auni, wanda ke sa lissafin ya fi wuya yayin canza yanayin.

Akwai murdiya saboda fasaha kamar haɓakar turbo ko, akasin haka, yanayin ceton kuzari, wanda, lokacin da ake ƙididdige amfani, na iya ƙarawa ko rage yawan mita ko ma yanki lokaci akan sabar. Haɓaka haɓakar turbo yana rage aikin zaren mai sarrafawa saboda haɓaka aikin wani. A wannan lokacin, ba a watsa bayanai game da mitar processor na yanzu zuwa na'ura mai mahimmanci, kuma ta yi imanin cewa wani yana satar lokacin sa (alal misali, ya buƙaci 2 GHz, amma ya karɓi rabin hakan).

Gabaɗaya, ana iya samun dalilai da yawa na murdiya. Kuna iya samun wani abu dabam akan wani tsari na musamman. Yana da kyau a fara da littattafan da na ba da haɗin kai a sama, da kuma dawo da kididdiga daga hypervisor ta amfani da kayan aiki irin su perf, sysdig, systemtap, wanda dozin.

5. Ƙarshe

  1. Wasu adadin sata na iya faruwa saboda paravirtualization, kuma ana iya ɗaukar shi al'ada. Suna rubuta akan Intanet cewa wannan ƙimar na iya zama 5-10%. Ya dogara da aikace-aikacen da ke cikin injin kama-da-wane da kuma kan nauyin da yake sanyawa a kan na'urorinsa na zahiri. Anan yana da mahimmanci a kula da yadda aikace-aikacen ke ji a cikin injina.
  2. Matsakaicin nauyin nauyi akan hypervisor da sata a cikin na'ura mai kama-da-wane ba koyaushe yana da alaƙa da juna ba; duka kiyasin sata na iya zama kuskure a takamaiman yanayi a ƙarƙashin kaya daban-daban.
  3. Mai tsarawa yana da mummunan hali ga tafiyar matakai da ke tambaya da yawa. Yana ƙoƙari ya ba da ƙasa ga waɗanda suka nemi ƙarin. Manyan injunan kama-da-wane mugunta ne.
  4. Sata kaɗan na iya zama al'ada ko da ba tare da paravirtualization ba (la'akari da nauyin da ke cikin injin kama-da-wane, halaye na nauyin maƙwabta, rarraba kaya a cikin zaren da sauran dalilai).
  5. Idan kuna son gano sata a cikin takamaiman tsari, dole ne ku bincika zaɓuɓɓuka daban-daban, tattara ma'auni, bincika su a hankali kuma kuyi tunanin yadda zaku rarraba kaya daidai. Bambance-bambance daga kowane lamari yana yiwuwa, wanda dole ne a tabbatar da shi ta gwaji ko kuma a duba shi a cikin mai gyara kernel.

source: www.habr.com

Add a comment