🥇Kuba: ndiani anoba CPU nguva kubva kumashini chaiwo

Mhoro! Ini ndoda kukuudza nemazwi akareruka nezve mechanics ekuba mukati memuchina chaiwo uye nezve zvimwe zvisiri pachena zvigadzirwa zvatakakwanisa kuziva panguva yekutsvagisa kwayo, iyo yandaifanira kunyura mairi sehunyanzvi director wepuratifomu yegore. Mail.ru Cloud Solutions. Chikuva chinomhanya paKVM.

CPU kuba nguva ndiyo nguva iyo chaiyo muchina usingagamuchire processor zviwanikwa pakuitwa kwayo. Iyi nguva inongoverengerwa muvaenzi masisitimu anoshanda munzvimbo dze virtualization. Zvikonzero zvekuti zviwanikwa izvi zvakanyanya kugoverwa zvinoenda, sehupenyu, hazvina kujeka. Asi isu takasarudza kuzvifungidzira, uye takatoita akati wandei ekuedza. Hazvirevi kuti isu tave kuziva zvese nezvekuba, asi isu tichakuudza chimwe chinhu chinonakidza ikozvino.

1. Chii chinonzi kuba

Saka, kuba ndiyo metric inoratidza kushomeka kwe processor nguva yemaitiro mukati memuchina chaiwo. Sezvakatsanangurwa muKVM kernel chigambaStealth inguva iyo iyo hypervisor iri kuita mamwe maitiro pane iyo host OS kunyangwe yakamisa iyo chaiyo muchina maitiro ekuurayiwa. Kureva kuti, kuba kunoverengerwa semusiyano uripo pakati penguva iyo maitiro akagadzirira kuita uye nguva iyo maitiro akagoverwa processor nguva.

Iyo chaiyo muchina kernel inogamuchira yekuba metric kubva kune hypervisor. Panguva imwecheteyo, iyo hypervisor hainyatso kutsanangura mamwe maitiro airi kuita, inongoti "ndichiri kubatikana, handikwanise kukupa nguva." PaKVM, tsigiro yekuverengera kuba yakawedzerwa zvigamba. Pane zvinhu zviviri zvakakosha pano:

Iwo chaiwo muchina unodzidza nezve kuba kubva kune hypervisor. Ndiko kuti, kubva pakuona kwekurasikirwa, kune maitiro pamushini chaiwo pachawo ichi chiyero chisina kunanga chinogona kuve pasi pekukanganiswa kwakasiyana.
Iyo hypervisor haigovane ruzivo nemuchina chaiwo nezve zvimwe zvauri kuita - chinhu chikuru ndechekuti haipe nguva pairi. Nekuda kweizvi, iyo chaiyo muchina pachayo haigone kuona kukanganisa muchiratidzo chekuba, iyo inogona kuongororwa nemhando yemakwikwi maitiro.

2. Chii chinokanganisa kuba

2.1. Kuba kukarukureta

Chaizvoizvo, kuba kunoverengerwa zvakangofanana neyakajairwa CPU yekushandisa nguva. Iko hakuna ruzivo rwakawanda pamusoro pekuti kudzokorodza kunotariswa sei. Zvichida nokuti vanhu vakawanda vanofunga kuti mubvunzo uyu wakajeka. Asi pano panewo matambudziko. Kuti uzive maitiro aya, unogona kuverenga chinyorwa naBrendan Gregg: iwe uchadzidza nezve akawanda nuances paunenge uchiverenga mashandisirwo uye nezve mamiriro apo iyi kuverenga ichave isiriyo nekuda kwezvikonzero zvinotevera:

Iyo processor inopisa, zvichikonzera kutenderera kusvetuka.
Gonesa/dzima turbo boost, iyo inoshandura processor wachi kumhanya.
Shanduko yehurefu hwechidimbu chenguva inoitika kana uchishandisa processor magetsi ekuchengetedza matekinoroji akadai seSpeedStep.
Dambudziko nekuverenga mavhareji: kufungidzira kushandiswa kweminiti imwe pa80% kunogona kuvanza kuputika kwenguva pfupi kwe100%.
Iyo spin lock inoita kuti processor idzorerwe, asi maitiro emushandisi haaone kufambira mberi mukuita kwayo. Nekuda kweizvozvo, iyo yakaverengerwa processor kushandiswa nemaitiro ichave zana muzana, kunyangwe maitiro acho asingazopedzi nguva ye processor.

Ini handisati ndawana chinyorwa chinotsanangura maverengero akafanana ekuba (kana iwe uchiziva, igovera mune zvakataurwa). Asi, tichitarisa nekodhi kodhi, iyo calculation mechanism yakafanana neyekudzokorodza. Zviri nyore, imwe counter inowedzerwa mu kernel, yakananga KVM maitiro (chaiyo muchina maitiro), iyo inoverenga nguva yeKVM process yakamirira CPU nguva. Iyo counter inotora ruzivo nezve processor kubva kune yayo yakatarwa uye inotarisa kana zvikwekwe zvayo zvese zvinoshandiswa neiyo chaiyo muchina maitiro. Kana zviri izvo zvese, saka tinofunga kuti processor yaingogarwa neiyo chaiyo muchina maitiro. Zvikasadaro, isu tinozivisa kuti processor yanga ichiita chimwe chinhu, kuba kwakaoneka.

Kuverengera kwekubira kunoenderana nematambudziko akafanana neanogara achiverengerwa kudzokororwa. Tisingarevi kuti matambudziko akadaro anowanzoitika, asi anoratidzika kuva anoodza mwoyo.

2.2. Mhando dze virtualization paKVM

Pakutaura, kune marudzi matatu ekuona, ese anotsigirwa neKVM. Iyo nzira yekubira kuitika inogona kuenderana nemhando ye virtualization.

Broadcast. Mune ino kesi, kushanda kweiyo chaiyo muchina unoshanda sisitimu ine yemuviri hypervisor zvishandiso kunoitika seizvi:

Iyo yevaenzi inoshanda sisitimu inotumira murairo kune yayo yevaenzi mudziyo.
Mutyairi wemudziyo wevaenzi anogamuchira murairo, anogadzira chikumbiro cheBIOS yemudziyo uye anotumira kune hypervisor.
Iyo hypervisor process inoshandura kuraira kuraira kwechigadzirwa chemuviri, zvichiita kuti, pakati pezvimwe zvinhu, ive yakachengeteka.
Mutyairi wemuchina wenyama anogashira murairo wakagadziridzwa uye anoutumira kumudziyo wega wega.
Mibairo yekuita mirairo inodzokera kumashure nenzira imwechete.

Kubatsira kweshanduro ndeyekuti inokutendera iwe kutevedzera chero mudziyo uye haidi yakakosha kugadzirira kweiyo inoshanda system kernel. Asi iwe unofanirwa kubhadhara kune izvi, kutanga kune zvese, nekukurumidza.

Hardware virtualization. Muchiitiko ichi, chigadziro pachiyero che hardware chinonzwisisa mirairo kubva kuhurongwa hwekushanda. Iyi ndiyo nzira inokurumidza uye yakanakisisa. Asi, zvinosuruvarisa, haitsigirwe nemidziyo yese yemuviri, hypervisors uye masisitimu anoshanda evaenzi. Parizvino, michina mikuru inotsigira hardware virtualization ndeye processors.

Paravirtualization. Iyo yakajairika sarudzo yedhizaini virtualization paKVM uye kazhinji yakajairika virtualization modhi yevaenzi masisitimu anoshanda. Chimiro chayo ndechekuti kushanda nemamwe ma hypervisor subsystems (semuenzaniso, netiweki kana disk stack) kana kugoverwa kwemapeji ekurangarira kunoitika uchishandisa hypervisor API, pasina kushandura yakaderera-level mirairo. Izvo zvakashata zveiyi virtualization nzira ndeyekuti muenzi anoshanda system kernel inofanirwa kugadziridzwa kuitira kuti igone kutaurirana ne hypervisor uchishandisa iyi API. Asi izvi zvinowanzogadziriswa nekuisa akakosha madhiraivha pane yevaenzi sisitimu yekushandisa. MuKVM iyi API inonzi Virtio API.

Neparavirtualization, zvichienzaniswa nekutepfenyura, nzira inoenda kumudziyo wenyama inodzikiswa zvakanyanya nekutumira mirairo yakananga kubva kumuchina chaiwo kuenda kune hypervisor maitiro pane anotambira. Izvi zvinokutendera kuti ukurumidze kuita mirairo yese mukati memuchina chaiwo. MuKVM, izvi zvinoitwa neiyo virtio API, inongoshanda kune mamwe maturusi, senge network kana disk adapta. Ichi ndicho chikonzero vatyairi vevirtio vakaiswa mukati memichina chaiyo.

Iyo yakaderera yekumhanyisa uku ndeyekuti haasi ese maitiro anomhanya mukati memuchina chaiwo anoramba ari mukati mawo. Izvi zvinogadzira zvimwe zvakakosha zvinogona kukonzera kubereka pakuba. Ini ndinokurudzira kutanga kudzidza kwakadzama kwenyaya iyi ne An API ye virtual I/O: virtio.

2.3. "Fair" kuronga

Muchina chaiwo pane hypervisor, chokwadi, yakajairika maitiro anoteerera mitemo yekuronga (kugovera zviwanikwa pakati pemaitiro) muLinux kernel, saka ngatitarisei zvakanyanya.

Linux inoshandisa inodaidzwa kuti CFS, Completely Fair Scheduler, yave iyo default scheduler kubvira kernel 2.6.23. Kuti unzwisise iyi algorithm, unogona kuverenga iyo Linux Kernel Architecture kana iyo kodhi kodhi. Izvo zvakakosha zveCFS kugovera nguva ye processor pakati pemaitiro zvichienderana nenguva yekuurayiwa kwavo. Iyo yakawanda CPU nguva yekuita inoda, iyo shoma CPU nguva yainogamuchira. Izvi zvinovimbisa kuti maitiro ese anoitwa "zvakanaka" - kuitira kuti imwe nzira irege kugara ichitora ese ma processor, uye mamwe maitiro anogona zvakare kuita.

Dzimwe nguva iyi paradigm inotungamira kune zvinonakidza zvigadzirwa. Vashandisi veLinux venguva refu vangangorangarira kutonhora kwemupepeti wenguva dzose padesktop paunenge uchimhanyisa zviwanikwa-zvakanyanya kuita senge compiler. Izvi zvakaitika nekuti mabasa asina-resource-yakanyanya mumashandisirwo edesktop aikwikwidzana nemabasa ane simba rekushandisa, senge muunganidzi. CFS inofunga kuti izvi hazvina kunaka, saka nguva nenguva inomisa mupepeti wezvinyorwa uye inoita kuti processor iite mabasa emuunganidzi. Izvi zvakagadziriswa pachishandiswa michina sched_autogroup, asi mamwe akawanda maficha ekugoverwa kwe processor nguva pakati pemabasa akasara. Chaizvoizvo, iyi haisi nyaya pamusoro pekuti zvinhu zvose zvakaipa sei muCFS, asi kuedza kukwevera pfungwa kune chokwadi chokuti kugoverwa "kwakanaka" kwepurogiramu yenguva haisi basa rakanyanya kuduku.

Imwe pfungwa yakakosha mugadziriso ndeye preemption. Izvi zvinodikanwa kudzinga iyo snickering maitiro kubva kune processor uye rega vamwe vashande. Iyo ejection process inonzi context switching. Muchiitiko ichi, chirevo chese chebasa chakachengetedzwa: mamiriro ekugadzirisa, zvinyorwa, nezvimwewo, mushure mokunge nzira yacho inotumirwa kumirira, uye imwe inotora nzvimbo yayo. Uku kuvhiya kunodhura kweOS uye hakuwanzo shandiswa, asi hapana chakakanganisika pazviri. Kuwanzochinja mamiriro ezvinhu kunogona kuratidza dambudziko muOS, asi kazhinji inoenderera uye hairatidze chero chinhu chakanyanya.

Nyaya yakareba yakadai inodiwa kutsanangura chokwadi chimwe: iyo yakawanda processor zviwanikwa maitiro anoedza kushandisa mune yakatendeseka Linux scheduler, inokurumidza kumiswa kuti mamwe maitiro ashandewo. Kunyangwe izvi zvakarurama kana kwete mubvunzo wakaoma unogona kugadziriswa zvakasiyana pasi pemitoro yakasiyana. MuWindows, kusvika nguva pfupi yadarika, mugadziri akatarisana nekutanga kugadzirisa kwedesktop application, izvo zvinogona kukonzera kumashure maitiro kuomesa. Sun Solaris aive nemakirasi mashanu akasiyana evagadziri. Patakatanga virtualization, takawedzera yechitanhatu, Fair share scheduler, nekuti mashanu apfuura haana kushanda zvakakwana neSolaris Zones virtualization. Ndinokurudzira kutanga ongororo yakadzama yenyaya iyi nemabhuku akadai Solaris Internals: Solaris 10 uye OpenSolaris Kernel Architecture kana Kunzwisisa iyo Linux Kernel.

2.4. Nzira yekucherechedza kuba?

Kutarisa kuba mukati memuchina chaiwo, senge chero imwe processor metric, iri nyore: unogona kushandisa chero processor metrics chishandiso. Chinhu chikuru ndechekuti iyo chaiyo muchina uri paLinux. Nechimwe chikonzero Windows haipe ruzivo urwu kune vashandisi vayo. 🙁

Kubuda kweiyo yepamusoro yekuraira: ruzivo rweiyo processor mutoro, mukona yekurudyi - kuba

Iko kuoma kunomuka kana uchiedza kuwana ruzivo urwu kubva kune hypervisor. Iwe unogona kuedza kufanotaura kubira pamushini wekutambira, semuenzaniso, uchishandisa iyo Load Average (LA) parameter - avhareji kukosha kwehuwandu hwemaitiro akamirira mumutsara wekuuraya. Nzira yekuverenga iyi paramende haisi nyore, asi kazhinji, kana LA yakajairwa nenhamba ye processor tambo inodarika 1, izvi zvinoratidza kuti sevha yeLinux yakaremerwa nechimwe chinhu.

Maitiro ese aya akamirira chii? Mhinduro iri pachena ndeye processor. Asi mhinduro haina kunyatso kurongeka, nekuti dzimwe nguva processor ndeyemahara, asi LA inoenda kure. Rangarira sei NFS inodonha uye kuti LA inokura sei. Zvimwe chetezvo zvinogona kuitika nedhisiki uye zvimwe zvekupinza/zvinobuda. Asi muchokwadi, maitiro anogona kumirira kupera kwechero kukiya, kungave kwemuviri, kwakabatana neI / O mudziyo, kana zvine musoro, senge mutex. Izvi zvinosanganisira kukiya pamwero wehardware (mhinduro yakafanana kubva kudhisiki), kana logic (iyo inonzi yekukiya primitives, iyo inosanganisira boka remasangano, mutex adaptive uye spin, semaphores, mamiriro akasiyana, rw kukiya, ipc kukiya. ...).

Chimwe chinhu cheLA ndechekuti inoonekwa seavhareji yekushandisa system. Semuenzaniso, zana maitiro ari kukwikwidza faira rimwe chete, uyezve LA=100. Kukosha kukuru kwakadaro kungaratidzika kuratidza kuti sisitimu yekushandisa yakashata. Asi kune imwe kodhi yakanyorwa yakatsveyama, iyi inogona kunge iri yakajairika mamiriro, kunyangwe hazvo yakashata, uye mamwe maitiro ari muhurongwa hwekushandisa haatamburi.

Nekuda kweavhareji iyi (uye pasina isingasviki miniti), kusarudza chero chinhu neiyo LA chiratidzo harisi iro rinopa mubairo zvakanyanya, rine mhedzisiro isina chokwadi mune chaiyo nyaya. Kana iwe ukaedza kuzvifunga, uchaona kuti zvinyorwa paWikipedia uye zvimwe zviripo zviwanikwa zvinotsanangura chete nyaya dzakareruka, pasina tsananguro yakadzama yemaitiro. Ndinotumira munhu wese anoda, zvakare, pano kuna Brendan Gregg - tevera zvinongedzo pazasi. Ndiani ane usimbe kutaura Chirungu - dudziro yechinyorwa chake chakakurumbira nezve LA.

3. Migumisiro inokosha

Zvino ngatitarisei nyaya huru dzekuba dzatakasangana nadzo. Ini ndichakuudza kuti vanotevera sei kubva pane zvese zviri pamusoro uye kuti vanobatana sei nezviratidzo pane hypervisor.

Kudzokorora. Iyo yakapusa uye yakajairika: iyo hypervisor yakashandiswa zvakare. Chokwadi, kune akawanda anomhanyisa machina emagetsi, akakwira processor mashandisiro mukati mawo, akawanda emakwikwi, LA kushandiswa kunopfuura 1 (yakajairwa ne processor tambo). Zvese zviri mukati mese virtual machines zvinodzikira. Kuba kunopfuudzwa kubva kune hypervisor kuri kukura zvakare, zvinodikanwa kugoverazve mutoro kana kudzima mumwe munhu. Kazhinji, zvinhu zvose zvine musoro uye zvinonzwisisika.

Paravirtualization vs. Single Instances. Pane chete muchina mumwe chete chaiwo pane hypervisor; inopedza chikamu chidiki chayo, asi inogadzira yakakura I / O mutoro, semuenzaniso pa diski. Uye kubva kune imwe nzvimbo kuba kudiki kunowanikwa mairi, kusvika ku10% (sezvinoratidzwa nekuedza kwakawanda).

Nyaya yacho inofadza. Kuba kunoonekwa pano chaizvo nekuda kwekuvharira pamwero wevatyairi veparavirtualized. Kuvhiringidza kunogadzirwa mukati memuchina chaiwo, anogadziriswa nemutyairi uye anotumirwa kune hypervisor. Nekuda kwekuvhiringidza kubata pane hypervisor, yemuchina chaiwo inoita senge chikumbiro chakatumirwa, yakagadzirira kuurayiwa uye yakamirira processor, asi haina kupihwa processor nguva. Musikana wepachokwadi anofunga kuti nguva ino yabiwa.

Izvi zvinoitika panguva iyo buffer inotumirwa, inopinda mu kernel nzvimbo ye hypervisor, uye isu tinotanga kuimirira. Kunyangwe, kubva pakuona kweiyo chaiyo muchina, anofanira kudzoka nekukurumidza. Naizvozvo, maererano nekuba calculation algorithm, ino nguva inonzi yakabiwa. Zvingangodaro, mumamiriro ezvinhu aya panogona kunge kune dzimwe nzira (somuenzaniso, kugadzirisa mamwe mafoni e-sys), asi haafaniri kunge akasiyana.

Scheduler inopesana neakanyanya akaremerwa chaiwo michina. Kana mumwe muchina chaiwo uchitambura nekuba kupfuura vamwe, izvi zvinokonzerwa neanoronga. Iyo yakawanda nzira inoremedza processor, nekukurumidza iyo scheduler inoikanda kunze kuitira kuti vamwe vashandewo. Kana iyo chaiyo muchina ichidya zvishoma, haizomboone kuba: maitiro ayo akatendeseka akagara uye akamirira, isu tinofanirwa kuipa imwe nguva. Kana muchina chaiwo ukaburitsa huremu hwepamusoro pamacores ayo ese, kazhinji unodzingwa kunze kwe processor uye vanoedza kusaupa nguva yakawanda.

Zvakatonyanya kuipa kana maitiro mukati memuchina chaiwo anoedza kuwana yakawanda processor nekuti ivo havagone kubata nekugadzirisa data. Ipapo iyo inoshanda sisitimu pane hypervisor, nekuda kwekutendeseka optimization, ichapa shoma uye shoma processor nguva. Kuita uku kunoitika sechimvuramabwe, uye kuba kunosvetukira kumatenga, kunyangwe mamwe machina chaiwo angasazviona. Uye iyo yakawanda cores, iyo yakanyanya kukanganiswa muchina. Muchidimbu, machina akaremerwa zvakanyanya ane akawanda macores anotambura zvakanyanya.

Low LA, asi kune kuba. Kana LA ingangoita 0,7 (kureva kuti, iyo hypervisor inoratidzika kunge yakaremerwa), asi kuba kunoonekwa mukati memunhu chaiwo muchina:

Sarudzo ine paravirtualization yatotsanangurwa pamusoro. Iwo chaiwo muchina unogona kugamuchira metrics anoratidza kuba, kunyangwe iyo hypervisor yakanaka. Zvinoenderana nemhedzisiro yezviyedzo zvedu, iyi yekubira sarudzo haipfuure gumi muzana uye haifanirwe kuve nekukanganisa kwakakosha mukuita kwemaapplication mukati memuchina chaiwo.
Iyo LA parameter yakaverengerwa zvisizvo. Kunyanya, panguva imwe neimwe yakatarwa inoverengerwa nenzira kwayo, asi kana yakaenzana pamusoro peminiti imwe inoshanduka kuve isingatarisirwe. Semuyenzaniso, kana imwe chaiyo muchina pazvitatu hypervisor anodya processors ayo yose hafu yeminiti, ipapo LA paminiti pamusoro hypervisor achava 0,15; michina mina yakadaro inoshanda panguva imwe chete ichapa 0,6. Uye chokwadi chekuti kwehafu yeminiti pane chimwe nechimwe chazvo pakanga paine kubiwa kwemusango pa25% maererano neiyo LA chiratidzo haichakwanise kuburitswa.
Zvakare, nekuda kwemugadziri akasarudza kuti mumwe munhu ari kudya zvakanyanya uye regai mumwe munhu amirire. Zvichakadaro, ini ndichachinja mamiriro, kubata zvinokanganisa uye kutarisira zvimwe zvakakosha sisitimu zvinhu. Nekuda kweizvozvo, mamwe makina chaiwo haaone chero matambudziko, nepo mamwe achiona kuderera kwekuita kwakakomba.

4. Zvimwe zvinomonyaniswa

Kune zvimwe miriyoni zvimwe zvikonzero zvekukanganisa kudzoka kwakanaka kwe processor nguva pamushini chaiwo. Semuenzaniso, hyperthreading uye NUMA zvinounza matambudziko mukuverenga. Vanovhiringidza zvachose sarudzo yekernel yekuita maitiro, nokuti mugadziri anoshandisa coefficients - uremu, izvo zvinoita kuti kuverenga kuwedzere kuoma pakushandura mamiriro ezvinhu.

Pane kukanganisa nekuda kwehunyanzvi hwekuita senge turbo boost kana, ukuwo, nzira yekuchengetedza simba, iyo, kana ichiverenga mashandisirwo, inogona kuwedzera zvisirizvo kana kuderedza frequency kana kunyange chidimbu chenguva pane server. Kugonesa turbo boost kunoderedza kuita kweimwe processor tambo nekuda kwekuwedzera kwekuita kweimwe. Panguva ino, ruzivo nezve yazvino processor frequency haina kuendeswa kumushini chaiwo, uye inotenda kuti mumwe munhu ari kuba nguva yayo (semuenzaniso, yakakumbira 2 GHz, asi yakagamuchira hafu iyo).

Kazhinji, panogona kuva nezvikonzero zvakawanda zvekukanganisa. Iwe unogona kuwana chimwe chinhu pane imwe system. Zviri nani kutanga nemabhuku andakapa ma link ari pamusoro, uye kudzoreredza manhamba kubva kune hypervisor uchishandisa zvinoshandiswa senge perf, sysdig, systemtap, iyo makumi.

5. Mhedziso

Imwe huwandu hwekuba hunogona kuitika nekuda kweparavirtualization, uye inogona kunzi yakajairika. Vanonyora paInternet kuti kukosha uku kunogona kuva 5-10%. Zvinoenderana nemashandisirwo ari mukati meiyo chaiyo muchina uye pamutoro wainoisa pamidziyo yayo yepanyama. Pano zvakakosha kutarisisa kuti maapplication anonzwa sei mukati memuchina chaiwo.
Huyero hwemutoro pane hypervisor uye kuba mukati memuchina chaiwo hauwanzo kufambidzana zvakajeka; fungidziro dzese dzekuba dzinogona kukanganisa mumamiriro ezvinhu pasi pemitoro yakasiyana.
Murongi ane maitiro akaipa kune maitiro anobvunza zvakawanda. Anoedza kupa zvishoma kune vanokumbira zvakawanda. Michina mikuru mikuru yakaipa.
Kubira kudiki kunogona kuve kwakajairwa kunyangwe pasina paravirtualization (tichifunga nezve mutoro mukati memuchina chaiwo, hunhu hwemutoro wevavakidzani, kugoverwa kwemutoro pane tambo nezvimwe zvinhu).
Kana iwe uchida kufunga nezvekuba mune yakasarudzika sisitimu, iwe unofanirwa kuongorora akasiyana sarudzo, kuunganidza metrics, kunyatso ongorora uye funga nezve nzira yekugovera zvakaenzana mutoro. Kutsauka kubva kune chero zviitiko zvinogoneka, izvo zvinofanirwa kusimbiswa kuyedza kana kutariswa mune kernel debugger.

Source: www.habr.com

Kuba: ndiani anoba processor nguva kubva kumashini chaiwo