Vhura sosi yegore mutambo paWebRTC: p2p, vakawanda, zero latency

Software sesevhisi, zvivakwa sesevhisi, chikuva sevhisi, chikuva chekukurukurirana sesevhisi, musangano wevhidhiyo sesevhisi, ko nezvekutamba kwegore sesevhisi? Pakatove nekuedza kwakati wandei kugadzira mutambo wegore (Cloud Gaming), senge Stadia, ichangoburwa neGoogle. Stadia haisi itsva kuWebRTC, asi vamwe vanogona kushandisa WebRTC nenzira imwe chete?

Thanh Nguyen akafunga kuyedza mukana uyu pane yake yakavhurika sosi chirongwa CloudRetro. CloudRetro yakavakirwa paPion, popular WebRTC raibhurari yakavakirwa paGo (ndatenda Ratidzwa kubva kuboka rePion Development nerubatsiro rwavo mukugadzirira chinyorwa ichi). Muchikamu chino, Thanh anopa mhedziso yezvivakwa zvepurojekiti yake, uye zvakare anotaura nezvezvinhu zvinobatsira zvaakadzidza uye matambudziko api aakasangana nawo panguva yebasa rake.

kupinda

Gore rapfuura, Google payakazivisa Stadia, zvakandishungurudza. Pfungwa iyi yakasarudzika uye ine hutsva zvekuti ndaigara ndichinetseka kuti izvi zvaigoneka sei nehunyanzvi hwaivepo. Chishuwo chekunzwisisa zvirinani chinyorwa ichi chakaita kuti ndigadzire yangu vhezheni yeyakavhurika-source cloud game. Mhedzisiro yacho yaingonakidza. Pazasi ini ndinoda kugovera maitiro ekushanda pagore rangu project.

TLDR: pfupi slide vhezheni ine zvakakwirira

Nei cloud mitambo iri mberi

Ini ndinotenda kuti Cloud Gaming ichakurumidza kuve chizvarwa chinotevera kwete chekutamba chete, asiwo dzimwe nzvimbo dzesainzi yekombuta. Cloud mitambo ndiyo yepamusoro yemutengi/sevha modhi. Iyi modhi inokwirisa backend manejimendi uye inoderedza basa rekumberi nekutambira mutambo logic pane iri kure server uye kutenderera mifananidzo / odhiyo kune mutengi. Sevha inoita iyo inorema kugadzirisa kuitira kuti mutengi asisiri patsitsi dzekugumira kwehardware.

Google Stadia inokuita kuti utambe AAA mitambo (kureva mitambo yepamusoro-yekupedzisira blockbuster) pane interface seYouTube. Iyoyo nzira inogona kuiswa kune mamwe anorema offline application senge anoshanda sisitimu kana 2D/3D graphic dhizaini, nezvimwe. kuitira kuti isu tigone kuamhanyisa nguva dzese pane yakaderera-spec zvishandiso pamapuratifomu akawanda.

Remangwana rehunyanzvi uhu: Fungidzira kana Microsoft Windows 10 yakamhanya paChrome browser?

Cloud mitambo ine hunyanzvi

Kubhejera ndeimwe yenzvimbo dzisina kujairika uko nguva dzose, nekukurumidza mushandisi mhinduro inodiwa. Kana pano neapo tikasangana nekunonoka kwechipiri kana tichidzvanya peji, izvi zvinotenderwa. Hupenyu hwevhidhiyo hova dzinowanzo kunonoka masekondi mashoma, asi dzichiri kupa zvine musoro usability. Nekudaro, kana mutambo wacho uchiwanzo mira ne2ms, haugone kutamba. Chinangwa chedu ndechekuwana yakanyanya kuderera latency kuitira kuti musiyano pakati pekuisa uye midhiya ive diki sezvinobvira. Naizvozvo, maitiro echinyakare ekutepfenyura vhidhiyo haashande pano.

General Cloud Game template

Vhura sosi chirongwa CloudRetro

Ndakafunga kugadzira bvunzo yemutambo wegore kuti ndione kana zvese izvi zvaigoneka nezvirambidzo zvenetiweki zvakadaro. Ndakasarudza Golang nokuda kwehumbowo hwepfungwa nokuti waiva mutauro wandaiziva zvikuru uye wakanga wakanyatsokodzera kushandiswa uku nokuda kwezvimwe zvikonzero zvakawanda, sezvandakazoona gare gare. Enda iri nyore uye inokura nekukurumidza; Zviteshi muGo zvakanakira kubata multithreading.

Iyo purojekiti CloudRetro.io ndeye yakavhurika sosi yemitambo sevhisi yemitambo ye retro. Chinangwa chepurojekiti ndechekuunza iyo yakagadzika mutambo wemitambo kumitambo yechinyakare retro uye kuwedzera vakawanda.
Iwe unogona kudzidza zvakawanda nezve purojekiti pano: https://github.com/giongto35/cloud-game.

CloudRetro kushanda

CloudRetro inoshandisa mitambo yeretro kuratidza simba remitambo yegore. Izvo zvinokutendera iwe kuti uwane akawanda akasiyana emitambo zviitiko.

• Kutakurika kwemutambo
  • Kutamba nekukasira paunovhura peji; hapana kurodha kana kuiswa kunodiwa
  • Inoshanda mubrowser yenharembozha, saka hapana software inodiwa kuimhanyisa

• Zvirongwa zvemitambo zvinogona kugovaniswa pamidziyo yakawanda uye kuchengetwa mugore kuitira nguva inotevera paunopinda
• Mutambo unogona kufambiswa, kana kuti unogona kuridzwa nevashandisi vakati wandei kamwechete:
  • Crowdplay seTwitchPlayPokemon, chete yakawanda-muchinjika-chikuva uye yakawanda chaiyo-nguva
  • Offline mitambo online. Vazhinji vashandisi vanogona kutamba pasina kumisikidza network. Samurai Shodown ikozvino inogona kuridzwa nevatambi vaviri pamusoro pe CloudRetro network

  
  Demo vhezheni yemutambo wevazhinji wepamhepo pamidziyo yakasiyana

  Infrastructure

  Zvinodiwa uye tekinoroji stack

  Pazasi pane rondedzero yezvinodiwa zvandinoisa ndisati ndatanga chirongwa.

  1. Mutambi mumwe
  Ichi chinodiwa chingasaite sechakanyanya kukosha kana pachena pano, asi ndechimwe chezvandinotora kutora, chinobvumira mutambo wegore kuti ugare kure kure nemasevhisi ekutepfenyura echinyakare sezvinobvira. Kana tikatarisa pamutambo wemutambi mumwe chete, tinogona kubvisa sevha yepakati kana CDN nekuti isu hatifanirwe kuyerera kune ruzhinji. Panzvimbo pekuisa nzizi kune sink server kana kupfuudza mapaketi kune yepakati WebSocket server, hova dzebasa dzinounzwa zvakananga kumushandisi kuburikidza nevezera-kune-peer WebRTC kubatana.

  2. Low latency media stream
  Kuverenga nezve Stadia, ini ndinowanzo kuona WebRTC ichitaurwa mune zvimwe zvinyorwa. Ndakaona kuti WebRTC tekinoroji yakatanhamara uye yakanakira kushandiswa mumakore emitambo. WebRTC ipurojekiti inopa mabhurawuza uye nharembozha ine chaiyo-nguva kutaurirana kuburikidza neiyo API iri nyore. Inopa peer-to-peer yekubatanidza, yakagadziridzwa kune midhiya, uye ine yakajairwa macodecs akavakirwa-mukati akadai seVP8 uye H264.

  Ini ndakaisa pamberi kuve nechokwadi cheyakanakisa inogoneka mushandisi ruzivo pamusoro pekuchengetedza emhando yepamusoro mifananidzo. Kumwe kurasikirwa kunogamuchirwa mune algorithm. Google Stadia ine imwe nhanho yekudzikisa saizi yemufananidzo pane sevha, uye mafuremu anokwidziridzwa kumhando yepamusoro asati aendeswa kune vezera.

  3. Yakagoverwa zvivakwa zvine geographic routing
  Hazvina mhosva kuti iyo compression algorithm uye kodhi yakagadziridzwa sei, iyo network ichiri iyo yekusarudza iyo inobatsira zvakanyanya kune latency. Iyo dhizaini inofanirwa kunge iine michina yekubatanidza sevha iri padyo nemushandisi kudzikisa nguva yekutenderera-rwendo (RTT). Iyo dhizaini inofanirwa kunge iine 1 murongi uye akati wandei ekufambisa maseva akagoverwa pasirese: US West, US East, Europe, Singapore, China. Ese maseva ekutepfenyura anofanirwa kuve akaparadzaniswa zvachose. Iyo sisitimu inogona kugadzirisa kugovera kwayo kana sevha yabatana kana kusiya network. Saka, netraffic yakakura, kuwedzera mamwe maseva anobvumira kuyera kwakatwasuka.

  4. Browser kugarisana
  Cloud mitambo ndiyo yakanyanya kunaka kana ichida zvishoma kubva kune vashandisi. Izvi zvinoreva kuti zvinokwanisika kumhanya mubrowser. Mabhurawuza anobatsira kuita kuti mutambo wemitambo ugadzikane sezvinobvira kune vashandisi, uchivachengetedza kubva pakuisa software uye hardware. Mabhurawuza anobatsirawo kupa muchinjika-chikuva mashandiro pakati penharembozha uye desktop shanduro. Sezvineiwo, WebRTC inotsigirwa zvakanaka mumhando dzakasiyana dzemabhurawuza.

  5. Kuparadzanisa kwakajeka kwemutambo wemutambo uye sevhisi
  Ini ndinoona sevhisi yemitambo yegore sepuratifomu. Wese munhu anofanira kukwanisa kubatanidza chero chinhu pachikuva. Iye zvino ndabatanidza LibRetro ine Cloud mitambo sevhisi nekuti LibRetro inopa yakanaka mutambo emulator interface yemitambo yeretro seSNES, GBA, PS.

  6. Makamuri evatambi vakawanda, kutamba kwevanhu vakawanda uye kunze kwekubatanidza (deep-link) nemutambo
  CloudRetro inotsigira akawanda matsva emitambo akadai seCrowdPlay uye Online MultiPlayer yemitambo yeretro. Kana vashandisi vakati wandei vakavhura imwechete yakadzika-link pamakomputa akasiyana, vanozoona iwo mutambo unomhanya uye vanotokwanisa kuujoinha.

  Uyezve, nzvimbo dzemitambo dzakachengetwa mukuchengetedza gore. Izvi zvinobvumira vashandisi kuti varambe vachitamba chero nguva pane chero imwe mudziyo.

  7. Horizontal scaling
  Senge chero SAAS mazuva ano, mutambo wegore unofanirwa kugadzirwa kuti uwedzere kuchinjika. Iyo coordinator-worker dhizaini inobvumidza iwe kuwedzera vamwe vashandi kuti vashumire yakawanda traffic.

  8. Hapana kubatana kune rimwe gore
  CloudRetro's sisitimu inotambirwa pane akasiyana makore vanopa (Dhijitari Ocean, Alibaba, tsika mupi) kumatunhu akasiyana. Ini ndinogonesa kumhanya mumudziyo weDocker wezvivakwa uye gadzirisa network zvigadziriso uchishandisa bash script kudzivirira kuvharirwa mune imwechete gore rinopa. Nekusanganisa izvi neNAT Traversal muWebRTC, tinogona kuve neshanduko yekuendesa CloudRetro pane chero makore chikuva uye kunyangwe pamakina emushandisi.

  Architectural design

  Mushandi: (kana sevha yekutepfenyura yataurwa pamusoro) inowanza mitambo, inomhanyisa encoding pombi, uye inoyerera encoded midhiya kune vashandisi. Mamiriro evashandi akagoverwa pasirese, uye mushandi wega wega anogona kubata akawanda evashandisi masesheni panguva imwe chete.

  Coordinator: ine basa rekubatanidza mushandisi mutsva nemushandi akakodzera kutenderera. Murongi anodyidzana nevashandi kuburikidza neWebSocket.

  Game state kuchengetedza: yepakati yekuchengetera kure kune ese mutambo nyika. Iyi chengetedzo inopa akakosha mabasa akadai seremote save/load.

  
  Yepamusoro-nhanho dhizaini yeCloudRetro

  Custom Script

  Kana mushandisi mutsva avhura CloudRetro mumatanho 1 uye 2 anoratidzwa mumufananidzo uri pazasi, murongi pamwe nerunyorwa rwevashandi varipo vanokumbirwa kune yekutanga peji. Mushure meizvi, munhanho 3 mutengi anoverengera kunonoka kwevamiriri vese vachishandisa chikumbiro cheHTTP ping. Rondedzero iyi yekunonoka inozodzoserwa kumurongi kuti ikwanise kuona mushandi akakodzera kwazvo kushandira mushandisi. Nhanho 4 pazasi inogadzira mutambo. Iyo WebRTC yekubatanidza yekushambadzira inotangwa pakati pemushandisi neakapihwa mushandi.
  
  Mushandisi script mushure mekuwana mukana

  Chii chiri mukati memushandi

  Mapaipi emutambo uye ekutepfenyura anochengetwa mukati memushandi ari ega uye nekuchinjana ruzivo ipapo kuburikidza neiyo interface. Parizvino, kutaurirana uku kunoitwa nekufambisa data mundangariro kuburikidza Golang channels mukuita kumwe chete. Chinangwa chinotevera ndechekuparadzanisa, i.e. kuvhurwa kwakazvimirira kwemutambo mune imwe nzira.

  
  Kudyidzana kwezvikamu zvevashandi

  Main components:

  • WebRTC: chikamu chemutengi chinobvuma kupinza kwevashandisi uye zvinoburitsa encoded media kubva kuseva.
  • emulator yemutambo: chikamu chemutambo. Kutenda kuraibhurari yeLibretro, iyo sisitimu inokwanisa kumhanyisa mutambo mukati memaitiro akafanana uye mukati inobata midhiya uye yekuisa rwizi.
  • In-game mafuremu anotorwa uye anotumirwa kune encoder.
  • Mufananidzo/Odhiyo Encoder: pombi encoding inotora midhiya mafuremu, inoaisa kumashure, uye inoburitsa encoded mifananidzo/odhiyo.

  Kutevedzera

  CloudRetro inovimba neWebRTC senge tekinoroji yayo yemusana, saka ndisati ndanyura mukati meruzivo rwekuita kweGolang, ndakafunga kutaura nezveWebRTC pachayo. Iyi inoshamisa tekinoroji yakandibatsira zvakanyanya mukuwana sub-yechipiri latency yekutepfenyura data.

  WebRTC

  WebRTC yakagadzirwa kuti ipe zvemhando yepamusoro peer-to-peer connections pamapurogiramu enharembozha nemabhurawuza vachishandisa maAPI ari nyore.

  NAT Traversal

  WebRTC inozivikanwa nekuita kwayo kweNAT Traversal. WebRTC yakagadzirirwa kutaurirana-kune-vezera. Chinangwa chayo ndechekutsvaga yakanyatsokodzera nzira yakananga, kudzivirira magedhi eNAT uye firewall yekutaurirana-kune-vezera kuburikidza nemaitiro anonzi. chando. Sechikamu chechiitiko ichi, maWebRTC APIs anowana yako yeruzhinji IP kero uchishandisa STUN maseva uye woitumira kune relay server (Tendeuka) apo kubatana kwakananga kusingagoni kusimbiswa.

  Nekudaro, CloudRetro haishandisi zvizere chimiro ichi. Yayo peer-to-peer yekubatanidza haipo pakati pevashandisi, asi pakati pevashandisi nemaseva emakore. Iyo sevha yedivi reiyo modhi ine mashoma akananga kurambidzwa kurambidzwa pane yakajairika mushandisi mudziyo. Izvi zvinokutendera kuti utange wavhura madoko anouya kana kushandisa eruzhinji IP kero zvakananga, sezvo sevha isiri kumashure kweNAT.

  Pakutanga, ndaida kushandura chirongwa ichi kuita chikuva chekugovera mutambo cheCloud Gaming. Pfungwa iyi yaive yekubvumidza vagadziri vemitambo kuti vape mitambo uye zviwanikwa zvekutepfenyura. Uye vashandisi vaizodyidzana nevanopa zvakananga. Nenzira iyi yakasarudzika, CloudRetro ingori dhizaini yekubatanidza yechitatu-bato yekutepfenyura zviwanikwa kune vashandisi, zvichiita kuti iwedzere scalable kana isisiri kubatwa. Basa reWebRTC NAT Traversal pano rakakosha kufambisa peer-to-peer connection kutanga pane yechitatu-party yekutepfenyura zviwanikwa, zvichiita kuti zvive nyore kumusiki kuti abatanidze kunetiweki.

  Video compression

  Vhidhiyo compression chikamu chakakosha chepombi uye inobatsira zvakanyanya mukuyerera kwakatsetseka. Nepo zvisiri madikanwa kuziva zvese zveVP8/H264 vhidhiyo encoding, kunzwisisa pfungwa kunogona kukubatsira kuti unzwisise kutepfenyura vhidhiyo yekumhanyisa sarudzo, kugadzirisa maitiro asingatarisirwe, uye kugadzirisa latency.

  Kudzvanya vhidhiyo yesevhisi yekufambisa kunonetsa nekuti iyo algorithm inofanirwa kuona kuti iyo yakazara encoding nguva + network yekufambisa nguva + decoding nguva yakadzikira sezvinobvira. Mukuwedzera, iyo coding process inofanirwa kuenderana uye inoenderera. Mamwe maencoding tradeoffs haashande — semuenzaniso, isu hatigone kufarira nguva refu yekukodha pane madiki efaira uye nguva dzekudhirodha, kana kushandisa kumanikidza kusingaenderane.

  Pfungwa yekuseri kwevhidhiyo compression ndeyekubvisa zvisina kufanira zvimedu zveruzivo uchichengetedza nhanho inogamuchirwa yekurongeka kwevashandisi. Kuwedzera kune encoding yega static mifananidzo mafuremu, iyo algorithm inokanganisa iyo yazvino furemu kubva kune yakapfuura uye inotevera, saka mutsauko wavo chete unotumirwa. Sezvinoonekwa kubva pamuenzaniso nePacman, chete mapoinzi akasiyana anofambiswa.

  
  Kuenzanisa kwemavhidhiyo mafuremu uchishandisa Pacman semuenzaniso

  Audio compression

  Saizvozvo, iyo odhiyo yekumanikidza algorithm inosiya data risingaonekwe nevanhu. Opus parizvino ndiyo yakanakisa kuita odhiyo codec. Yakagadzirirwa kufambisa odhiyo wave pamusoro peyakarairwa datagram protocol senge RTP (Real Time Transport Protocol). Yayo latency yakaderera pane mp3 uye aac, uye mhando yacho yakakwira. Iyo latency inowanzo kutenderedza 5 ~ 66,5ms.

  Pion, WebRTC muGolang

  Pawn ndeye yakavhurika sosi purojekiti inounza WebRTC kuGolang. Panzvimbo peyakajairwa kuputira kwemaraibhurari eC++ WebRTC, Pion ndeyekuzvarwa Golang kuitiswa kweWebRTC nekuita kurinani, Go kubatanidzwa, uye vhezheni kutonga pane WebRTC protocol.

  Iyo raibhurari zvakare inogonesa kutenderera neakawanda makuru akavakirwa-ins ane sub-second latency. Iyo ine yayo yekuitisa STUN, DTLS, SCTP, nezvimwe. uye zvimwe zviedzo neQUIC uye WebAssembly. Iyi yakavhurika sosi raibhurari pachayo yakanaka chaizvo yekudzidza sosi ine akanakisa zvinyorwa, network protocol kuita, uye inotonhorera mienzaniso.

  Nharaunda yePion, inotungamirwa nemusiki ane shungu kwazvo, ine moyo murefu, ine hurukuro zhinji dzemhando yepamusoro dziri kuitika nezveWebRTC. Kana iwe uchifarira tekinoroji iyi, batanidza http://pion.ly/slack - iwe uchadzidza zvakawanda zvezvinhu zvitsva.

  Kunyora CloudRetro muGolang

  
  Kuitwa kwemushandi muGo

  Enda Zviteshi muChiito

  Nekuda kweiyo Go's yakanaka dhizaini dhizaini, matambudziko ekutepfenyura chiitiko uye concurrency anorerutswa zvakanyanya. Sezviri mudhayagiramu, maGoRoutines akasiyana ane akawanda mazvikamu anomhanya mukufanana. Chimwe nechimwe chikamu chinogadzirisa mamiriro acho uye chinotaurirana kuburikidza nemachaneli. Zvaanosarudza Golang zvinomanikidza chiitiko chimwe chete cheatomu kuti chigadziriswe nguva dzese mumutambo (tiki yemutambo). Izvi zvinoreva kuti hapana kukiya kunodiwa pakugadzirwa uku. Semuenzaniso, kana mushandisi akachengetedza, mufananidzo uzere wemamiriro emutambo unodiwa. Iyi nyika inofanirwa kuramba ichienderera mberi, ichipinda mukati kusvika savha yapera. Munguva yega yega mutambo wetiki, iyo backend inogona chete kubata kuchengetedza kana kupinza mashandiro, zvichiita kuti tambo tambo ive yakachengeteka.

  func (e *gameEmulator) gameUpdate() {
  for {
  	select {
  		case <-e.saveOperation:
  			e.saveGameState()
  		case key := <-e.input:
  			e.updateGameState(key)
  		case <-e.done:
  			e.close()
  			return
  	}
      }
  }

  Fan-in/Fan-out

  Iyi Golang template inokodzera CrowdPlay yangu uye Multiple Player yekushandisa kesi zvakakwana. Kutevedzera iyi pateni, ese ekushandisa mukamuri imwe anovakwa mukati mepakati pekupinda chiteshi. Game media inozoiswa kune vese vashandisi mukamuri imwechete. Nenzira iyi, isu tinowana kupatsanurwa kweiyo mutambo mamiriro pakati akati wandei emitambo zvikamu zvevashandisi vakasiyana.

  
  Synchronization pakati pezvikamu zvakasiyana

  Zvakaipa zveGolang

  Golang haina kukwana. Chanera chinononoka. Kana ichienzaniswa nekuvharisa, Go chiteshi ingori nzira iri nyore yekubata zviitiko panguva imwe chete uye neshinda, asi chiteshi hachipi kuita kwakanyanya. Pane yakaoma blocking logic pasi pe chiteshi. Saka ndakaita zvimwe zvigadziriso pakuita, kushandisazve makiyi uye maatomu hunhu pakutsiva chiteshi kuti ukwidze mashandiro.

  Pamusoro pezvo, muunganidzi wemarara muGolang haatungamirirwe, izvo dzimwe nguva zvinokonzeresa kumira kwenguva refu. Izvi zvinokanganisa zvakanyanya iyo chaiyo-nguva yekufambisa application.

  COG

  Iyo purojekiti inoshandisa iripo yakavhurika sosi Golang VP8/H264 raibhurari yekumanikidza midhiya uye Libretro yemitambo emulators. Ese aya maraibhurari anongori wrappers yeC library muGo uchishandisa COG. Zvimwe zvezvakashata zvakanyorwa mukati iyi post naDave Cheney. Matambudziko andakasangana nawo:

  • kusakwanisa kubata tsaona muCGO, kunyangwe neGolang RecoveryCrash;
  • kutadza kuona mabhodhoro ekuita kana isu tisingakwanise kuona zvakadzama matambudziko muCGO.

  mhedziso

  Ndakazadzisa chinangwa changu chekunzwisisa Cloud mitambo masevhisi uye kugadzira chikuva chinondibatsira kutamba nostalgic retro mitambo neshamwari dzangu online. Iyi purojekiti ingadai isina kuitika pasina raibhurari yePion uye kutsigirwa kwenharaunda yePion. Ndinotenda zvikuru nekukura kwayo kwakanyanya. Iwo akareruka APIs akapihwa neWebRTC nePion akave nechokwadi chekubatanidzwa kusina musono. Humbowo hwangu hwekutanga hwepfungwa hwakaburitswa svondo rimwe chetero, kunyangwe ini ndakanga ndisina ruzivo rwekutanga peer-to-peer (P2P) kutaurirana.

  Pasinei nekureruka kwekubatanidzwa, P2P kutenderera inzvimbo yakaoma kwazvo musainzi yekombuta. Anofanirwa kutarisana nekuoma kweakareba-akamira network architectures akadai seIP neNAT kugadzira musangano wepeer-to-peer. Ndichiri kushanda pachirongwa ichi, ndakawana ruzivo rwakawanda rwakakosha nezve networking nekuita optimization, saka ndinokurudzira munhu wese kuti aedze kuvaka zvigadzirwa zveP2P vachishandisa WebRTC.

  CloudRetro inobata kune ese ekushandisa makesi andaitarisira kubva pamaonero angu senge retro gamer. Zvisinei, ndinofunga kuti kune nzvimbo dzakawanda mupurojekiti dzandinogona kuvandudza, dzakadai sekuita kuti network iwedzere kuvimbika uye iite, ichipa mifananidzo yepamusoro yemutambo, kana kukwanisa kugovera mitambo pakati pevashandisi. Ndiri kushanda nesimba pane izvi. Ndapota teverai project uye tsigira kana uchida.

Source: www.habr.com