Kubwerera kumbuyo, kuphunzira pamakina komanso opanda seva - zinthu zosangalatsa kwambiri kuchokera ku msonkhano wa July Habr
Msonkhano wa Habr si nkhani yoyamba. M'mbuyomu, tidachita zochitika zazikulu za Toaster kwa anthu 300-400, koma tsopano tidaganiza kuti misonkhano yaying'ono ingakhale yoyenera, momwe mungakhazikitsire, mwachitsanzo, mu ndemanga. Msonkhano woyamba wamtunduwu udachitika mu Julayi ndipo adadzipereka kuti athandizire chitukuko. Ophunzirawo adamvetsera malipoti okhudza kusintha kuchokera kumbuyo kupita ku ML komanso za mapangidwe a ntchito ya Quadrupel pa portal ya State Services, komanso adatenga nawo gawo pa tebulo lozungulira loperekedwa kwa Serverless. Kwa iwo omwe sanathe kukhala nawo pamwambowu pamasom'pamaso, mu positi iyi tikukuuzani zinthu zosangalatsa kwambiri.
Kuchokera ku chitukuko chakumbuyo mpaka kuphunzira makina
Kodi mainjiniya a data amachita chiyani ku ML? Kodi ntchito za wopanga backend ndi injiniya wa ML ndizofanana bwanji komanso zosiyana? Kodi muyenera kuchita chiyani kuti musinthe ntchito yanu yoyamba kukhala yachiwiri? Izi zidanenedwa ndi Alexander Parinov, yemwe adapita kukaphunzira makina pambuyo pa zaka 10 za ntchito yakumbuyo.
Alexander Parinov
Masiku ano Alexander amagwira ntchito ngati womanga makina owonera makompyuta ku X5 Retail Group ndipo amathandizira mapulojekiti a Open Source okhudzana ndi masomphenya apakompyuta komanso kuphunzira mozama (github.com/creafz). Maluso ake amatsimikiziridwa ndi kutenga nawo mbali mu 100 yapamwamba ya dziko la Kaggle Master (kaggle.com/creafz), nsanja yotchuka kwambiri ya mpikisano wophunzirira makina.
Chifukwa chiyani kusintha kuphunzira makina
Chaka ndi theka chapitacho, Jeff Dean, wamkulu wa Google Brain, kafukufuku wozama wa Google wogwiritsa ntchito nzeru zopangapanga, adafotokoza momwe mizere ya theka la miliyoni mu Google Translate idasinthidwa ndi netiweki ya Tensor Flow neural yokhala ndi mizere ya 500 yokha. Pambuyo pophunzitsa maukonde, khalidwe la deta linakula ndipo zomangamanga zinakhala zosavuta. Zingawoneke kuti ili ndi tsogolo lathu lowala: sitiyeneranso kulemba code, ndikwanira kupanga ma neuroni ndikuwadzaza ndi deta. Koma muzochita zonse zimakhala zovuta kwambiri.
ML Infrastructure ku Google
Neural network ndi gawo laling'ono chabe la zomangamanga (malo ang'onoang'ono akuda pachithunzi pamwambapa). Machitidwe ambiri othandizira amafunikira kuti alandire deta, kuikonza, kuisunga, kuyang'ana khalidwe, ndi zina zotero, timafunikira zipangizo zophunzitsira, kutumiza makina ophunzirira makina popanga, ndikuyesa code iyi. Ntchito zonsezi ndizofanana ndendende ndi zomwe opanga backend amachita.
Njira yophunzirira makina
Kodi pali kusiyana kotani pakati pa ML ndi backend?
M'mapulogalamu apamwamba, timalemba code ndipo izi zimayang'anira khalidwe la pulogalamuyo. Mu ML, tili ndi kachidindo kakang'ono kachitsanzo ndi deta zambiri zomwe timaponyera pa chitsanzo. Deta mu ML ndi yofunika kwambiri: chitsanzo chomwecho chophunzitsidwa pazinthu zosiyanasiyana chikhoza kusonyeza zotsatira zosiyana kwambiri. Vuto ndilakuti zambiri zimabalalika nthawi zonse ndikusungidwa m'makina osiyanasiyana (zosungirako zaubale, ma database a NoSQL, zipika, mafayilo).
Kusintha kwa data
ML imafuna kumasulira osati kachidindo kokha, monga mwachitukuko chachikale, komanso deta: m'pofunika kumvetsetsa bwino zomwe chitsanzocho chinaphunzitsidwa. Kuti muchite izi, mutha kugwiritsa ntchito laibulale yotchuka ya Data Science Version Control (dvc.org).
Kuyika kwa data
Ntchito yotsatira ndikulemba zolemba. Mwachitsanzo, chongani zinthu zonse pachithunzichi kapena nenani kuti ndi za gulu liti. Izi zimachitika ndi mautumiki apadera monga Yandex.Toloka, ntchito yomwe imakhala yosavuta kwambiri ndi kukhalapo kwa API. Zovuta zimadza chifukwa cha "chinthu chaumunthu": mutha kusintha kuchuluka kwa data ndikuchepetsa zolakwika pang'ono popereka ntchito yomweyo kwa ochita angapo.
Pali nsanja zambiri zophunzirira makina, koma palibe imodzi yomwe imakwaniritsa 70% ya zosowa
Vuto loyamba lomwe munthu ayenera kukumana nalo poika chitsanzo chophunzitsidwa bwino pakupanga likugwirizana ndi chida chomwe amachikonda kwambiri asayansi a data - Jupyter Notebook. Palibe modularity mmenemo, ndiko kuti, zotuluka ndi "nsalu ya phazi" ya code yomwe siinagawidwe mu zidutswa zomveka - ma modules. Chilichonse chimasakanizidwa: makalasi, ntchito, masinthidwe, ndi zina. Khodi iyi ndi yovuta kuyisintha ndikuyesa.
Kodi kuthana ndi izi? Mutha kusiya ntchito, monga Netflix, ndikupanga nsanja yanu yomwe imakulolani kuti mutsegule ma laputopuwa mwachindunji pakupanga, kusamutsa deta kwa iwo monga zolowera ndikupeza zotsatira. Mutha kukakamiza opanga omwe akuyendetsa mtunduwu kuti alembenso kachidindo nthawi zonse, kuiphwanya kukhala ma module. Koma ndi njira iyi n'zosavuta kulakwitsa, ndipo chitsanzo sichingagwire ntchito monga momwe tafunira. Chifukwa chake, njira yabwino ndikuletsa kugwiritsa ntchito Jupyter Notebook pama code code. Ngati, ndithudi, asayansi a data amavomereza izi.
Chitsanzo ngati bokosi lakuda
Njira yosavuta yopezera chitsanzo pakupanga ndikuigwiritsa ntchito ngati bokosi lakuda. Muli ndi mtundu wina wa kalasi yachitsanzo, munapatsidwa zolemera zachitsanzo (magawo a ma neuroni a netiweki yophunzitsidwa), ndipo ngati mutayambitsa kalasi iyi (itanani njira yolosera, idyetseni chithunzi), mudzapeza zina. kulosera ngati zotsatira. Zomwe zimachitika mkatimo zilibe kanthu.
Phatikizani ndondomeko ya seva ndi chitsanzo
Mutha kukwezanso njira ina yosiyana ndikuitumiza pamzere wa RPC (ndi zithunzi kapena deta ina. Pazotulutsa tidzalandira zolosera.
Chitsanzo chogwiritsira ntchito chitsanzo mu Flask:
Vuto ndi njira iyi ndi kuchepa kwa magwiridwe antchito. Tinene kuti tili ndi Phyton code yolembedwa ndi asayansi a data yomwe imachedwa, ndipo tikufuna kufinya magwiridwe antchito. Kuti muchite izi, mutha kugwiritsa ntchito zida zomwe zimasinthira kachidindo kukhala komweko kapena kusinthira kukhala chimango china chopangidwira kupanga. Pali zida zotere pa chimango chilichonse, koma palibe zabwino; muyenera kuziwonjezera nokha.
Zomangamanga mu ML ndizofanana ndi kumbuyo kwanthawi zonse. Pali Docker ndi Kubernetes, kwa Docker kokha muyenera kukhazikitsa nthawi yothamanga kuchokera ku NVIDIA, yomwe imalola njira mkati mwa chidebe kuti mupeze makhadi avidiyo omwe ali nawo. Kubernetes amafunikira pulogalamu yowonjezera kuti athe kuyang'anira ma seva okhala ndi makhadi avidiyo.
Mosiyana ndi mapulogalamu akale, pankhani ya ML pali zinthu zambiri zosunthika pazomangamanga zomwe ziyenera kufufuzidwa ndikuyesedwa - mwachitsanzo, ma code processing data, payipi yophunzitsira yachitsanzo ndi kupanga (onani chithunzi pamwambapa). Ndikofunikira kuyesa kachidindo komwe kumagwirizanitsa mapaipi osiyanasiyana: pali zidutswa zambiri, ndipo mavuto amadza nthawi zambiri pamalire a module.
Momwe AutoML imagwirira ntchito
Ntchito za AutoML zimalonjeza kusankha mtundu woyenera pazolinga zanu ndikuphunzitsa. Koma muyenera kumvetsetsa: deta ndi yofunika kwambiri mu ML, zotsatira zake zimadalira kukonzekera kwake. Markup imachitidwa ndi anthu, omwe ali odzaza ndi zolakwika. Popanda kuwongolera mosamalitsa, zotsatira zake zitha kukhala zinyalala, ndipo sikutheka kupanga makinawo; kutsimikiziridwa ndi akatswiri - asayansi a data - ndikofunikira. Apa ndipamene AutoML imasweka. Koma zingakhale zothandiza posankha zomangamanga - pamene mwakonzekera kale deta ndipo mukufuna kuyendetsa mayesero angapo kuti mupeze chitsanzo chabwino.
Momwe mungalowe mu maphunziro a makina
Njira yosavuta yolowera mu ML ndikukhala mu Python, yomwe imagwiritsidwa ntchito pamaphunziro akuzama (ndi machitidwe okhazikika). Chilankhulochi ndi chovomerezeka pazochitika izi. C ++ imagwiritsidwa ntchito pazinthu zina zamakompyuta, mwachitsanzo, pakuwongolera magalimoto odziyendetsa okha. JavaScript ndi Shell - zowonera ndi zinthu zachilendo monga kuyendetsa neuron mu msakatuli. Java ndi Scala amagwiritsidwa ntchito pogwira ntchito ndi Big Data komanso kuphunzira pamakina. R ndi Julia amakondedwa ndi anthu amene amaphunzira masamu.
Njira yabwino kwambiri yopezera chidziwitso choyambira ndi pa Kaggle; kutenga nawo mbali mumpikisano umodzi wapapulatifomu kumapereka kupitilira chaka chimodzi chophunzirira chiphunzitso. Pa nsanja mukhoza kutenga wina anaika ndi ndemanga kachidindo ndi kuyesa kusintha, kukhathamiritsa ndi zolinga zanu. Bonasi - udindo wanu wa Kaggle umakhudza malipiro anu.
Njira ina ndikulowa nawo gulu la ML ngati woyambitsa backend. Pali zoyambira zambiri zophunzirira makina komwe mungapeze luso pothandiza anzanu kuthetsa mavuto awo. Pomaliza, mutha kujowina gulu limodzi la asayansi a data - Open Data Science (ods.ai) ndi ena.
"Quadrupel" - ntchito ya zidziwitso zomwe zikugwirizana ndi portal "State Services"
Evgeny Smirnov
Wokamba nkhani wotsatira anali mtsogoleri wa dipatimenti ya chitukuko cha e-government, Evgeny Smirnov, yemwe analankhula za Quadruple. Uwu ndi ntchito yodziwitsira zidziwitso za Gosuslugi portal (gosuslugi.ru), zomwe zidayendera kwambiri boma pa Runet. Omvera tsiku ndi tsiku ndi 2,6 miliyoni, palimodzi pali 90 miliyoni olembetsa olembetsa patsamba, omwe 60 miliyoni amatsimikiziridwa. Katundu pa portal API ndi 30 zikwi RPS.
Matekinoloje omwe amagwiritsidwa ntchito kumbuyo kwa State Services
"Quadrupel" ndi ntchito yazidziwitso yomwe imayang'aniridwa, mothandizidwa ndi zomwe wogwiritsa ntchito amalandila chithandizo panthawi yoyenera kwambiri kwa iye mwa kukhazikitsa malamulo apadera azidziwitso. Zofunikira zazikulu popanga ntchitoyo zinali zosinthika komanso nthawi yokwanira yotumizira makalata.
Kodi Quadrupel imagwira ntchito bwanji?
Chithunzi pamwambapa chikuwonetsa imodzi mwamalamulo ogwirira ntchito a Quadrupel pogwiritsa ntchito chitsanzo cha vuto lomwe likufunika kusintha chiphaso choyendetsa. Choyamba, ntchitoyi imayang'ana ogwiritsa ntchito omwe tsiku lawo lotha ntchito limatha mwezi umodzi. Amawonetsedwa chikwangwani chokhala ndi mwayi woti alandire chithandizo choyenera ndipo uthenga umatumizidwa ndi imelo. Kwa ogwiritsa ntchito omwe nthawi yawo yomaliza yatha kale, banner ndi imelo zimasintha. Pambuyo pakusinthana bwino kwaufulu, wogwiritsa ntchito amalandira zidziwitso zina - ndi lingaliro losinthira deta muzodziwika.
Kuchokera pamalingaliro aukadaulo, awa ndi ma groovy scripts momwe code imalembedwera. Zomwe zalowetsedwa ndi deta, zotulukapo ndi zoona / zabodza, zofananira / sizinafanane. Pali malamulo opitilira 50 onse - kuyambira pakuzindikira tsiku lobadwa la wogwiritsa ntchito (tsiku lomwe lili pano ndi lofanana ndi tsiku lobadwa la wogwiritsa ntchito) mpaka zovuta. Tsiku lililonse, malamulowa amazindikiritsa machesi pafupifupi miliyoni miliyoni βanthu ofunikira kudziwitsidwa.
Makanema azidziwitso a Quadrupel
Pansi pa Quadrupel pali nkhokwe yomwe deta ya ogwiritsa ntchito imasungidwa, ndi ntchito zitatu:
Wogwira ntchito cholinga chosinthira deta.
API Yotsalira imatenga ndikupereka zikwangwani zokha ku portal ndi pulogalamu yam'manja.
Nikolay Markov, Senior Data Engineer at Aligned Research Group.
Andrey Tomilenko, wamkulu wa dipatimenti yachitukuko ya RUVDS.
Zokambiranazo zidayendetsedwa ndi wopanga wamkulu Alexander Borgart. Timapereka makangano omwe omvera nawonso adatenga nawo gawo mu chidule chachidule.
- Kodi Serverless mu kumvetsetsa kwanu ndi chiyani?
Andrei: Ichi ndi chitsanzo cha makompyuta - ntchito ya Lambda yomwe imayenera kukonza deta kuti zotsatira zake zimadalira deta yokha. Mawuwa adachokera ku Google kapena kuchokera ku Amazon ndi ntchito yake ya AWS Lambda. Ndikosavuta kuti wothandizira agwire ntchito yotereyi popereka dziwe la mphamvu zake. Ogwiritsa ntchito osiyanasiyana amatha kudziwerengera okha pa ma seva omwewo. Nikolai: Kunena mwachidule, tikusamutsa gawo lina lazinthu zathu za IT ndi malingaliro abizinesi kupita kumtambo, kupita ku kunja. Ara: Kumbali ya omanga - kuyesa kwabwino kupulumutsa chuma, kwa ogulitsa - kupeza ndalama zambiri.
- Kodi Serverless ndi yofanana ndi ma microservices?
Nikolai: Ayi, Serverless ndi gulu lazomangamanga. Microservice ndi gawo la atomiki lamalingaliro ena. Serverless ndi njira, osati "gulu losiyana." Ara: Ntchito Yopanda Seva ikhoza kuikidwa mu microservice, koma izi sizidzakhalanso Zopanda Server, zidzasiya kukhala ntchito ya Lambda. Mu Serverless, ntchito imangoyamba kugwira ntchito ikafunsidwa. Andrei: Amasiyana m'moyo wawo. Tidayambitsa ntchito ya Lambda ndikuyiwala. Zinagwira ntchito kwa masekondi angapo, ndipo kasitomala wotsatira akhoza kukonza pempho lake pamakina ena akuthupi.
- Ndi mamba ati abwinoko?
Ara: Mukakulitsa mopingasa, ntchito za Lambda zimakhala zofanana ndendende ndi ma microservices. Nikolai: Zirizonse zofananira zomwe mungakhazikitse, zidzakhala zochulukirapo; Serverless ilibe vuto pakukulitsa. Ndidapanga chofanizira ku Kubernetes, ndikuyambitsa zochitika 20 "kwinakwake", ndipo maulalo 20 osadziwika adabwezedwa kwa inu. Patsogolo!
- Kodi n'zotheka kulemba backend pa Serverless?
Andrei: Mwachidziwitso, koma sizomveka. Ntchito za Lambda zidzadalira malo amodzi - tiyenera kutsimikizira. Mwachitsanzo, ngati wogwiritsa ntchitoyo adachitapo kanthu, ndiye kuti nthawi ina akadzakumananso ayenera kuwona: ntchitoyo yachitika, ndalamazo zawerengedwa. Ntchito zonse za Lambda zidzatsekereza kuyimba uku. M'malo mwake, gulu la ntchito zopanda Serverless lidzasanduka ntchito imodzi yokhala ndi malo amodzi ofikira botolo ku database.