Malingaliro ndi misonkhano yokhudzana ndi njira zina zomwe zitha kukhala zokha zimatuluka m'mabizinesi amitundu yosiyanasiyana tsiku lililonse. Koma kuwonjezera pa mfundo yakuti nthawi yochuluka ingagwiritsidwe ntchito popanga chitsanzo, muyenera kuigwiritsa ntchito poyesa ndikuwunika kuti zotsatira zomwe mwapeza sizongochitika mwachisawawa. Pambuyo pa kukhazikitsidwa, chitsanzo chilichonse chiyenera kuyang'aniridwa ndikuwunika nthawi ndi nthawi.
Ndipo awa ndi magawo onse omwe akuyenera kumalizidwa mu kampani iliyonse, mosasamala kanthu za kukula kwake. Ngati tikukamba za kukula ndi cholowa cha Sberbank, chiwerengero cha kukonza bwino chikuwonjezeka kwambiri. Pofika kumapeto kwa 2019, Sber anali atagwiritsa kale mitundu yopitilira 2000. Sikokwanira kupanga chitsanzo; m'pofunika kugwirizanitsa ndi machitidwe a mafakitale, kupanga ma data marts a zitsanzo zomangira, ndikuwonetsetsa kulamulira kwake pamagulu.
Gulu lathu likupanga nsanja ya Sber.DS. Zimakuthandizani kuthetsa mavuto ophunzirira makina, kufulumizitsa njira yoyesera malingaliro, makamaka imathandizira njira yopangira ndi kutsimikizira zitsanzo, ndikuwongolera zotsatira za chitsanzo mu PROM.
Kuti musanyenge zomwe mukuyembekezera, ndikufuna kunena pasadakhale kuti positi iyi ndi yoyambira, ndipo pansi pa odulidwa, poyambira, timalankhula za zomwe, makamaka, zili pansi pa nsanja ya Sber.DS. Tidzafotokoza nkhani ya moyo wa chitsanzo kuchokera ku chilengedwe mpaka kukhazikitsidwa mosiyana.
Sber.DS ili ndi zigawo zingapo, zofunika kwambiri kukhala laibulale, dongosolo lachitukuko ndi machitidwe opangira zitsanzo.
Laibulale imayang'anira kayendetsedwe ka moyo wachitsanzo kuyambira pomwe lingaliro loti liyipangitse likuwonekera mpaka kukhazikitsidwa kwake mu PROM, kuyang'anira ndikuchotsa ntchito. Maluso ambiri a library amawunikidwa ndi malamulo owongolera, mwachitsanzo, kupereka lipoti ndi kusungirako zitsanzo zophunzitsira ndi zovomerezeka. Ndipotu, iyi ndi kaundula wa zitsanzo zathu zonse.
Dongosolo lachitukuko limapangidwa kuti lipangitse chitukuko chazithunzi ndi njira zotsimikizira. Zitsanzo zopangidwira zimatsimikiziridwa koyambirira ndipo zimaperekedwa ku dongosolo lokonzekera kuti ligwire ntchito zawo zamalonda. Komanso, mu nthawi yothamanga, chitsanzocho chikhoza kuikidwa pa polojekiti kuti nthawi ndi nthawi zikhazikitse njira zovomerezeka kuti ziwonetsetse ntchito yake.
Pali mitundu ingapo ya node mu dongosolo. Zina zimapangidwira kuti zigwirizane ndi magwero osiyanasiyana a deta, zina zimapangidwira kuti zisinthe deta ndikulemeretsa (markup). Pali ma node ambiri opangira mitundu yosiyanasiyana ndi ma node kuti awatsimikizire. Wopanga mapulogalamu amatha kutsitsa deta kuchokera kugwero lililonse, kusintha, kusefa, kuwona deta yapakatikati, ndikuiphwanya m'magawo.
Pulatifomu ilinso ndi ma module okonzeka omwe amatha kukokedwa ndikugwetsedwa pamalo opangira. Zochita zonse zimachitika pogwiritsa ntchito mawonekedwe owonera. Ndipotu, mungathe kuthetsa vutoli popanda mzere umodzi wa code.
Ngati mphamvu zomangidwa sizikwanira, dongosololi limapereka mphamvu yopangira ma modules anu mwamsanga. Tinapanga njira yophatikizira yotukula kutengera kwa iwo omwe amapanga ma module atsopano kuyambira pachiyambi.
Zomangamanga za Sber.DS zimamangidwa pa ma microservices. Pali malingaliro ambiri okhudza microservices. Anthu ena amaganiza kuti ndikwanira kugawa kachidindo ka monolithic m'zigawo, koma nthawi yomweyo amapitabe kumalo omwewo. Microservice yathu iyenera kulumikizana ndi microservice ina kudzera pa REST API. Palibe ma workaround kuti mupeze database mwachindunji.
Timayesa kuwonetsetsa kuti ntchito zisakhale zazikulu komanso zovutirapo: nthawi imodzi sayenera kugwiritsa ntchito ma gigabytes a 4-8 a RAM ndipo iyenera kupereka kuthekera kokweza zopempha mopingasa poyambitsa zochitika zatsopano. Ntchito iliyonse imalumikizana ndi ena kudzera pa REST API (). Gulu lomwe limayang'anira ntchitoyo likufunika kuti API ikhale kumbuyo mpaka kasitomala womaliza yemwe azigwiritsa ntchito.
Pakatikati pa pulogalamuyi idalembedwa ku Java pogwiritsa ntchito Spring Framework. Yankho lake poyambirira lidapangidwa kuti lizitumizidwa mwachangu mumtambo wamtambo, chifukwa chake ntchitoyo idamangidwa pogwiritsa ntchito makina osungira. (). Pulatifomu ikusintha nthawi zonse, pokhudzana ndi kuchuluka kwa ntchito zamabizinesi (zolumikizira zatsopano, AutoML zikuwonjezedwa) komanso potengera luso laukadaulo.
Chimodzi mwazinthu za nsanja yathu ndikuti titha kuyendetsa kachidindo kopangidwa mu mawonekedwe owonera pamtundu uliwonse wa Sberbank wopha. Tsopano pali awiri aiwo: imodzi pa Hadoop, ina pa OpenShift (Docker). Sitikuyimira pamenepo ndikupanga ma module ophatikizira kuti tiyendetse ma code pazipangizo zilizonse, kuphatikiza pamwambo komanso pamtambo. Ponena za kuthekera kophatikizana bwino mu chilengedwe cha Sberbank, tikukonzekeranso kuthandizira ntchito ndi malo omwe akupha. M'tsogolomu, yankho likhoza kuphatikizidwa mosavuta "kunja kwa bokosi" kumalo aliwonse a bungwe lililonse.
Amene adayesapo kuthandizira yankho lomwe limayendetsa Python pa Hadoop mu PROM amadziwa kuti sikokwanira kukonzekera ndikupereka malo ogwiritsira ntchito Python ku datanode iliyonse. Chiwerengero chachikulu cha malaibulale a C / C ++ ophunzirira makina omwe amagwiritsa ntchito ma module a Python sangakulole kuti mupumule mosavuta. Tiyenera kukumbukira kusintha phukusi powonjezera malaibulale atsopano kapena maseva, ndikusunga zobwerera m'mbuyo ndi code yomwe yakhazikitsidwa kale.
Pali njira zingapo zochitira izi. Mwachitsanzo, konzani malaibulale angapo omwe amagwiritsidwa ntchito pafupipafupi ndikuwagwiritsa ntchito mu PROM. Pakugawa kwa Cloudera's Hadoop, nthawi zambiri amagwiritsa ntchito . Komanso tsopano mu Hadoop ndizotheka kuthamanga -zotengera. Nthawi zina zosavuta ndizotheka kupereka code pamodzi ndi phukusi .
Banki imatenga chitetezo chogwiritsa ntchito nambala yachipani chachitatu mozama kwambiri, kotero timapindula kwambiri ndi zatsopano za Linux kernel, pomwe njira ikuyenda kumalo akutali. , mutha kuchepetsa, mwachitsanzo, kupeza ma netiweki ndi disk yakomweko, zomwe zimachepetsa kwambiri kuthekera kwa code yoyipa. Madera a data a dipatimenti iliyonse amatetezedwa komanso kupezeka kwa eni ake okhawo. Pulatifomu imatsimikizira kuti deta yochokera kudera lina ikhoza kufika kudera lina pokhapokha kudzera mu ndondomeko yosindikizira deta ndi kulamulira pazigawo zonse kuchokera ku magwero mpaka kufika kumtunda wa deta mu malo osungiramo zinthu zomwe mukufuna.
Chaka chino tikukonzekera kumaliza MVP yoyambitsa zitsanzo zolembedwa mu Python / R / Java pa Hadoop. Tadziyika tokha ntchito yofuna kuphunzira momwe tingayendetsere malo aliwonse achikhalidwe pa Hadoop, kuti tisachepetse ogwiritsa ntchito nsanja yathu mwanjira iliyonse.
Kuonjezera apo, monga momwe zinakhalira, akatswiri ambiri a DS ndi abwino kwambiri pa masamu ndi ziwerengero, amapanga zitsanzo zabwino, koma sadziwa bwino kusintha kwakukulu kwa deta, ndipo amafunikira thandizo la akatswiri athu a deta kuti akonzekere zitsanzo za maphunziro. Tidaganiza zothandiza anzathu ndikupanga ma module osavuta akusintha kokhazikika ndikukonzekera mawonekedwe amitundu pa injini ya Spark. Izi zikuthandizani kuti mutenge nthawi yochulukirapo ndikupanga zitsanzo osati kudikirira akatswiri opanga ma data kuti akonze deta yatsopano.
Timalemba ntchito anthu odziwa zinthu m'madera osiyanasiyana: Linux ndi DevOps, Hadoop ndi Spark, Java ndi Spring, Scala ndi Akka, OpenShift ndi Kubernetes. Nthawi ina tidzakambirana za laibulale yachitsanzo, momwe chitsanzocho chimayendera pa moyo wa kampani, momwe kutsimikiziridwa ndi kukhazikitsa kumachitika.
Source: www.habr.com