Hayi Habr!
Idata yeyona asethi ixabisekileyo yenkampani. Phantse yonke inkampani egxile kwidijithali iyakuxela oku. Kunzima ukuphikisana nale nto: akukho nkomfa enye enkulu ye-IT ebanjwe ngaphandle kokuxoxa ngeendlela zokulawula, ukugcina kunye nokucubungula idatha.
Idatha iza kuthi ivela ngaphandle, iphinda iveliswe ngaphakathi kwinkampani, kwaye ukuba sithetha ngedatha evela kwinkampani ye-telecom, ke kubasebenzi bangaphakathi le ndawo yokugcina ulwazi malunga nomxhasi, izinto anomdla kuzo, imikhwa kunye nendawo. Ngokwenza iprofayili efanelekileyo kunye nokwahlulahlula, unikezelo lwentengiso lusebenza kakhulu. Nangona kunjalo, ekusebenzeni, ayizizo zonke izinto ezilunge kakhulu. Idatha egcinwa ziinkampani isenokuphelelwa lixesha, ingafuneki, iphindeke, okanye ubukho bayo ayaziwa nakubani na ngaphandle kwesangqa esimxinwa sabasebenzisi. ¯_(ツ)_/¯
Ngelizwi, idatha kufuneka ilawulwe ngokufanelekileyo - kuphela emva koko iya kuba yi-asethi ezisa inzuzo yokwenene kunye nenzuzo kwishishini. Ngelishwa, ukusombulula imiba yolawulo lwedatha kufuna ukoyisa ubunzima obuninzi. Zibangelwa ubukhulu becala kuzo zombini ilifa lembali ngendlela “yezoosi” zeenkqubo kunye nokungabikho kweenkqubo ezimanyeneyo kunye neendlela zolawulo lwazo. Kodwa kuthetha ukuthini "ukuqhutywa kwedatha"?
Yile nto kanye esiza kuthetha ngayo phantsi kokusikwa, kunye nendlela i-opensource stack esincede ngayo.
Ingqikelelo yolawulo lwedatha yobuchule (DG) sele isaziwa kakuhle kwimarike yaseRussia, kwaye iinjongo eziphunyezwe lishishini ngenxa yokuphunyezwa kwayo zicacile kwaye zibhengezwe ngokucacileyo. Inkampani yethu yayingekho ngaphandle kwaye yazibekela umsebenzi wokuzisa ingcamango yolawulo lwedatha.
Siqale ngaphi ke? Ukuqala, sizenzele iinjongo eziphambili:
- Gcina idatha yethu ifikeleleka.
- Ukuqinisekisa ukungafihli komjikelo wedatha.
- Nika abasebenzisi benkampani idatha engaguqukiyo, engaguqukiyo.
- Nika abasebenzisi benkampani idatha eqinisekisiweyo.
Namhlanje, kukho izixhobo zeklasi zoLawulo lweDatha kwimarike yesoftware.
Kodwa emva kohlalutyo oluneenkcukacha kunye nophononongo lwezisombululo, sabhala inani lamagqabaza abalulekileyo kuthi:
- Uninzi lwabavelisi banikezela ngeseti ebanzi yezisombululo, kuthi kuthi ezingafunekiyo kwaye ziphindaphinde imisebenzi ekhoyo. Ngaphezu koko, kuyabiza ngokwezibonelelo, ukudibanisa kwi-IT landscape yangoku.
- Ukusebenza kunye nojongano lwenzelwe itekhnoloji, hayi abasebenzisi bokuphela kweshishini.
- Izinga eliphantsi lokusinda kweemveliso kunye nokungabikho kokuphunyezwa okuyimpumelelo kwimarike yaseRashiya.
- Iindleko eziphezulu zesoftware kunye nenkxaso eyongezelelweyo.
Iikhrayitheriya kunye nezindululo ezivakaliswe ngasentla malunga nokutshintshwa kwesoftware endaweni yeenkampani zaseRussia ziye zasiqinisekisa ukuba siqhubele phambili kuphuhliso lwethu kwisitaki somthombo ovulekileyo. Iqonga esilikhethileyo laliyi-Django, isakhelo somthombo sasimahla kunye nesivulekileyo esibhalwe kwiPython. Kwaye ke sichonge iimodyuli eziphambili eziza kuba negalelo kwiinjongo ezichazwe ngasentla:
- Irejista yeengxelo.
- Iglosari yeshishini.
- Imodyuli yokuchaza iinguqu zobugcisa.
- Imodyuli yokuchaza umjikelo wobomi bedatha ukusuka kumthombo ukuya kwisixhobo seBI.
- Imodyuli yolawulo lomgangatho wedatha.
Irejista yeengxelo
Ngokweziphumo zezifundo zangaphakathi kwiinkampani ezinkulu, xa kusonjululwa iingxaki ezinxulumene nedatha, abasebenzi bachitha i-40-80% yexesha labo befuna. Ke ngoko, sizibekele umsebenzi wokwenza ulwazi oluvulekileyo malunga neengxelo ezikhoyo ezazifumaneka kuphela kubathengi. Ngaloo ndlela, sinciphisa ixesha lokuvelisa iingxelo ezintsha kunye nokuqinisekisa idemokhrasi yedatha.
Irejista yokunika ingxelo ibe yifestile enye yokunika ingxelo kubasebenzisi bangaphakathi abasuka kwiingingqi ezahlukeneyo, kumasebe nakumacandelo. Idibanisa ulwazi kwiinkonzo zolwazi ezenziwe kwiindawo ezininzi zokugcina iinkampani zenkampani, kwaye zininzi zazo kwiRostelecom.
Kodwa i-registry ayikho nje uluhlu olomileyo lweengxelo eziphuhlisiweyo. Kwingxelo nganye, sinikezela ngolwazi oluyimfuneko ukuze umsebenzisi aziqhelanise nayo:
- inkcazo emfutshane yengxelo;
- ubunzulu bokufumaneka kwedatha;
- icandelo labathengi;
- isixhobo sokujonga;
- igama logcino lweshishini;
- iimfuno zokusebenza kweshishini;
- ikhonkco kwingxelo;
- ikhonkco kwisicelo sofikelelo;
- ubume bokuphunyezwa.
Uhlalutyo lwenqanaba lokusetyenziswa luyafumaneka kwiingxelo, kwaye iingxelo zibekwe phezulu kuluhlu olusekelwe kwi-log analytics ngokusekelwe kwinani labasebenzisi abakhethekileyo. Kwaye akunjalo. Ukongeza kwiimpawu eziqhelekileyo, siye sanikezela ngenkcazo ecacileyo yokwakheka kweengxelo kunye nemizekelo yamaxabiso kunye neendlela zokubala. Iinkcukacha ezinjalo ngokukhawuleza zinika umsebenzisi impendulo ukuba ingxelo iluncedo kuye okanye hayi.
Uphuhliso lwale modyuli yayilinyathelo elibalulekileyo kwidemokhrasi yedatha kwaye lanciphisa kakhulu ixesha elithathayo ukufumana ulwazi olufunekayo. Ukongeza ekunciphiseni ixesha lokukhangela, inani lezicelo kwiqela lenkxaso ukubonelela ngokubonisana liye lancipha. Akunakwenzeka ukuba singaqapheli esinye isiphumo esiluncedo esisifumene ngokuphuhlisa irejista edibeneyo yeengxelo - ukuthintela ukuphuhliswa kweengxelo eziphindwe kabini kwiiyunithi ezahlukeneyo zesakhiwo.
Iglosari yeshishini
Niyazi nonke ukuba nakwinkampani enye, amashishini athetha iilwimi ezahlukeneyo. Ewe, basebenzisa amagama afanayo, kodwa athetha izinto ezahlukeneyo ngokupheleleyo. Iglosari yeshishini yenzelwe ukusombulula le ngxaki.
Kuthi, isichazi-magama seshishini asiyoncwadi yereferensi nje enenkcazelo yamagama kunye nendlela yokubala. Le yindawo epheleleyo yokuphuhlisa, ukuvuma kunye nokuvuma isigama, ukwakha ubudlelwane phakathi kwemigaqo kunye nezinye izinto zexabiso zenkampani. Ngaphambi kokungena kwiglosari yeshishini, ixesha kufuneka lihambe kuzo zonke izigaba zokuvunywa kunye nabathengi beshishini kunye neziko lomgangatho wedatha. Kuphela emva kokuba oku kufumaneka ukuba kusetyenziswe.
Njengoko ndibhale ngasentla, ukungafani kwesi sixhobo kukuba ivumela uxhulumaniso ukusuka kwinqanaba lexesha leshishini ukuya kwiingxelo ezithile zomsebenzisi apho isetyenziswe khona, kunye nenqanaba lezinto eziphathekayo zesiseko sedatha.
Oku kwenziwa kube nokwenzeka ngokusetyenziswa kwezichazi zegama leglosari kwinkcazo ethe kratya yeengxelo zobhaliso kunye nenkcazo yezinto eziphathekayo zesiseko sedatha.
Ngoku, ngaphezulu kwamagama angama-4000 achaziweyo kwaye kwavunyelwana ngawo kuLuhlu lweeNkcazo. Ukusetyenziswa kwayo kwenza kube lula kwaye kukhawulezise ukusetyenzwa kwezicelo ezingenayo zotshintsho kwiinkqubo zolwazi zenkampani. Ukuba isalathisi esifunekayo sele siphunyeziwe kuyo nayiphi na ingxelo, ngoko umsebenzisi uya kubona ngokukhawuleza isethi yeengxelo esele zilungile apho esi salathisi sisetyenzisiweyo, kwaye uya kuba nako ukwenza isigqibo malunga nokusetyenziswa okusebenzayo kokusebenza okukhoyo okanye ukuguqulwa kwayo okuncinci, ngaphandle kokuqalisa. izicelo ezintsha zophuhliso lwengxelo entsha.
Imodyuli yokuchaza iinguqu zobugcisa kunye neDathaLineage
Zeziphi ezi modyuli, uyabuza? Akwanelanga ukuphumeza ngokulula iRejista yeNgxelo kunye noLuhlu lwamagama; kukwayimfuneko ukumisela yonke imiqathango yoshishino kwimodeli yesiseko sedatha. Ngaloo ndlela, sakwazi ukugqiba inkqubo yokwenza umjikelo wobomi bedatha ukusuka kwiinkqubo zomthombo ukuya kumbono we-BI kuzo zonke iileyile zedatha yokugcina idatha. Ngamanye amazwi, yakha iDataLineage.
Siphuhlise i-interface esekelwe kwifomathi esetyenziswe ngaphambili kwinkampani ekuchazeni imithetho kunye nengqiqo yokuguqulwa kwedatha. Ulwazi olufanayo lungeniswa nge-interface njengangaphambili, kodwa inkcazo yegama lokuchonga ukusuka kuluhlu lweshishini ibe yinto efunekayo kuqala. Yile ndlela esakha ngayo unxibelelwano phakathi kweshishini kunye neengqimba zomzimba.
Ngubani oyidingayo? Yintoni eyayingalunganga kwifomati endala osebenze nayo iminyaka emininzi? Zinyuke kangakanani iindleko zabasebenzi ukulungiselela iimfuno zokuvelisa? Kwafuneka sijongane nemibuzo enjalo ngexesha lokuphunyezwa kwesixhobo. Iimpendulo apha zilula kakhulu - sonke siyayifuna le, iofisi yedatha yenkampani yethu kunye nabasebenzisi bethu.
Ngokwenene, abasebenzi kwafuneka balungelelanise; ekuqaleni, oku kukhokelele ekunyukeni okuncinci kweendleko zabasebenzi ukulungiselela amaxwebhu, kodwa sawulungisa lo mba. Ukuziqhelanisa, ukuchonga kunye nokuphucula iindawo ezinengxaki zenze umsebenzi wazo. Siphumelele into ephambili - siphucule umgangatho weemfuno eziphuhlisiwe. Imimandla enyanzelekileyo, iincwadi zereferensi ezidibeneyo, iimaski zokufaka, iitshekhi ezakhelwe ngaphakathi - konke oku kwenza ukuba kube lula ukuphucula umgangatho weenkcazo zenguqu. Siye sasuka kwindlela yokuhambisa izikripthi njengeemfuno zophuhliso kunye nolwazi olwabelana ngalo olufumaneka kuphela kwiqela lophuhliso. I-database ye-metadata eveliswayo inciphisa kakhulu ixesha elifunekayo ukwenza uhlalutyo lokubuyisela kwaye inika amandla okuvavanya ngokukhawuleza impembelelo yeenguqu kunoma yimuphi umgca we-IT landscape (iingxelo zokubonisa, i-aggregates, imithombo).
Oku kunantoni na kubasebenzisi abaqhelekileyo beengxelo, ziziphi iingenelo kubo? Siyabulela kwisakhono sokwakha iDataLineage, abasebenzisi bethu, kwanabo bakude neSQL kunye nezinye iilwimi zeprogram, ngokukhawuleza bafumana ulwazi malunga nemithombo kunye nezinto ngokusekelwe kwingxelo ethile eyenziwa.
Imodyuli yoLawulo loMgangatho weDatha
Yonke into ebesithetha ngayo ngasentla malunga nokuqinisekisa ukungafihli kwedatha ayibalulekanga ngaphandle kokuqonda ukuba idatha esiyinika abasebenzisi ichanekile. Enye yeemodyuli ezibalulekileyo zengqikelelo yethu yoLawulo lweDatha yimodyuli yolawulo lomgangatho wedatha.
Kwinqanaba langoku, le yikhathalogu yeetshekhi kumaziko akhethiweyo. Injongo ekhawulezileyo yokuphuhliswa kwemveliso kukwandisa uluhlu lweetshekhi kunye nokudibanisa nerejista yokunika ingxelo.
Liza kunika ntoni yaye kubani? Umsebenzisi wokugqibela wobhaliso uya kuba nokufikelela kulwazi malunga nemihla ecwangcisiweyo kunye neyokwenene yokulungela ingxelo, iziphumo zokutshekishwa okugqityiweyo nge-dynamics, kunye nolwazi kwimithombo elayishwe kwingxelo.
Kuthi, imodyuli yomgangatho wedatha edityaniswe kwiinkqubo zethu zomsebenzi yile:
- Ukwenziwa ngokukhawuleza kwezinto ezilindelwe ngabathengi.
- Ukwenza izigqibo malunga nokusetyenziswa ngakumbi kwedatha.
- Ukufumana iseti yokuqala yamanqaku eengxaki kumanqanaba okuqala omsebenzi wophuhliso lolawulo lomgangatho rhoqo.
Ngokuqinisekileyo, la ngamanyathelo okuqala ekwakheni inkqubo yokulawula idatha epheleleyo. Kodwa siqinisekile ukuba kuphela ngokwenza lo msebenzi ngenjongo kuphela, ukwazisa ngenkuthalo izixhobo zoLawulo lweDatha kwinkqubo yomsebenzi, siya kubonelela abathengi bethu ngomxholo wolwazi, inqanaba eliphezulu lokuthembela kwidatha, ukungafihli ekufumaneni kwabo kunye nokwandisa isantya sokuqalisa. umsebenzi omtsha.
Iqela le-DataOffice
umthombo: www.habr.com