Sawubona Habr!
Idatha iyimpahla ebaluleke kakhulu yenkampani. Cishe yonke inkampani egxile kwidijithali imemezela lokhu. Kunzima ukuphikisana nalokhu: ayikho ingqungquthela eyodwa enkulu ye-IT ebanjwayo ngaphandle kokuxoxa ngezindlela zokuphatha, ukugcina nokucubungula idatha.
Idatha iza kithi ivela ngaphandle, ibuye yenziwe ngaphakathi kwenkampani, futhi uma sikhuluma ngedatha evela enkampanini ye-telecom, khona-ke kubasebenzi bangaphakathi lokhu kuyinqolobane yolwazi mayelana neklayenti, izithakazelo zakhe, imikhuba, nendawo. Ngokwenza iphrofayela okufanele nokuhlukaniswa, okunikezwayo kokukhangisa kusebenza kahle kakhulu. Kodwa-ke, ekusebenzeni, akuyona yonke into enhle kakhulu. Idatha egcinwa yizinkampani ingase iphelelwe yisikhathi ngokungenathemba, iphelelwe isikhathi, iphindaphindeke, noma ukuba khona kwayo akwaziwa yinoma ubani ngaphandle kwendingiliza encane yabasebenzisi. ¯_(ツ)_/¯
Ngamafuphi nje, idatha kufanele iphathwe ngempumelelo - yilapho kuphela lapho izoba yimpahla eletha izinzuzo zangempela kanye nenzuzo ebhizinisini. Ngeshwa, ukuxazulula izinkinga zokuphathwa kwedatha kudinga ukunqoba izinto eziningi eziyinkimbinkimbi. Ngokuyinhloko kungenxa yakho kokubili ifa elingokomlando elisesimweni “sama-zoo” wezinhlelo kanye nokuntuleka kwezinqubo ezihlangene nezindlela zokuphatha kwazo. Kodwa kusho ukuthini “ukuqhutshwa kwedatha”?
Yilokhu kanye esizokhuluma ngakho ngaphansi kokusikwa, kanye nendlela isitaki se-opensource esisisize ngayo.
Umqondo wokuphathwa kwedatha yamasu we-Data Governance (DG) usuvele waziwa kahle emakethe yaseRussia, futhi imigomo efinyelelwe yibhizinisi ngenxa yokuqaliswa kwayo icacile futhi imenyezelwe ngokucacile. Inkampani yethu ayizange ihluke futhi izibekele umsebenzi wokwethula umqondo wokuphathwa kwedatha.
Pho siqale ngaphi? Okokuqala, sizakhele imigomo ebalulekile:
- Gcina idatha yethu ifinyeleleka.
- Qinisekisa ukuba sobala komjikelezo wempilo wedatha.
- Nikeza abasebenzisi benkampani ngedatha engaguquki, engaguquki.
- Nikeza abasebenzisi benkampani idatha eqinisekisiwe.
Namuhla, kunamathuluzi ekilasi Lokubusa Kwedatha ayishumi nambili emakethe yesoftware.
Kodwa ngemva kokuhlaziya okuningiliziwe kanye nocwaningo lwezixazululo, sizibhalele inqwaba yamazwana abalulekile:
- Abakhiqizi abaningi banikeza isethi ebanzi yezixazululo, kithina engenamsebenzi futhi ephinda ukusebenza okukhona. Futhi, kuyabiza ngokwezinsiza, ukuhlanganiswa endaweni yamanje ye-IT.
- Ukusebenza nesixhumi esibonakalayo sakhelwe ochwepheshe, hhayi abasebenzisi bokugcina bebhizinisi.
- Izinga lokusinda eliphansi lemikhiqizo kanye nokuntuleka kokuqaliswa okuphumelelayo emakethe yaseRussia.
- Izindleko eziphezulu zesofthiwe nokusekelwa okwengeziwe.
Imibandela nezincomo ezivezwe ngenhla mayelana nokufakwa esikhundleni kwesofthiwe yezinkampani zaseRussia zisiqinisekise ukuthi siqhubekele ekuthuthukisweni kwethu esitakini somthombo ovulekile. Inkundla esiyikhethile kwakuyi-Django, uhlaka lomthombo wamahhala noluvulekile olubhalwe ngePython. Ngakho-ke sihlonze amamojula abalulekile azofaka isandla ezinhlosweni ezishiwo ngenhla:
- Irejista yemibiko.
- Iglosari yebhizinisi.
- Imojula yokuchaza izinguquko zobuchwepheshe.
- Imojula yokuchaza umjikelezo wempilo yedatha ukusuka emthonjeni kuya ethuluzini le-BI.
- Imojuli yokulawula ikhwalithi yedatha.
Irejista yemibiko
Ngokwemiphumela yezifundo zangaphakathi ezinkampanini ezinkulu, lapho kuxazululwa izinkinga ezihlobene nedatha, abasebenzi bachitha u-40-80% wesikhathi sabo bezifuna. Ngakho-ke, sizibekele umsebenzi wokwenza ulwazi oluvulekile mayelana nemibiko ekhona ngaphambili ebitholakala kumakhasimende kuphela. Ngakho, sinciphisa isikhathi sokukhiqiza imibiko emisha futhi siqinisekise ukwenziwa kwedatha yeningi.
Irejista yokubika isiphenduke iwindi elilodwa lokubika labasebenzisi bangaphakathi abavela ezifundeni ezihlukahlukene, iminyango, nezigaba. Ihlanganisa ulwazi lwezinsizakalo zolwazi ezidalwe kumakhosombe ezinkampani eziningana zenkampani, futhi ziningi zazo e-Rostelecom.
Kodwa ukubhalisa akulona nje uhlu olomile lwemibiko ethuthukisiwe. Kumbiko ngamunye, sihlinzeka ngolwazi oludingekayo ukuze umsebenzisi azijwayeze ngawo:
- incazelo emfushane yombiko;
- ukujula kokutholakala kwedatha;
- ingxenye yamakhasimende;
- ithuluzi lokubuka;
- igama lendawo yokugcina impahla;
- izidingo zokusebenza kwebhizinisi;
- isixhumanisi sombiko;
- isixhumanisi sesicelo sokufinyelela;
- isimo sokuqalisa.
Izibalo zeleveli yokusetshenziswa ziyatholakala kumibiko, futhi imibiko ibalwa phezulu ohlwini ngokusekelwe kuzibalo zelogi ngokusekelwe enanini labasebenzisi abahlukile. Futhi akunjalo. Ngokungeziwe kuzici ezijwayelekile, siphinde sanikeza incazelo enemininingwane yokwakheka kwesibaluli semibiko enezibonelo zamanani nezindlela zokubala. Ukwaziswa okunjalo ngokushesha kunikeza umsebenzisi impendulo ukuthi umbiko uwusizo yini kuye noma cha.
Ukuthuthukiswa kwale mojula kwakuyisinyathelo esibalulekile ekusetshenzisweni kwedatha yeningi futhi kwanciphisa kakhulu isikhathi esisithathayo ukuthola ulwazi oludingekayo. Ngaphezu kokunciphisa isikhathi sokusesha, inani lezicelo ethimbeni losekelo ukuze linikeze ukubonisana nalo lehlile. Akwenzeki ukuthi singawuboni omunye umphumela owusizo esiwuzuzile ngokwakha irejista ehlanganisiwe yemibiko - ukuvimbela ukwakhiwa kwemibiko eyimpinda yamayunithi esakhiwo ahlukene.
Iglosari yebhizinisi
Niyazi nonke ukuthi ngisho enkampanini eyodwa, amabhizinisi akhuluma izilimi ezahlukene. Yebo, basebenzisa amagama afanayo, kodwa asho izinto ezihluke ngokuphelele. Iglosari yebhizinisi yakhelwe ukuxazulula le nkinga.
Kithina, uhlu lwamagama ebhizinisi aluyona nje incwadi eyireferensi enencazelo yamagama nendlela yokubala. Lena indawo egcwele yokuthuthukisa, ukuvumelana nokugunyaza amagama, ukwakha ubudlelwano phakathi kwemigomo nezinye izimpahla zolwazi zenkampani. Ngaphambi kokungena kuhlu lwamagama ebhizinisi, igama kufanele lidlule kuzo zonke izigaba zokugunyazwa namakhasimende ebhizinisi kanye nesikhungo sekhwalithi yedatha. Kuphela ngemva kwalokhu lapho itholakala khona ukuze isetshenziswe.
Njengoba ngibhale ngenhla, okuhlukile kwaleli thuluzi ukuthi livumela ukuxhumana kusuka ezingeni lebhizinisi kuya emibikweni ethile yomsebenzisi lapho lisetshenziswa khona, kanye nezinga lezinto ezibonakalayo zesizindalwazi.
Lokhu kwenzeka ngokusetshenziswa kwezihlonzi zegama loluhlu lwamagama encazelweni enemininingwane yemibiko yokubhalisa kanye nencazelo yezinto ezibonakalayo zesizindalwazi.
Njengamanje, ngaphezu kwamagama angu-4000 achaziwe futhi kwavunyelwana ngawo oHlulweni Lwamagama. Ukusetshenziswa kwayo kwenza kube lula futhi kusheshise ukucutshungulwa kwezicelo ezingenayo zoshintsho ezinhlelweni zolwazi zenkampani. Uma inkomba edingekayo isivele isetshenziswa kunoma yimuphi umbiko, umsebenzisi uzobona ngokushesha isethi yemibiko esenziwe isilungile lapho kusetshenziswa khona le nkomba, futhi uzokwazi ukunquma ngokusetshenziswa kabusha okusebenzayo kokusebenza okukhona noma ukuguqulwa kwayo okuncane, ngaphandle kokuqalisa. izicelo ezintsha zokwenziwa kombiko omusha.
Imojula yokuchaza ukuguqulwa kwezobuchwepheshe kanye ne-DataLineage
Ayini la mamojula, uyabuza? Akwanele ukuvele usebenzise iRejista yoMbiko kanye noHlulo Lwamagama; kuyadingeka futhi ukugxilisa yonke imigomo yebhizinisi kumodeli yesizindalwazi esiphathekayo. Ngakho-ke, sikwazile ukuqedela inqubo yokwenza umjikelezo wempilo yedatha kusuka kumasistimu omthombo kuya ekubukeni kwe-BI kuzo zonke izendlalelo zenqolobane yedatha. Ngamanye amazwi, yakha i-DataLineage.
Senze isixhumi esibonakalayo esisuselwe kufomethi eyayisetshenziswe ngaphambilini enkampanini ukuchaza imithetho nokunengqondo kokuguqulwa kwedatha. Ulwazi olufanayo lufakwa ngesixhumi esibonakalayo njengangaphambili, kodwa incazelo yegama elithi isihlonzi kusukela kuhlu lwamagama ebhizinisi isibe imfuneko. Lena yindlela esakha ngayo ukuxhumana phakathi kwezendlalelo zebhizinisi nezomzimba.
Ubani oyidingayo? Yini eyayingalungile ngefomethi endala osebenze ngayo iminyaka eminingana? Zikhuphuke kangakanani izindleko zabasebenzi zokukhiqiza? Kwadingeka sibhekane nemibuzo enjalo ngesikhathi sokusetshenziswa kwethuluzi. Izimpendulo lapha zilula kakhulu - sonke siyakudinga lokhu, ihhovisi ledatha yenkampani yethu kanye nabasebenzisi bethu.
Ngempela, abasebenzi kwadingeka bazivumelanise nezimo; ekuqaleni, lokhu kwaholela ekwenyukeni okuncane kwezindleko zabasebenzi zokulungiselela imibhalo, kodwa salulungisa lolu daba. Ukuzijwayeza, ukuhlonza kanye nokwenza kahle izindawo zezinkinga zenze umsebenzi wazo. Sizuze into esemqoka - sithuthukise ikhwalithi yezidingo ezithuthukisiwe. Izinkambu eziphoqelekile, izincwadi eziyisithenjwa ezihlanganisiwe, imaski yokufaka, amasheke akhelwe ngaphakathi - konke lokhu kwenze kwaba nokwenzeka ukuthuthukisa kakhulu izinga lezincazelo zoguquko. Sayeka umkhuba wokunikeza izikripthi njengezidingo zokuthuthukiswa kanye nolwazi olwabiwe olwalutholakala kuphela ethimbeni lokuthuthukisa. Isizindalwazi esikhiqiziwe semethadatha sinciphisa kakhulu isikhathi esidingekayo sokwenza ukuhlaziya kokuhlehla futhi sinikeza ikhono lokuhlola ngokushesha umthelela wezinguquko kunoma yisiphi isendlalelo se-IT landscape (imibiko yokubonisa, izilinganiso, imithombo).
Lokhu kuhlangene ngani nabasebenzisi abajwayelekile bemibiko, yiziphi izinzuzo zabo? Ngenxa yekhono lokwakha i-DataLineage, abasebenzisi bethu, ngisho nalabo abakude ne-SQL nezinye izilimi zokuhlela, bathola ngokushesha ulwazi mayelana nemithombo nezinto ngesisekelo sokuthi umbiko othile ukhiqizwa.
Imojula Yokulawula Ikhwalithi Yedatha
Konke esikhulume ngakho ngenhla mayelana nokuqinisekisa ukubonisa ngale kwedatha akubalulekile ngaphandle kokuqonda ukuthi idatha esiyinikeza abasebenzisi ilungile. Elinye lamamojula abalulekile omqondo wethu Wokulawulwa Kwedatha imojula yokulawula ikhwalithi yedatha.
Esigabeni samanje, lena ikhathalogi yamasheke amabhizinisi akhethiwe. Umgomo osheshayo wokuthuthukiswa komkhiqizo uwukwandisa uhlu lwamasheke futhi uhlanganise nerejista yokubika.
Lizonika bani futhi kubani? Umsebenzisi wokugcina wokubhalisa uzokwazi ukufinyelela olwazini mayelana nezinsuku ezihleliwe nezangempela zokulungela umbiko, imiphumela yokuhlolwa okuqediwe okunamandla, kanye nolwazi ngemithombo elayishwe embikweni.
Kithina, imojula yekhwalithi yedatha ehlanganiswe ezinqubweni zethu zomsebenzi ithi:
- Ukwakhiwa ngokushesha kokulindelwe yikhasimende.
- Ukwenza izinqumo ngokuqhubeka nokusetshenziswa kwedatha.
- Ukuthola isethi yokuqala yamaphuzu ezinkinga ezigabeni zokuqala zomsebenzi ukuze kuthuthukiswe izilawuli zekhwalithi ezijwayelekile.
Yiqiniso, lezi yizinyathelo zokuqala zokwakha inqubo yokuphatha idatha egcwele. Kodwa siyaqiniseka ukuthi ngokwenza lo msebenzi ngamabomu kuphela, ngokwethula ngenkuthalo amathuluzi Okulawulwa Kwedatha enqubweni yomsebenzi, sizonikeza amakhasimende ethu okuqukethwe kolwazi, izinga eliphezulu lokuthembela kudatha, obala ekwamukeleni kwawo futhi sandise ijubane lokwethula. ukusebenza okusha.
Ithimba le-DataOffice
Source: www.habr.com