Pa Habr!
Deta ndi chinthu chamtengo wapatali kwambiri chakampani. Pafupifupi kampani iliyonse yomwe ili ndi digito imalengeza izi. Ndizovuta kutsutsana ndi izi: palibe msonkhano waukulu wa IT womwe umachitikira popanda kukambirana njira zoyendetsera, kusunga ndi kukonza deta.
Deta imabwera kwa ife kuchokera kunja, imapangidwanso mkati mwa kampaniyo, ndipo ngati tilankhula za data kuchokera ku kampani ya telecom, ndiye kwa ogwira ntchito mkati iyi ndi nyumba yosungiramo zambiri za kasitomala, zomwe amakonda, zizolowezi zake, ndi malo. Ndi mbiri yoyenera ndi magawo, zotsatsa ndizothandiza kwambiri. Komabe, muzochita, si zonse zomwe zimakhala zabwino kwambiri. Zomwe makampani amasunga zitha kukhala zachikale, zosafunikira, zobwerezabwereza, kapena kukhalapo kwake sikudziwika kwa wina aliyense kupatula ogwiritsa ntchito ochepa. ¯_(ツ)_/¯
Mwachidule, deta iyenera kuyendetsedwa bwino - pokhapokha ngati idzakhala chuma chomwe chimabweretsa phindu lenileni ndi phindu ku bizinesi. Tsoka ilo, kuthetsa nkhani zowongolera deta kumafuna kuthana ndi zovuta zambiri. Zimakhala makamaka chifukwa cha mbiri yakale monga "zoo" za machitidwe ndi kusowa kwa njira zogwirizanitsa ndi njira zoyendetsera kasamalidwe kawo. Koma kodi kukhala “data driven” kumatanthauza chiyani?
Izi ndi zomwe tikambirana mwatsatanetsatane, komanso momwe opensource stack idatithandizira.
Lingaliro la Strategic Data Management Data Governance (DG) limadziwika kale pamsika waku Russia, ndipo zolinga zomwe bizinesi zimakwaniritsa chifukwa cha kukhazikitsidwa kwake zimamveka bwino komanso zimafotokozedwa momveka bwino. Kampani yathu inalinso chimodzimodzi ndipo idadziyika yokha ntchito yoyambitsa lingaliro la kasamalidwe ka data.
Ndiye tinayambira kuti? Poyamba, tinadzipangira zolinga zazikulu:
- Sungani deta yathu kupezeka.
- Onetsetsani kuwonetsetsa kwa moyo wa data.
- Perekani kwa ogwiritsa ntchito akampani deta yokhazikika, yosasinthika.
- Patsani ogwiritsa ntchito akampani ndi data yotsimikizika.
Masiku ano, pali zida khumi ndi ziwiri za Ulamuliro wa Data pamsika wa mapulogalamu.
Koma titaunika mwatsatanetsatane ndikuwunika mayankho, tidalemba ndemanga zingapo zofunika kwa ife tokha:
- Opanga ambiri amapereka mayankho athunthu, omwe kwa ife ndi ofunikira ndipo amabwereza magwiridwe antchito omwe alipo. Kuphatikiza apo, okwera mtengo pankhani yazachuma, kuphatikiza mu mawonekedwe amakono a IT.
- Magwiridwe ake ndi mawonekedwe adapangidwira akatswiri aukadaulo, osati ogwiritsa ntchito mabizinesi.
- Kutsika kwapang'onopang'ono kwazinthu komanso kusowa kochita bwino pamsika waku Russia.
- Mtengo wapamwamba wa mapulogalamu ndi chithandizo china.
Njira ndi malingaliro omwe adanenedwa pamwambapa okhudzana ndi kulowetsa mapulogalamu amakampani aku Russia m'malo mwa makampani aku Russia zidatipangitsa kuti tipite patsogolo pathu pamasamba otseguka. Pulatifomu yomwe tidasankha inali Django, dongosolo laulere komanso lotseguka lolembedwa mu Python. Chifukwa chake tazindikira ma module ofunikira omwe angathandizire ku zolinga zomwe tazitchula pamwambapa:
- Kaundula wa malipoti.
- Bizinesi glossary.
- Module yofotokozera kusintha kwaukadaulo.
- Module yofotokozera mayendedwe a moyo wa data kuchokera kugwero kupita ku chida cha BI.
- Module yowongolera khalidwe la data.
Kaundula wa malipoti
Malinga ndi zotsatira za maphunziro amkati m'makampani akuluakulu, pothetsa mavuto okhudzana ndi deta, ogwira ntchito amathera 40-80% ya nthawi yawo kufunafuna. Chifukwa chake, tadziyika tokha ntchito yopanga zidziwitso zomveka za malipoti omwe analipo kale omwe anali kupezeka kwa makasitomala okha. Chifukwa chake, timachepetsa nthawi yopangira malipoti atsopano ndikuwonetsetsa kuti deta ya demokarasi.
Kaundula wamalipoti wakhala zenera limodzi loperekera malipoti kwa ogwiritsa ntchito mkati kuchokera kumadera osiyanasiyana, madipatimenti, ndi magawo. Imaphatikiza zidziwitso zamagwiritsidwe azidziwitso opangidwa m'malo angapo akampani, ndipo ambiri mwa iwo ali ku Rostelecom.
Koma kaundula si mndandanda wouma wa malipoti opangidwa. Pa lipoti lililonse, timapereka chidziwitso chofunikira kuti wogwiritsa alidziwe bwino:
- kufotokozera mwachidule za lipoti;
- kuya kwa kupezeka kwa deta;
- gawo lamakasitomala;
- chida chowonera;
- dzina la malo osungirako makampani;
- zofunikira zabizinesi;
- kulumikizana ndi lipoti;
- ulalo wofunsira mwayi wopeza;
- kukhazikitsa udindo.
Ma analytics a kagwiritsidwe ntchito amapezeka pamalipoti, ndipo malipoti ali pamwamba pa mndandanda kutengera zolemba za log kutengera kuchuluka kwa ogwiritsa ntchito apadera. Ndipo si zimenezo. Kuphatikiza pa mawonekedwe wamba, taperekanso tsatanetsatane wa momwe malipoti amapangidwira ndi zitsanzo zamakhalidwe ndi njira zowerengera. Kufotokozera koteroko nthawi yomweyo kumapereka yankho kwa wogwiritsa ntchito ngati lipotilo ndi lothandiza kwa iye kapena ayi.
Kupanga gawoli kunali gawo lofunika kwambiri pa demokalase ya data ndikuchepetsa kwambiri nthawi yomwe imatenga kuti mupeze chidziwitso chofunikira. Kuwonjezera pa kuchepetsa nthawi yosaka, chiwerengero cha zopempha ku gulu lothandizira kuti apereke zokambirana zachepanso. Ndikosatheka kuti tisazindikire chotsatira china chothandiza chomwe tidapeza popanga kaundula wogwirizana wa malipoti - kuletsa kupangidwa kwa malipoti obwereza a magawo osiyanasiyana.
Bizinesi glossary
Nonse mukudziwa kuti ngakhale mkati mwa kampani imodzi, mabizinesi amalankhula zilankhulo zosiyanasiyana. Inde, amagwiritsa ntchito mawu ofanana, koma amatanthauza zinthu zosiyana kwambiri. Kalozera wabizinesi adapangidwa kuti athetse vutoli.
Kwa ife, glossary yabizinesi si buku lofotokozera lomwe limafotokoza mawu ndi njira zowerengera. Awa ndi malo athunthu opangira, kuvomereza ndi kuvomereza mawu, kupanga ubale pakati pa mawu ndi zidziwitso zina zakampani. Musanalowe muzolemba zamabizinesi, mawu akuyenera kudutsa magawo onse ovomerezeka ndi makasitomala abizinesi ndi malo opangira data. Pokhapokha izi zimayamba kupezeka kuti zigwiritsidwe ntchito.
Monga ndalembera pamwambapa, chodabwitsa cha chida ichi ndikuti chimalola kulumikizana kuchokera pamlingo wa nthawi yamalonda kupita ku malipoti apadera a ogwiritsa ntchito momwe amagwiritsidwira ntchito, komanso pamlingo wazinthu zama database.
Izi zimatheka pogwiritsa ntchito zizindikiritso za glossary term kufotokoza mwatsatanetsatane malipoti a registry komanso kufotokozera zinthu zankhokwe.
Pakali pano, mawu oposa 4000 afotokozedwa ndikuvomerezedwa mu Glossary. Kugwiritsa ntchito kwake kumathandizira komanso kufulumizitsa kukonzedwa kwa zopempha zomwe zikubwera kuti zisinthidwe pamakina amakampani. Ngati chizindikiro chofunikira chakhazikitsidwa kale mu lipoti lililonse, ndiye kuti wogwiritsa ntchito awona nthawi yomweyo malipoti okonzeka pomwe chizindikirochi chikugwiritsidwa ntchito, ndipo azitha kusankha momwe angagwiritsire ntchito bwino ntchito zomwe zilipo kapena kusinthidwa kwake pang'ono, popanda kuyambitsa. zopempha zatsopano za chitukuko cha lipoti latsopano.
Module yofotokozera kusintha kwaukadaulo ndi DataLineage
Kodi ma module awa ndi chiyani, mukufunsa? Sikokwanira kungogwiritsa ntchito Report Register ndi Glossary; ndikofunikiranso kuyika mawu onse abizinesi pamtundu wa database. Chifukwa chake, tidatha kumaliza njira yopangira mayendedwe amoyo wa data kuchokera kumagwero oyambira kupita kumawonekedwe a BI kudzera m'magawo onse osungiramo data. Mwanjira ina, pangani DataLineage.
Tinapanga mawonekedwe otengera mawonekedwe omwe amagwiritsidwa ntchito kale mu kampani pofotokozera malamulo ndi malingaliro akusintha kwa data. Zomwezo zimalowetsedwa kudzera mu mawonekedwe monga kale, koma tanthauzo la mawu oti chizindikiritso kuchokera ku glossary ya bizinesi lakhala chofunikira. Umu ndi momwe timapangira mgwirizano pakati pa bizinesi ndi zigawo zakuthupi.
Ndani akuchifuna? Kodi cholakwika ndi chiyani ndi mawonekedwe akale omwe mudagwira nawo ntchito kwa zaka zingapo? Kodi ndalama zogwirira ntchito zakwera bwanji? Tinayenera kuthana ndi mafunso oterowo panthawi yogwiritsira ntchito chida. Mayankho apa ndi osavuta - tonse timafunikira izi, ofesi yamakampani yathu komanso ogwiritsa ntchito.
Zowonadi, ogwira ntchitowo adayenera kuzolowera; poyamba, izi zidapangitsa kuti ndalama zogwirira ntchito ziwonjezeke pang'ono pokonzekera zolemba, koma tidakonza nkhaniyi. Kuchita, kuzindikira ndi kukhathamiritsa madera ovuta achita ntchito yawo. Takwaniritsa chinthu chachikulu - tawongolera zofunikira zomwe zapangidwa. Minda yovomerezeka, mabuku olumikizana ogwirizana, masks olowera, macheke omangidwa - zonsezi zidapangitsa kuti zitheke kupititsa patsogolo kufotokozera kwakusintha. Tinachoka ku chizoloŵezi chopereka zolemba monga zofunikira zachitukuko ndikugawana chidziwitso chomwe chinalipo kokha ku gulu lachitukuko. Dongosolo la metadata lopangidwa limachepetsa kwambiri nthawi yomwe ikufunika kuti ipange kusanthula kwa regression ndipo imapereka kuthekera kowunika mwachangu zomwe zasintha pagawo lililonse la IT landscape (malipoti owonetsera, ma aggregates, magwero).
Kodi izi zikukhudzana bwanji ndi ogwiritsa ntchito wamba amalipoti, zabwino zake ndi zotani? Chifukwa cha luso lopanga DataLineage, ogwiritsa ntchito athu, ngakhale omwe ali kutali ndi SQL ndi zilankhulo zina zamapulogalamu, amalandira mwamsanga zambiri zokhudza magwero ndi zinthu zomwe zimachokera ku lipoti linalake.
Data Quality Control Module
Chilichonse chomwe tidakambirana pamwambapa powonetsetsa kuti deta ikuwonekera momveka bwino sizofunikira popanda kumvetsetsa kuti zomwe timapereka kwa ogwiritsa ntchito ndizolondola. Chimodzi mwamagawo ofunikira a lingaliro lathu la Ulamuliro wa data ndi gawo lowongolera za data.
Pakali pano, iyi ndi kalozera wa macheke a mabungwe osankhidwa. Cholinga chaposachedwa pakupanga zinthu ndikukulitsa mndandanda wamacheke ndikuphatikiza ndi kaundula wa malipoti.
Idzapereka chiyani ndipo kwa ndani? Wogwiritsa ntchito kumapeto kwa registry adzakhala ndi mwayi wodziwa zambiri zamasiku omwe adakonzedwa komanso enieni akukonzekera lipoti, zotsatira za cheke chomalizidwa ndi mphamvu, ndi chidziwitso pa magwero omwe ali mu lipotilo.
Kwa ife, gawo lamtundu wa data lophatikizidwa munjira zathu zantchito ndi:
- Kupanga mwachangu zomwe makasitomala amayembekeza.
- Kupanga zisankho pakugwiritsanso ntchito deta.
- Kupeza zoyambira zamavuto pazoyambira zoyambira ntchito kuti pakhale zowongolera zamakhalidwe.
Inde, awa ndi masitepe oyambirira pomanga ndondomeko yoyendetsera deta yonse. Koma tili ndi chidaliro kuti pokhapokha pochita ntchitoyi mwadala, kuyambitsa mwachangu zida za Ulamuliro wa Data pakugwira ntchito, tidzapatsa makasitomala athu zomwe zili ndi chidziwitso, kudalira kwambiri deta, kuwonekera polandila ndikuwonjezera liwiro loyambitsa. ntchito zatsopano.
Gulu la DataOffice
Source: www.habr.com