Ekuqaleni kukaZibandlela, ngenza iphutha elikhulu futhi ngenza inguquko empilweni yami njengomthuthukisi futhi ngathuthela ethimbeni Lobunjiniyela Bedatha (DE) ngaphakathi kwenkampani. Kulesi sihloko ngizokwabelana ngemibono engiye ngayenza phakathi nezinyanga ezimbili ngisebenza eqenjini le-DE.
Kungani Ubunjiniyela Bedatha?
Uhambo lwami oluya e-DE lwaqala ehlobo lika-2019, lapho thina asambeni siye , futhi lapho ngathola ukukhanyiselwa. Ngaqala ukuba nentshisekelo esihlokweni, ngifunda ama-algorithms ngisho nangawo , wabe esecabanga ngobubanzi besicelo futhi washeshe wathola ukuthi uhlelo lokusebenza olusebenzayo enkampanini yethu lusakazwa imininingwane egciniwe.
Iqembu lethu lenzani ngempela? Thina, njengabo bonke abafana namantombazane asemfashinini, sifuna ukuba Inkampani Eqhutshwa Ngedatha. Futhi ukuze lokhu kwenzeke, sidinga okungenani sakhe indawo yokugcina okuthembekile, engasetshenziswa ukwakha noma yimiphi imibiko edingwa yinkampani. Kodwa into ebaluleke kakhulu ukuthi idatha ekulesi sitoreji kufanele ithenjwe. Ngaphezu kwalokho, usebenzisa le datha, udinga ukwazi ukubuyisela isimo sesistimu ngesikhathi t. Konke lokhu kuyinkimbinkimbi yiqiniso lokuthi siphila ezweni elisha elinesibindi lama-microservices, futhi lo mbono ubonisa ukuthi insizakalo ngayinye isebenzisa ukusebenza kwayo okuncane, i-database yayo iyibhizinisi layo, futhi ingayisusa okungenani nsuku zonke, kodwa ngesikhathi esifanayo kufanele sikwazi ukwamukela nokucubungula isimo senkonzo.
Uma ufuna ukuqhutshwa ngedatha, qala ngokuqhutshwa komcimbi
Akulula kangako. Imicimbi ihlukile, futhi unjiniyela kanye nonjiniyela wedatha bayibheka ngendlela ehlukile. Ukukhuluma ngemicimbi kuyisihloko sendatshana ehlukile, ngakho-ke ngeke ngingene kukho lapha. Ngaphezu kwalokho, isihloko esinjalo sesivele senziwe uMartin Fowler othize, angeke ngimthathele udumo, naye makadume.
Ngokuvamile, kuningi okufanele ucabange ngakho futhi yingakho le ndawo ikhangayo. Kwenzeka lokhu nje enkampanini yethu, i-Data Engineer iyindawo ebanzi kakhulu yomthwalo wemfanelo kunomuntu obhala amapayipi e-ETL/ELT (uma ungazi ukuthi lezi zifinyezo zisho ukuthini, woza . Njengokukhangisa kokuqukethwe).
Sibhekana nezakhiwo zesitoreji, ukumodeliswa kwedatha, izinkinga ezihlobene nokuphepha kwedatha, kanye namapayipi ngokwawo, kunjalo. Kudingeka futhi siqinisekise ukuthi, ngakolunye uhlangothi, ukuba khona kwethu akuwona umthwalo omningi kubathuthukisi bomkhiqizo futhi kufanele baphazanyiswe kancane ngangokunokwenzeka ngezimfuneko zethu lapho sisika izici ezintsha kusistimu, futhi ngakolunye uhlangothi, thina kudingeka bazinikeze ezibekwe ngendlela efanele kudatha yokugcina yabahlaziyi kanye nethimba le-BI. Siphila kanjalo.
Ubunzima lapho usuka ekuthuthukisweni
Ngosuku lwami lokuqala lomsebenzi, ngihlangabezane nenani lobunzima engifuna ukwabelana nawe ngalo.
1. Into yokuqala engayibona kwakuwukungabikho kwe-tuling kanye nemikhuba ethile. Thatha, isibonelo, ukumbozwa kwekhodi ngezivivinyo. Sinamakhulu ezinhlaka zokuhlola ezisathuthukiswayo. Lapho usebenza ngedatha, yonke into iyinkimbinkimbi kakhulu. Yebo, singahlola amapayipi e-ETL kudatha yokuhlola, kodwa kufanele sikwenze konke mathupha futhi sibheke izixazululo zecala ngalinye elithile. Ngenxa yalokho, ukufakwa kokuhlolwa kubi kakhulu. Ngenhlanhla, kunolunye ungqimba lwempendulo ngendlela yokuqapha namalogi, kodwa lokhu kakade kudinga ukuthi sisabele ngokusabela kunokuba sithathe isinyathelo, okuyinto ecasulayo futhi ethusayo.
2. Umhlaba ngombono we-DE awukona neze lokho okubonakala kumthuthukisi womkhiqizo ojwayelekile (kahle, yiqiniso, umfundi akanjalo, futhi useyazi konke, kodwa bengingazi futhi manje ngishaya. it up). Njengonjiniyela, ngizakhela eyami i-microservice, ngibeke idatha [kusizindalwazi ozikhethele], londoloza isimo sami lapho, ngithole okuthile nge-ID futhi kulungile. Isevisi ihamba kancane, ama-oda ayadida, yilokho kuphela. Bangicela ukuthi ngibheke isimo sami kwenye inkonzo, ngakho-ke ngizophonsa umcimbi kwenye i-RabbitMQ futhi yilokho. Futhi lapha siphinde sabuyela endabeni yezenzakalo ezichazwe ngenhla.
Okudingwa yisevisi emsebenzini wokusebenza akuhambisani nathi ngedatha yomlando, ngakho-ke umbuzo wokusebenza kabusha kwezinkontileka zesevisi nomsebenzi wokuvala namathimba okuthuthukisa uyaqala. Awukwazi ngisho nokucabanga ukuthi mangaki amahora okusithathe ukuze sivumelane: hlobo luni lwe-Event Driven ayinkampani yethu.
3. Kudingeka ucabange ngekhanda lakho. Cha, angisho ukuthi onjiniyela abacabangi (yize ngingubani mina ukuthi ngingakhulumela wonke umuntu), ukuthi nje ekuthuthukisweni komkhiqizo kaningi usuvele unohlobo oluthile lwezakhiwo, futhi usika ukushova okuhlukile kokusalela emuva. Yiqiniso, lokhu kudinga ukuhlela nokucabanga, kodwa lona umsebenzi wokusakaza, lapho inkinga eyinhloko iwukwenza nje kahle nangempumelelo.
Kithina, akulula kangako ngoba ukudluliswa kwezingxenye zesistimu ezahlukahlukene kusuka ku-monolith efudumele nethokomele emhlabeni wehlathi elincane le-microservice akulula kangako. Lapho isevisi iqala ukukhafula izenzakalo, udinga ukucabangela kabusha i-logic yokugcwalisa isitoreji, ngoba idatha manje ibukeka ihlukile. Lapha yilapho udinga ukucabanga kakhulu futhi ngokucophelela, ungabe usasebenza njengonjiniyela, kodwa njengonjiniyela wedatha. Kuyindaba evamile uma uchitha izinsuku nencwajana yokubhala nepeni noma unomaka ebhodini. Kunzima kakhulu, angithandi ukucabanga, ngithanda ukukhiqiza futhi.
4. Mhlawumbe into ebaluleke kakhulu ulwazi. Senzenjani uma singenalo ulwazi? Ubani othe stackoverflow? Khipha lo muntu ekamelweni. Sihamba sifunda imibhalo, izincwadi ngesihloko, futhi kukhona nomphakathi ohlela izinkundla, imihlangano nezingqungquthela. Amadokhumenti mahle, kodwa ngeshwa, angeke aphelele. Sisebenzisa i-Cosmos DB kumaphrojekthi amaningi. Ngikufisela inhlanhla ngokufunda imibhalo yalo mkhiqizo. Izincwadi ziwukuphela kwensindiso; ngenhlanhla, zikhona futhi zingatholakala, ziqukethe ulwazi oluningi oluyisisekelo futhi kufanele ufunde kakhulu futhi njalo. Kodwa inkinga ikumphakathi.
Manje sekunzima ukuthola okungenani ingqungquthela eyodwa eyanele noma umhlangano endaweni yethu. Cha, kunjalo, kunenqwaba yokuhlangana negama elithi Data, kodwa eduze kwaleli gama kuvame ukuba nezifinyezo eziyinqaba njenge-ML noma i-AI. Ngakho-ke, lokhu akusikho okwethu, sikhuluma ngendlela yokwakha izindawo zokugcina, hhayi ukuthi singazigcoba kanjani ngama-neurons. Lawa ma-hipsters athathe yonke into. Ngenxa yalokho, asinawo umphakathi. Ngendlela, uma ungunjiniyela wedatha futhi wazi imiphakathi emihle, sicela ubhale kumazwana.
Iziphetho kanye nesimemezelo somhlangano
Siphetha ngani? Okuhlangenwe nakho kwami ββkokuqala kungitshela ukuthi umuzwa ezicathulweni zikanjiniyela wedatha uzoba usizo kuwo wonke unjiniyela. Kuvele kusivumele ukuthi sibheke izinto ngendlela ehlukile futhi singamangali lapho amehlo ethu eshaya igazi lapho sibona indlela onjiniyela abaphatha ngayo idatha yabo. Ngakho-ke, uma kukhona i-DE enkampanini yakho, vele ukhulume nalaba bafana, uzofunda izinto eziningi ezintsha (mayelana nawe).
Futhi ekugcineni, isimemezelo. Njengoba kunzima ukuthola imihlangano esihlokweni sethu phakathi nosuku, sanquma ukwenza okwethu. Kungani sibabi kakhulu? Ngenhlanhla sinesimangaliso nabangane bethu abavela , abafana nathi, abanomuzwa wokuthi onjiniyela bedatha bancishwa ukunakwa ngokungafanele.
Ngisebenzisa leli thuba, ngimema wonke umuntu okhathalelayo ukuba eze emhlanganweni wethu wokuqala womphakathi onesihloko esithembisayo esithi βDE or DIEβ, ozoba ngoFebhuwari 27.02.2020, XNUMX ehhovisi laseDodo Pizza. Imininingwane ku .
Uma kwenzeka noma yini, ngizobe ngilapho, ungangitshela mathupha ukuthi nginephutha kangakanani ngabathuthukisi.
Source: www.habr.com