Injineli yedatha okanye kufa: ibali lomphuhlisi omnye

Ekuqaleni kukaDisemba, ndenza impazamo enkulu kwaye ndenza utshintsho ebomini bam njengomphuhlisi kwaye ndafudukela kwiqela lobuNjineli beDatha (DE) ngaphakathi kwenkampani. Kweli nqaku ndiza kubelana ngemigqaliselo endiyenzileyo kwiinyanga ezimbini zokusebenza kwiqela le-DE.

Kutheni iNjineli yeDatha?

Uhambo lwam oluya e-DE lwaqala ngehlobo lika-2019, xa thina Xneg masiye ku Isikolo sekhompyutha esasazwayo, yaye apho ndafumana ukukhanyiselwa. Ndaqala ukuba nomdla kwisihloko, ndifunda ii-algorithms kunye nangazo bhala, kwaye emva koko wacinga malunga nobubanzi besicelo kwaye wafumanisa ngokukhawuleza ukuba isicelo esisebenzayo kwinkampani yethu sisasazwa i-database.

Lenza ntoni kanye kanye iqela lethu? Thina, njengawo onke amakhwenkwe kunye namantombazana asefashonini, sifuna ukuba yiNkampani yeDatha. Kwaye ukuze oku kwenzeke, kufuneka ubuncinane sakhe indawo yokugcina ethembekileyo, engasetyenziselwa ukwakha naziphi na iingxelo ezifunwa yinkampani. Kodwa eyona nto ibalulekileyo kukuba idatha kule ndawo yokugcina kufuneka uthenjwe. Ngaphezu koko, usebenzisa ezi data, kufuneka ukwazi ukubuyisela imeko yenkqubo ngexesha t. Konke oku kuyinkimbinkimbi kukuba siphila kwihlabathi elitsha elinesibindi le-microservices, kwaye le ngcamango ithetha ukuba inkonzo nganye isebenzisa umsebenzi wayo omncinci, i-database yayo yishishini layo, kwaye inokucima ubuncinane yonke imihla, kodwa kwangelo xesha linye kufuneka sikwazi ukufumana kwaye sisebenzisane nemeko yenkonzo.

Ukuba ufuna ukuqhutywa yiDatha, qala ube nguMsitho oqhutywayo

Ayilula kangako. Iziganeko zahlukile, kwaye umphuhlisi kunye nonjiniyela wedatha bazijonga ngokwahlukileyo. Ukuthetha ngeziganeko sisihloko senqaku elahlukileyo, ngoko andiyi kungena kulo apha. Ukongeza, inqaku elinjalo sele likhona wabhala uMartin Fowler othile, andisayi kumhlutha laurels, naye makadume.

Ngokubanzi, kuninzi ukucinga ngako kwaye kungenxa yoko le ndawo inomtsalane. Kwenzeka njalo ukuba kwinkampani yethu, iNjineli yeDatha yindawo ebanzi kakhulu yoxanduva kunomntu obhala imibhobho ye-ETL/ELT (ukuba awuyazi ukuba zithetha ukuthini ezi zishunqulelo, yiza dibana. Njengentengiso yeemeko).

Sijongana nokugcinwa kwezakhiwo, imodeli yedatha, imiba enxulumene nokhuseleko lwedatha, kunye nemibhobho ngokwayo, kunjalo. Kukwafuneka siqinisekise ukuba, kwelinye icala, ubukho bethu abukho bunzima kakhulu kubaphuhlisi bemveliso kwaye kufuneka baphazamiseke kancinci ngokweemfuno zethu xa sisika izinto ezintsha kwinkqubo, kwaye kwelinye icala, thina kufuneka babonelele ngokufanelekileyo kwidatha yokugcina abahlalutyi kunye neqela le-BI. Siphila ngolo hlobo.

Ubunzima xa usuka kuphuhliso

Ngosuku lwam lokuqala lomsebenzi, ndiye ndadibana nobunzima obuninzi endifuna ukwabelana nawe.

1. Into yokuqala endayibonayo kukungabikho kwe-tuling kunye nezenzo ezithile. Thatha, umzekelo, ukugubungela ikhowudi kunye neemvavanyo. Sinamakhulu eenkqubo zovavanyo kuphuhliso. Xa usebenza ngedatha, yonke into inzima ngakumbi. Ewe, sinokuvavanya imibhobho ye-ETL kwidatha yovavanyo, kodwa kufuneka siyenze yonke ngesandla kwaye sijonge izisombululo kwimeko nganye ethile. Ngenxa yoko, ukhuselo lovavanyo lubi kakhulu. Ngethamsanqa, kukho omnye umaleko wengxelo ngendlela yokubeka iliso kunye neelog, kodwa oku sele kufuna ukuba siphendule ngokuphendulayo kunokuba siphendule, nto leyo ecaphukisayo kwaye ingoyiki.

2. Ihlabathi elivela kumbono we-DE alikho konke oko libonakala kumphuhlisi wemveliso oqhelekileyo (kakuhle, ngokuqinisekileyo umfundi akanjalo, kwaye sele eyazi yonke into, kodwa andizange ndiyazi kwaye ngoku ndiphosa. phezulu). Njengomphuhlisi, ndidala eyam i-microservice, ndibeke idatha kwi [database yokhetho lwakho], gcina ilizwe lam apho, fumana into nge-ID kwaye kulungile. Inkonzo iyacotha, iiodolo ziyabhidisa, kuphelele apho. Bandicela ukuba ndikhangele imeko yam kwenye inkonzo, ngoko ndiya kuphosa isiganeko kwi-RabbitMQ ethile kwaye yiloo nto. Kwaye apha siphinde sabuyela kumbandela weziganeko ezichazwe ngasentla.

Oko inkonzo idinga umsebenzi wokusebenza ayihambelani nathi kwiinkcukacha zembali, ngoko ke umbuzo wokuphinda usebenze iikontraki zenkonzo kunye nomsebenzi osondeleyo kunye namaqela ophuhliso uqala. Awunakukwazi ukucinga ukuba zingaphi iiyure ezisithathileyo ukuba sivumelane: luhlobo luni loMnyhadala oqhutywayo kwinkampani yethu.

3. Kufuneka ucinge ngentloko yakho. Hayi, andithethi ukuba abaphuhlisi abacingi (nangona ndingubani ukuba ndithethele wonke umntu), yinto nje yokuba kuphuhliso lwemveliso kaninzi sele unohlobo oluthile lolwakhiwo, kwaye usike iishuffles ezahlukeneyo ukusuka kumqolo osemva. Ngokuqinisekileyo, oku kufuna ukucwangciswa kunye nokucinga, kodwa lo ngumsebenzi womlambo, apho ingxaki ephambili kukwenza nje kakuhle nangempumelelo.

Kithina, akulula kangako kuba ukuhanjiswa kwezinto ezahlukeneyo zenkqubo ukusuka kwi-monolith efudumeleyo nepholileyo ukuya kwihlabathi lehlathi le-microservice yasendle ayilula kangako. Xa inkonzo iqala ukukhupha iziganeko, kufuneka uphinde uhlolisise ingqiqo yokuzalisa ukugcinwa, kuba idatha ngoku ibonakala ihluke. Kulapho kufuneka ucinge kakhulu kwaye ngokucokisekileyo, awusekho njengomphuhlisi, kodwa njengenjineli yedatha. Libali eliqhelekileyo xa uchitha iintsuku kunye ne-notebook kunye nosiba okanye nge-marker ebhodini. Kunzima kakhulu, andithandi ukucinga, ndiyayithanda imveliso.

4. Mhlawumbi eyona nto ibalulekileyo lulwazi. Senza ntoni xa singenalwazi? Ngubani othe stackoverflow? Mkhuphe lomntu egumbini. Siya kufunda amaxwebhu, iincwadi ngesihloko, kwaye kukho noluntu oluququzelela iiforamu, iintlanganiso kunye neenkomfa. Amaxwebhu alungile, kodwa ngelishwa, anokungaphelelanga. Sisebenzisa iCosmos DB kwinani leeprojekthi. Ithamsanqa lokufunda amaxwebhu ale mveliso. Iincwadi kuphela kosindiso; ngethamsanqa, zikho kwaye zinokufunyanwa, ziqulethe ulwazi oluninzi olusisiseko kwaye kufuneka ufunde kakhulu kwaye rhoqo. Kodwa ingxaki ikuluntu.

Ngoku kunzima ukufumana ubuncinci inkomfa enye eyaneleyo okanye indibano kwindawo yethu. Hayi, kunjalo, zininzi iindibano ezinegama elithi Data, kodwa ecaleni kweli gama kukho izifinyezo ezingaqhelekanga njengeML okanye i-AI. Ngoko, oku akusiyo yethu, sithetha malunga nendlela yokwakha izibonelelo zokugcina, kwaye kungekhona indlela yokuzigcoba nge-neurons. Ezi hipsters zithathe yonke into. Ngenxa yoko, asinalo uluntu. Ngendlela, ukuba uyiNjineli yeDatha kwaye uyazi uluntu oluhle, nceda ubhale kwizimvo.

Isiphelo kunye nesibhengezo sendibano

Sigqiba ngantoni? Amava am okuqala andixelela ukuba ukuziva kwizihlangu zenjineli yedatha kuya kuba luncedo kuye wonke umphuhlisi. Ivumela nje ukuba sijonge izinto ngendlela eyahlukileyo kwaye singamangalisi xa amehlo ethu efumana igazi xa sibona indlela abaphuhlisi abaphatha ngayo idatha yabo. Ke, ukuba kukho i-DE kwinkampani yakho, thetha nje naba bafana, uya kufunda izinto ezininzi ezintsha (malunga nawe).

Kwaye ekugqibeleni, isaziso. Kuba kunzima ukufumana ukudibana kwisihloko sethu emini, sagqiba ekubeni senze ezethu. Kutheni si worse? Ngethamsanqa sinento emangalisayo Schvepsss kunye nabahlobo bethu ukusuka ILebhu yeeNzululwazi eziNtsha, abathi, njengathi, bacinge ukuba iinjineli zedatha zivinjwa ngokungafanelekanga ingqalelo.

Ndithatha eli thuba, ndimema wonke umntu onenkathalo ukuba eze kwindibano yethu yokuqala yoluntu enesihloko esithembisayo esithi “DE okanye DIE”, eya kuqhubeka nge-27.02.2020 kaFebruwari XNUMX kwiofisi yaseDodo Pizza. Iinkcukacha ku TimePad.

Ukuba kukho nantoni na eyenzekayo, ndiya kuba lapho, ungandixelela ngokobuqu ebusweni bam ukuba ndiphosakele kangakanani ngabaphuhlisi.

umthombo: www.habr.com