Kumayambiriro kwa mwezi wa December, ndinalakwitsa kwambiri ndipo ndinasintha kwambiri moyo wanga monga wopanga mapulogalamu ndipo ndinasamukira ku gulu la Data Engineering (DE) mkati mwa kampaniyo. M'nkhaniyi ndigawana zomwe ndidaziwona m'miyezi iwiri ndikugwira ntchito mu gulu la DE.
Chifukwa chiyani Data Engineering?
Ulendo wanga wopita ku DE unayamba chilimwe cha 2019, pomwe ife tiyeni tipite , ndipo kumeneko ndinapeza chidziwitso. Ndinayamba kuchita chidwi ndi mutuwo, kuphunzira ma algorithms komanso za iwo , kenako ndinaganiza za kukula kwa ntchitoyo ndipo mwamsanga ndinazindikira kuti ntchito yothandiza pakampani yathu imagawidwa nkhokwe.
Kodi timu yathu imachita chiyani kwenikweni? Ife, monga anyamata ndi atsikana onse apamwamba, tikufuna kukhala Kampani Yoyendetsedwa ndi Data. Ndipo kuti izi zitheke, tifunika kumanga malo osungira odalirika, omwe angagwiritsidwe ntchito pomanga malipoti aliwonse omwe kampani ikufuna. Koma chofunika kwambiri ndi chakuti deta mu yosungirako izi ayenera kudaliridwa. Komanso, ntchito deta, muyenera kukhala wokhoza kubwezeretsa boma la dongosolo pa nthawi t. Zonsezi zimakhala zovuta chifukwa tikukhala m'dziko latsopano lolimba mtima la ma microservices, ndipo lingaliro ili likutanthauza kuti ntchito iliyonse imagwiritsa ntchito ntchito yake yaying'ono, nkhokwe yake ndi bizinesi yakeyake, ndipo ikhoza kuichotsa osachepera tsiku lililonse, koma nthawi yomweyo tiyenera kulandira ndi kukonza mmene utumiki.
Ngati mukufuna kukhala ndi Data Driven, choyamba khalani Event Driven
Osati mophweka. Zochitika ndizosiyana, ndipo wopanga ndi wopanga deta amaziwona mosiyana. Kulankhula za zochitika ndi mutu wankhani ina, kotero sindilowamo apa. Kuwonjezera apo, nkhani yotereyi yakhalapo kale Martin Fowler wina, ine sindidzamulanda zomkomera, msiyeni iyenso akhale wotchuka.
Kawirikawiri, pali zambiri zoti muganizire ndipo chifukwa chake derali ndi lokongola. Zimangochitika kuti m'kampani yathu, Data Engineer ndi gawo lalikulu kwambiri laudindo kuposa munthu amene amalemba mapaipi a ETL/ELT (ngati simukudziwa zomwe zidulezi zikutanthawuza, bwerani . Monga kutsatsa kwanthawi zonse).
Timathana ndi kamangidwe ka malo osungira, kutengera deta, nkhani zokhudzana ndi chitetezo cha data, ndi mapaipi omwe, ndithudi. Tiyeneranso kuwonetsetsa kuti, kumbali imodzi, kupezeka kwathu sikolemetsa kwambiri kwa opanga mankhwala ndipo ayenera kusokonezedwa pang'ono ndi zomwe tikufuna podula zatsopano mu dongosolo, ndipo kumbali ina, ife afunika kuwapatsa zoyikidwa bwino muzosunga zosungirako owunika ndi gulu la BI. Ndimo momwe timakhalira.
Zovuta pamene mukusintha kuchokera ku chitukuko
Pa tsiku langa loyamba la ntchito, ndinakumana ndi zovuta zingapo zomwe ndikufuna kugawana nanu.
1. Chinthu choyamba chimene ndinawona chinali kusowa kwa tuling ndi machitidwe ena. Tengani, mwachitsanzo, kuphimba ma code ndi mayeso. Tili ndi mazana a machitidwe oyesera omwe akutukuka. Mukamagwira ntchito ndi deta, zonse zimakhala zovuta kwambiri. Inde, titha kuyesa mapaipi a ETL pa data yoyeserera, koma tiyenera kuchita zonse pamanja ndikuyang'ana njira zothetsera vuto lililonse. Zotsatira zake, kufalikira kwa mayeso kumakhala koyipa kwambiri. Mwamwayi, palinso ndemanga ina monga kuyang'anira ndi zolemba, koma izi zimafuna kuti tizichitapo kanthu m'malo mochita chidwi, zomwe ndi zokwiyitsa komanso zosautsa.
2. Dziko lochokera ku DE kawonedwe kake silili momwe likuwonekera kwa wopanga mankhwala wamba (chabwino, ndithudi wowerenga sali choncho, ndipo amadziwa kale zonse, koma sindinadziwe ndipo tsopano ndikusokoneza. izo up). Monga wopanga mapulogalamu, ndimapanga microservice yanga, ndikuyika deta mu [database yomwe mwasankha], sungani dziko langa pamenepo, pezani china chake ndi ID ndipo zili bwino. Utumikiwu ndi wochedwa, madongosolo akusokoneza, ndizo zonse. Amandifunsa kuti ndiyang'ane dziko langa muutumiki wina, kotero ndiponyera chochitika mu RabbitMQ ina ndipo ndizomwezo. Ndipo apa tinabwereranso ku nkhani ya zochitika zomwe tafotokozazi.
Zomwe ntchitoyo ikufunikira pa ntchito yogwira ntchito sizikugwirizana ndi mbiri yakale, choncho funso la kukonzanso mgwirizano wautumiki ndi ntchito yotseka ndi magulu a chitukuko imayamba. Simungaganize kuti zidatitengera maola angati kuti tivomereze: ndi mtundu wanji wa Chochitika Choyendetsedwa ndi kampani yathu.
3. Muyenera kuganiza ndi mutu wanu. Ayi, sindikutanthauza kuti opanga sakuganiza (ngakhale kuti ine ndine ndani kuti ndilankhule kwa aliyense), kungoti pakukula kwazinthu nthawi zambiri mumakhala ndi zomanga zamtundu wina, ndipo mumadula masinthidwe osiyanasiyana kuchokera kumbuyo. Inde, izi zimafuna kukonzekera ndi kulingalira, koma iyi ndi ntchito yamtsinje, kumene vuto lalikulu ndikungochita bwino komanso moyenera.
Kwa ife, sizophweka chifukwa kusamutsa kwazinthu zosiyanasiyana zamakina kuchokera ku monolith yotentha komanso yabwino kupita kudziko lankhalango ya microservice sikophweka. Utumiki ukayamba kutulutsa zochitika, muyenera kuganiziranso malingaliro odzaza zosungirako, chifukwa deta tsopano ikuwoneka mosiyana. Apa ndipamene muyenera kuganiza mozama komanso mosamalitsa, osakhalanso ngati wopanga, koma ngati mainjiniya a data. Ndi nkhani yachilendo mukakhala masiku ndi cholembera ndi cholembera kapena cholembera pa bolodi. Ndizovuta kwambiri, sindimakonda kuganiza, ndimakondanso kupanga.
4. Mwina chinthu chofunika kwambiri ndi chidziwitso. Kodi timatani tikapanda kudziwa? Ndani adati stackoverflow? Chotsani munthu uyu mchipindamo. Timapita kukawerenga zikalata, mabuku pamutuwu, ndipo palinso gulu lomwe limapanga ma forum, misonkhano ndi misonkhano. Zolemba ndizabwino, koma mwatsoka, zitha kukhala zosakwanira. Timagwiritsa ntchito Cosmos DB pama projekiti angapo. Zabwino zonse powerenga zolembedwa za mankhwalawa. Mabuku ndiye chipulumutso chokha; mwamwayi, alipo ndipo atha kupezeka, ali ndi chidziwitso chochuluka ndipo muyenera kuwerenga kwambiri komanso mosalekeza. Koma vuto lili ndi anthu ammudzi.
Tsopano ndizovuta kupeza ngakhale msonkhano umodzi wokwanira kapena kukumana m'dera lathu. Ayi, ndithudi, pali zambiri zokumana nazo ndi mawu akuti Data, koma pafupi ndi mawu awa nthawi zambiri pamakhala zilembo zachilendo monga ML kapena AI. Kotero, izi siziri zathu, tikukamba za momwe tingamangire malo osungiramo zinthu, osati momwe tingadzipaka tokha ndi ma neuroni. Ma hipsters awa atenga chilichonse. Zotsatira zake, tilibe gulu. Mwa njira, ngati ndinu Data Engineer ndipo mukudziwa madera abwino, chonde lembani mu ndemanga.
Mapeto ndi kulengeza za msonkhano
Timaliza ndi chiyani? Chochitika changa choyamba chimandiuza kuti kumverera mu nsapato za injiniya wa data kudzakhala kothandiza kwa wopanga aliyense. Zimangotilola kuti tiyang'ane zinthu mosiyana ndipo tisadabwe pamene maso athu amawombera magazi tikawona momwe omanga amachitira deta yawo. Kotero, ngati pali DE mu kampani yanu, ingolankhulani ndi anyamatawa, muphunzira zinthu zambiri zatsopano (za inu nokha).
Ndipo potsiriza, kulengeza. Popeza zimakhala zovuta kupeza misonkhano pamutu wathu masana, tinaganiza zopanga zathu. Chifukwa chiyani tili oyipa? Mwamwayi tili ndi zodabwitsa ndi anzathu ochokera , omwe, monga ife, amaona kuti akatswiri opanga deta akumanidwa chidwi.
Potengera mwayiwu, ndikuitana aliyense amene amasamala kuti abwere kumsonkhano wathu woyamba wadera wokhala ndi mutu wolonjeza "DE kapena DIE", womwe udzachitike pa February 27.02.2020, XNUMX ku ofesi ya Dodo Pizza. Tsatanetsatane pa .
Chilichonse chikachitika, ndidzakhalapo, mutha kundiuza ndekha pamaso panga momwe ndikulakwitsa ponena za opanga.
Source: www.habr.com