Kushandiswa kweyakaderera-code mumapuratifomu ekuongorora

Vadiwa vaverengi, zuva rakanaka!

Iro basa rekuvaka IT mapuratifomu ekuunganidza uye kuongorora data nekukurumidza kana gare gare kunomuka kune chero kambani ine bhizinesi rakavakirwa pahungwaru yakarongedzerwa sevhisi modhi kana kugadzira zvigadzirwa zvakaomarara. Kuvaka mapuratifomu ekuongorora ibasa rakaoma uye rinotora nguva. Zvisinei, chero basa rinogona kurerutswa. Muchikamu chino ndinoda kugovera ruzivo rwangu mukushandisa maturusi e-low-code kubatsira kugadzira mhinduro dzekuongorora. Ichi chiitiko chakawanikwa panguva yekuitwa kwehuwandu hwezvirongwa muBig Data Solutions kutungamira kwekambani yeNeoflex. Kubva 2005, iyo Big Data Solutions kutungamira kweNeoflex yanga ichibata nenyaya dzekuvaka matura edhata nemadziva, kugadzirisa matambudziko ekugadzirisa kukurumidza kwekugadzirisa ruzivo uye kushanda panzira yekutarisira data data.

Hapana anozokwanisa kudzivirira kuunganidzwa kweruzivo rusina kusimba uye / kana rakanyatsorongeka data. Pamwe kunyangwe tiri kutaura nezvemabhizinesi madiki. Mushure mezvose, pakuyera bhizinesi, muzvinabhizimusi anovimbisa achasangana nenyaya dzekugadzira chirongwa chekuvimbika, anozoda kuongorora kushanda kwemapoinzi ekutengesa, achafunga nezve kwakanangwa kushambadzira, uye achashamiswa nekudiwa kwekuperekedza zvigadzirwa. . Kumwe kufungidzira kwekutanga, dambudziko rinogona kugadziriswa "pamabvi". Asi sezvo bhizinesi richikura, kuuya kune analytical platform kuchiri kusingadzivisiki.

Nekudaro, muchiitiko chipi chinogona kuita data analytics mabasa kuita "Rocket Science" matambudziko ekirasi? Pamwe panguva yatiri kutaura nezve data hombe chaizvo.
Kuita kuti Rocket Sainzi ive nyore, unogona kudya nzou chidimbu nechidimbu.

Izvo zvakanyanya discrete uye kuzvimiririra maapplication ako / masevhisi / mamicroservices ari, zvichave nyore kwauri iwe, vaunoshanda navo uye bhizinesi rese kugaya nzou.

Vanenge vese vatengi vedu vakauya kune ino postute, vavakazve nzvimbo zvichienderana nemaitiro einjiniya ezvikwata zveDevOps.

Asi kunyangwe ne "yakaparadzana, nzou" yekudya, isu tine mukana wakanaka we "oversaturation" yeIT landscape. Panguva ino zvakakodzera kumira, kufema uye kutarisa kudivi yakaderera-code engineering chikuva.

Vazhinji vanogadzira vanotyiswa netarisiro yekufa mumabasa avo kana vabva kure nekunyora kodhi yakananga kune "kudhonza" miseve muUI inopindirana yeakaderera-code masisitimu. Asi kuuya kwezvishandiso zvemuchina hakuna kutungamirira mukutsakatika kweinjiniya, asi kwakaunza basa ravo kune imwe nhanho!

Ngatione kuti sei.

Ongororo yedata mumunda we Logistics, telecom indasitiri, midhiya yekutsvaga, chikamu chezvemari chinogara chakabatana nemibvunzo inotevera:

• Kumhanya kwe otomatiki kuongororwa;
• Kugona kuitisa zviedzo pasina kukanganisa iyo huru yekugadzira data kuyerera;
• Kuvimbika kweiyo data yakagadzirirwa;
• Shandura yekutevera uye shanduro;
• Kuratidzwa kwedata, mutsara weData, CDC;
• Kukurumidza kuendesa kwezvinhu zvitsva kunzvimbo yekugadzira;
• Uye iyo inozivikanwa: mutengo wekusimudzira uye kutsigirwa.

Ndokunge, mainjiniya ane huwandu hukuru hwemabasa epamusoro-soro, ayo anogona kupedzwa nekubudirira kwakakwana chete nekujekesa kuziva kwavo kwepasi-chikamu chekusimudzira mabasa.

Izvo zvinodikanwa kuti vagadziri vaende kune imwe nhanho yaive shanduko uye digitalization yebhizinesi. Kukosha kwemugadziri kuri kushanduka zvakare: pane kushomeka kukuru kwevagadziri vanogona kunyudza ivo pachavo mune pfungwa dzebhizinesi riri otomatiki.

Ngatidhirowe fananidzo nemitauro yakaderera uye yepamusoro-soro yekuronga. Shanduko kubva kumitauro yakaderera kuenda kune yepamusoro-soro ishanduko kubva pakunyora "zvakananga mirairo mumutauro wehardware" kuenda ku "mirairo mumutauro wevanhu". Kureva, kuwedzera imwe layer yekubvisa. Muchiitiko ichi, shanduko kuenda kune yakaderera-code mapuratifomu kubva padanho repamusoro-purogiramu mitauro ishanduko kubva ku "mirayiridzo mumutauro wevanhu" kuenda ku "mirayiridzo mumutauro webhizinesi." Kana paine vagadziri vanosuwa neichi chokwadi, saka ivo vakasuwa, pamwe, kubva panguva yakazvarwa Java Script, iyo inoshandisa array kurongedza mabasa. Uye mabasa aya, ehe, ane software yekumisikidza pasi pehodhi nedzimwe nzira dzeiyo yepamusoro-level programming.

Naizvozvo, yakaderera-code ingori kutaridzika kweimwe nhanho yekubvisa.

Kushandiswa ruzivo uchishandisa yakaderera-code

Musoro wekodhi-yakadzika wakafara, asi ikozvino ndinoda kutaura nezve mashandisiro anoshanda e "yakaderera-code concepts" ndichishandisa muenzaniso weimwe yemapurojekiti edu.

Iyo Big Data Solutions divisheni yeNeoflex inonyanya kushanda muchikamu chezvemari chebhizinesi, kuvaka matura edhata nemadziva uye otomatiki akasiyana kushuma. Mune iyi niche, kushandiswa kwekodhi-yakaderera kwave kwenguva refu kwave chiyero. Pakati pezvimwe zvakaderera-code maturusi, tinogona kutaura maturusi ekuronga ETL maitiro: Informatica Power Center, IBM Datastage, Pentaho Data Integration. Kana Oracle Apex, iyo inoshanda senzvimbo yekukurumidza kusimudzira kwenzvimbo dzekuwana uye kugadzirisa data. Nekudaro, kushandiswa kweakaderera-code ekuvandudza maturusi haagare anosanganisira kuvaka zvakanyanya kunangwa zvikumbiro pane yekushambadzira tekinoroji stack nekutsamira kwakajeka kune mutengesi.

Uchishandisa yakaderera-code mapuratifomu, iwe unogona zvakare kuronga kurongeka kwe data kuyerera, kugadzira data sainzi mapuratifomu kana, semuenzaniso, ma module ekutarisa data data.

Mumwe wemienzaniso yakashandiswa yeruzivo mukushandisa yakaderera-kodhi maturusi ekuvandudza maturusi kubatirana pakati peNeoflex neMediascope, mumwe wevatungamiriri mumusika wekuRussia wekutsvagisa media. Chimwe chezvinangwa zvebhizinesi rekambani iyi kugadzira data pahwaro hwekuti vashambadziri, mapuratifomu eInternet, chiteshi cheTV, nhepfenyuro, masangano ekushambadzira uye mhando vanoita sarudzo nezvekutenga kushambadzira uye kuronga kutaurirana kwavo kwekushambadzira.

Media research inzvimbo inoremerwa tekinoroji yebhizinesi. Kuziva kutevedzana kwevhidhiyo, kuunganidza data kubva kumidziyo inoongorora kuona, kuyera chiitiko pane zvewebhu zviwanikwa - zvese izvi zvinoreva kuti kambani ine hombe yevashandi veIT uye ruzivo rwakakura mukuvaka mhinduro dzekuongorora. Asi kukura kwakanyanya muhuwandu hwemashoko, nhamba uye zvakasiyana-siyana zvezvinyorwa zvayo zvinomanikidza iyo IT data indasitiri kuti irambe ichifambira mberi. Mhinduro yakapusa yekuyera iyo yatove kushanda yeMediascope yekuongorora chikuva inogona kunge iri yekuwedzera vashandi veIT. Asi imwe yakanyanya kushanda mhinduro ndeyekukurumidza kusimudzira maitiro. Imwe yenhanho inotungamira munzira iyi inogona kunge iri kushandiswa kwepasi-code mapuratifomu.

Panguva yakatanga chirongwa, kambani yanga yatove nemhinduro yechigadzirwa. Zvisinei, kuitwa kwechigadziro muMSSQL hakwaigona kuzadzisa zvizere zvinotarisirwa zvekuwedzera kushanda uku uchichengetedza mutengo unogamuchirwa webudiriro.

Basa raive pamberi pedu raive rekuda chaizvo - Neoflex neMediascope vaifanira kugadzira mhinduro mumaindasitiri mukati megore risingasviki, zvichienderana nekuburitswa kweMVP mukati mechikamu chekutanga chemusi wekutanga.

Iyo Hadoop tekinoroji stack yakasarudzwa senheyo yekuvaka nyowani data chikuva chakavakirwa pasi-code komputa. HDFS yave chiyero chekuchengetedza data uchishandisa parquet mafaera. Kuti uwane iyo data iri papuratifomu, Hive yakashandiswa, umo zvese zviripo zvezvitoro zvinounzwa nenzira yematafura ekunze. Kuisa data mudura kwakaitwa uchishandisa Kafka neApache NiFi.

Chishandiso cheLowe-code mune iyi pfungwa chakashandiswa kukwenenzvera basa rinonyanya kushanda mukuvaka chikuva chekuongorora - basa rekuverenga data.

Iyo yakaderera-code Datagram chishandiso chakasarudzwa senzira huru yekugadzira data. Neoflex Datagram chishandiso chekugadzira shanduko uye kuyerera kwedata.
Uchishandisa chishandiso ichi, unogona kuita pasina kunyora Scala kodhi nemaoko. Scala kodhi inogadzirwa otomatiki uchishandisa iyo Model Driven Architecture nzira.

Kubatsira kuri pachena kweiyi nzira ndiko kukurumidza kufambisa nzira. Nekudaro, kunze kwekumhanya, kune zvakare zvinotevera zvakanakira:

• Kuona zviri mukati uye chimiro chezvinyorwa / vanogamuchira;
• Kutsvaga kwakabva data kuyerera zvinhu kuminda yega (mutsara);
• Kuitwa kwechikamu cheshanduko nekuona mhedzisiro yepakati;
• Kuongorora iyo kodhi kodhi uye kuigadzirisa isati yaitwa;
• Otomatiki kusimbisa shanduko;
• Otomatiki data kudhawunirodha 1 mu1.

Chipinganidzo chekupinda mune yakaderera-code mhinduro dzekugadzira shanduko yakadzikira: mugadziri anofanirwa kuziva SQL uye ave neruzivo rwekushanda neETL zvishandiso. Zvakakodzera kutaura kuti kodhi-inotyairwa shanduko jenareta haisi ETL maturusi mupfungwa yakafara yezwi. Maturusi e-low-code anogona kunge asina yavo yekodhi kodhi nharaunda. Ndiko kuti, iyo inogadzirwa kodhi ichaitwa munharaunda yaivepo pane cluster kunyangwe isati yaisa yakaderera-code mhinduro. Uye iyi ingangove imwe yekuwedzera kune yakaderera-code karma. Sezvo, pamwe chete neakaderera-code timu, "classic" timu inogona kushanda iyo inoshandisa mashandiro, semuenzaniso, mune yakachena Scala kodhi. Kuunza kuvandudzwa kubva kuzvikwata zviviri izvi mukugadzira kuchave kuri nyore uye kusina musono.

Zvichida zvakakosha kuziva kuti mukuwedzera kune yakaderera-kodhi, kune zvakare hapana-kodhi mhinduro. Uye pakati pavo, izvi zvinhu zvakasiyana. Yakaderera-kodhi inobvumira mugadziri kupindira zvakanyanya nekodhi yakagadzirwa. Panyaya yeDatagram, zvinokwanisika kuona uye kugadzirisa iyo yakagadzirwa Scala kodhi; hapana-kodhi inogona kusapa mukana wakadaro. Uyu mutsauko unokosha zvikuru kwete chete maererano nekushanduka kwemhinduro, asiwo maererano nekunyaradza uye kukurudzira mubasa revainjiniya ve data.

Solution architecture

Ngatiedzei kuona kuti sei yakaderera-code chishandiso chinobatsira kugadzirisa dambudziko rekugadzirisa kukurumidza kwekugadzira data calculation performance. Kutanga, ngatitarisei kune inoshanda mavakirwo ehurongwa. Muenzaniso mune iyi kesi ndiyo yekugadzira data modhi yekutsvagisa midhiya.

Data masosi munyaya yedu ane heterogeneous uye akasiyana:

• People metres (TV metres) isoftware nehardware inoverenga maitiro emushandisi kubva kune vakapindura paterevhizheni - ndiani, riini uye chii chiteshi cheTV chakatariswa mumhuri iri kutora chikamu muongororo. Ruzivo rwunopihwa irukova rwenguva dzekutarisa kutepfenyura dzakabatana ne media package uye media chigadzirwa. Dhata padanho rekurodha muData Lake inogona kupfumiswa nedemographic hunhu, geostratification, nguva yenguva uye rumwe ruzivo rwakakosha pakuongorora terevhizheni kuona kwechimwe chigadzirwa chenhau. Izvo zviyero zvakatorwa zvinogona kushandiswa kuongorora kana kuronga mishandirapamwe yekushambadzira, kuongorora chiitiko uye zvido zvevateereri, uye kuunganidza network yenhepfenyuro;
• Iyo data inogona kuuya kubva kune ekutarisa masisitimu ekutepfenyura terevhizheni uye kuyera kutariswa kwevhidhiyo zviwanikwa zvemukati paInternet;
• Maturusi ekuyeresa munzvimbo yewebhu, kusanganisira ese-saiti-centric uye mushandisi-centric metres. Iyo inopa data yeData Lake inogona kunge iri yekutsvagisa bar browser yekuwedzera uye nharembozha ine yakavakirwa-mukati VPN.
• Data inogonawo kubva kumasaiti anobatanidza mibairo yekuzadza mibvunzo yepamhepo uye mhedzisiro yekubvunzurudzwa kwefoni muongororo yekambani;
• Kuwedzera kupfumisa kwedhamu data kunogona kuitika nekudhawunirodha ruzivo kubva kumatanda emakambani evanodyidzana.

Kuitwa kweiyo kurodha kubva kusource masisitimu kuenda kune yekutanga nhanho yedata data inogona kurongeka nenzira dzakasiyana. Kana yakaderera-kodhi yakashandiswa kune izvi zvinangwa, otomatiki chizvarwa chekurodha zvinyorwa zvinoenderana nemetadata zvinogoneka. Mune ino kesi, hapana chikonzero chekudzika pasi kune iyo nhanho yekuvandudza sosi kune kunanga mepu. Kuti tiite otomatiki kurodha, isu tinofanirwa kumisikidza chinongedzo kune sosi, tozotsanangura mune yekurodha interface rondedzero yezvimiro zvinotakurwa. Iyo dhairekitori chimiro muHDFS ichagadzirwa otomatiki uye ichaenderana neiyo data yekuchengetedza chimiro pane sosi system.

Nekudaro, muchirevo chechirongwa ichi, takasarudza kusashandisa chikamu ichi chepuratifomu yakaderera-kodhi nekuda kwekuti kambani yeMediascope yakatotanga yakazvimirira basa rekugadzira sevhisi yakafanana ichishandisa Nifi + Kafka musanganiswa.

Izvo zvakakosha nekukasira kuratidza kuti maturusi aya haachinjike, asi anowirirana. Nifi naKafka vanokwanisa kushanda zvese zvakanangana (Nifi -> Kafka) uye kumashure (Kafka -> Nifi) kubatana. Kune iyo media yekutsvagisa chikuva, yekutanga vhezheni yebundle yakashandiswa.

Mune yedu kesi, NayFi yaida kugadzirisa marudzi akasiyana-siyana e data kubva kune masosi masisitimu uye kuvatumira kune Kafka broker. Muchiitiko ichi, mameseji akatumirwa kune chaiyo Kafka musoro uchishandisa PublishKafka Nifi processors. Kurongeka uye kugadzirisa kwemapaipi aya kunoitwa mune inotaridzika interface. Iyo Nifi chishandiso uye kushandiswa kweiyo Nifi + Kafka musanganiswa inogona zvakare kunzi yakaderera-code nzira yekuvandudza, iyo ine yakaderera chipingamupinyi chekupinda muBig Data tekinoroji uye inomhanyisa nzira yekuvandudza application.

Nhanho inotevera mukuitwa kweprojekiti yaive yekuunza data rakadzama kune imwechete semantic layer fomati. Kana sangano riine hunhu hwenhoroondo, kuverenga kunoitwa maererano nechikamu chiri mubvunzo. Kana sangano racho risiri renhoroondo, zvinokwanisika kuti uverengezve zvese zviri mukati mechinhu, kana kuramba zvachose kuverengazve chinhu ichi (nekuda kwekushaikwa kwekuchinja). Panguva ino, makiyi anogadzirwa kune ese masangano. Makiyi anochengetwa mumadhairekitori eHbase anoenderana neatenzi zvinhu, izvo zvine kunyorerana pakati pemakiyi ari papuratifomu yekuongorora uye makiyi kubva kune masisitimu. Kubatanidzwa kweatomic entities kunoperekedzwa nekupfumisa nemhedzisiro yekutanga kuverenga kwe data yekuongorora. Chimiro chekuverenga data chaive Spark. Iyo yakatsanangurwa mashandiro ekuunza data kune imwechete semantics yakashandiswa zvakare zvichibva pamamepu kubva kune yakaderera-code Datagram chishandiso.

Chivako chekuvaka chaida SQL kuwana data kune vashandisi vebhizinesi. Mukoko wakashandiswa pane iyi sarudzo. Zvinhu zvinonyoreswa muHive otomatiki kana iwe uchigonesa iyo "Registr Hive Tafura" sarudzo mune yakaderera-code chishandiso.

Calculation flow control

Datagram ine interface yekugadzira mafambiro ekuyerera madhizaini. Mamepu anogona kutangwa uchishandisa Oozie scheduler. Murukova yekuvandudza interface, zvinokwanisika kugadzira zvirongwa zvekufanana, kutevedzana, kana kuuraya-inotsamira data shanduko. Iko kune kutsigirwa kwe shell scripts uye java zvirongwa. Izvo zvakare zvinogoneka kushandisa iyo Apache Livy server. Apache Livy inoshandiswa kumhanyisa maapplication zvakananga kubva kunharaunda yekusimudzira.

Kana iyo kambani yatova neyayo maitiro orchestrator, zvinogoneka kushandisa iyo REST API kumisikidza mappings mune iripo kuyerera. Semuenzaniso, takava neruzivo rwakanyatsobudirira rwekuisa mepu muScala kuva orchestrator yakanyorwa muPLSQL neKotlin. Iyo REST API yechishandiso chepasi-code inosanganisira mashandiro akaita sekugadzira gore rinoitwa zvichibva pakugadzirwa kwemepu, kudaidza mepu, kudaidza nhevedzano yemepu, uye, hongu, kupfuudza paramita kuURL kuti uite mepu.

Pamwe chete naOozie, zvinokwanisika kuronga kuyerera kwekuverenga uchishandisa Airflow. Zvichida ini handizogari kwenguva refu pakuenzanisa pakati peOozie neAirflow, asi ndinongotaura kuti mumamiriro ekushanda pane chirongwa chekutsvakurudza midhiya, sarudzo yakawira mukufarira Airflow. Nharo huru panguva ino yaive nharaunda inoshanda zvakanyanya kugadzira chigadzirwa uye yakagadziridzwa interface + API.

Kuyerera kwemhepo zvakare kwakanaka nekuti inoshandisa inodiwa Python kutsanangura maitiro ekuverenga. Uye kazhinji, hakuna akawanda akavhurika sosi workflow manejimendi mapuratifomu. Kutangisa uye nekutarisa kuitwa kwemaitiro (kusanganisira Gantt chati) inongowedzera mapoinzi kune Airflow's karma.

Iyo yekumisikidza faira fomati yekutanga yakaderera-code mhinduro mepu yave spark-submit. Izvi zvakaitika nokuda kwezvikonzero zviviri. Kutanga, spark-submit inokutendera kuti umhanye zvakananga faira rejagi kubva kune console. Chechipiri, inogona kuve neruzivo rwese rwunodiwa kugadzirisa mafambiro ebasa (izvo zvinoita kuti zvive nyore kunyora zvinyorwa zvinogadzira Dag).
Chinhu chakajairika cheAirflow workflow mune yedu yaive SparkSubmitOperator.

SparkSubmitOperator inokutendera kuti umhanye zvirongo - zvakaiswa muDatagram mepu ine pre-yakagadzirwa ekuisa paramita kwavari.

Zvakakodzera kutaura kuti rimwe nerimwe Airflow basa rinomhanya mune rakasiyana shinda uye hapana chaanoziva nezve mamwe mabasa. Naizvozvo, kudyidzana pakati pemabasa kunoitwa uchishandisa control operators, senge DummyOperator kana BranchPythonOperator.

Kutorwa pamwe chete, kushandiswa kweDatagram yakaderera-code mhinduro pamwe chete nekusarudzika kwemafaira ekugadzirisa (kuumba Dag) kwakatungamira kukurumidza kukuru uye kurerutsa nzira yekuvandudza data kuyerera kuyerera.

Showcase calculation

Zvichida chikamu chehungwaru chakaremerwa mukugadzirwa kwe data yekuongorora inhanho yekuvaka showcases. Mumamiriro ezvinhu eimwe yekambani yekutsvagisa data kuverenga inoyerera, panguva ino, iyo data inoderedzwa kuita referefesheni, ichifunga nezvekugadzirisa kwenzvimbo dzenguva uye yakabatana neiyo nhepfenyuro grid. Izvo zvakare zvinogoneka kugadzirisa kune yemunharaunda nhepfenyuro network (nhau dzemunharaunda uye kushambadzira). Pakati pezvimwe zvinhu, danho iri rinoputsa nguva dzekuenderera mberi kwekuona kwezvigadzirwa zvemidhiya zvichienderana nekuongorora kwekuona nguva. Pakarepo, maonero ekuona "akayerwa" zvichienderana neruzivo nezve kukosha kwavo (kuverengera kwekugadzirisa chinhu).

Imwe nhanho yakaparadzana mukugadzirira showcases ndeyekusimbisa data. Iyo yekusimbisa algorithm inosanganisira kushandiswa kwehuwandu hwemasvomhu esainzi modhi. Zvisinei, kushandiswa kwepuratifomu yakaderera-code kunokubvumira kuputsa algorithm yakaoma kuita nhamba yakasiyana-siyana inooneka inoverengwa mappings. Imwe neimwe yemepu inoita basa rakamanikana. Nekuda kweizvozvo, kugadzirisa kwepakati, kutema matanda uye kuona kwematanho ekugadzirira data zvinogoneka.

Zvakasarudzwa kusiyanisa algorithm yekusimbisa mune zvinotevera substages:

• Kuvaka kudzoreredzwa kweTV network yekutarisa kutsamira munharaunda ine kutarisa kwese network mudunhu kwemazuva makumi matanhatu.
• Kuverengera kwevakasarira vadzidzi (kutsauka kweiyo chaiyo kukosha kubva kune iyo yakafanotaurwa neiyo regression modhi) kune ese mapoinzi ekudzoreredza uye kwezuva rakaverengerwa.
• Sarudzo yeanomalous region-network pairs, apo chiyero chevadzidzi chezuva rekugadzirisa chinodarika zvakajairwa (zvinotsanangurwa nemaitiro ekushanda).
• Kuverengerwazve kweyakagadziriswa yakasara yemudzidzi yedunhu risinganzwisisike-TV network pairs kune mumwe nemumwe aipindura akatarisa network mudunhu, achitarisa mupiro wemupinduri uyu (huwandu hweshanduko mune yakasara yevadzidzi) kana isingabatanidze kutarisa kwemupinduri uyu kubva kumuenzaniso. .
• Tsvaga vavhoti vane kusarudzika kunounza chiyero chemudzidzi chezuva rekubhadhara kudzoka kune zvakajairika.

Muenzaniso wepamusoro unosimbisa fungidziro yekuti mugadziri wedata atova nezvakawanda mupfungwa dzake ... Uye, kana ichi chiri chaizvo "injiniya" uye kwete "coder," ipapo kutya kwekuderedzwa kwehunyanzvi kana uchishandisa yakaderera-code maturusi iye. inofanira kudzoka shure.

Chii chimwe chinogona kuita yakaderera-code?

Iyo chiyero chekushandiswa kweiyo yakaderera-code chishandiso chebatch uye kuyerera data kugadzirisa pasina kukosha kwekunyora kodhi muScala haigumire ipapo.

Iko kushandiswa kwekodhi-yakaderera mukugadzirwa kwedatalake yakatove chiyero kwatiri. Tinogona kutaura kuti mhinduro dzakavakirwa paHadoop stack dzinotevera nzira yekusimudzira yekirasi DWHs yakavakirwa paRDBMS. Zvishandiso zvakaderera-kodhi paHadoop stack zvinogona kugadzirisa ese ari maviri ekugadzirisa data mabasa uye basa rekuvaka yekupedzisira BI interfaces. Uyezve, zvinofanirwa kucherechedzwa kuti BI inogona kureva kwete chete kumiririrwa kwedata, asiwo kugadziridzwa kwavo nevashandisi vebhizinesi. Isu tinowanzo shandisa mashandiro aya pakuvaka mapuratifomu ekuongorora echikamu chemari.

Pakati pezvimwe zvinhu, kushandisa pasi-code uye, kunyanya, Datagram, zvinokwanisika kugadzirisa dambudziko rekutevera kwakabva data stream zvinhu neatomicity kusvika kune imwe neimwe minda (mutsara). Kuti uite izvi, iyo yakaderera-code chishandiso inoshandisa interface neApache Atlas uye Cloudera Navigator. Chaizvoizvo, mugadziri anofanirwa kunyoresa seti yezvinhu mumaduramazwi eAtlas uye kunongedzera zvinhu zvakanyoreswa pakuvaka mepu. Iyo meshini yekutevera kwakabva data kana kuongorora kutsamira kwechinhu inochengetedza nguva yakawanda pazvinenge zvichidikanwa kuita kuvandudzwa kune maalgorithms ekuverenga. Semuenzaniso, kana uchigadzira zvinyorwa zvemari, chimiro ichi chinokutendera kuti urarame zvakasununguka panguva yekuchinja kwemutemo. Mushure mezvose, zviri nani isu tinonzwisisa kutsamira kwe-inter-form mumamiriro ezvinhu ezvinhu zvechinyorwa chakadzama, zvishoma isu tichasangana ne "kamwe kamwe" kukanganisa uye kuderedza nhamba yezvekugadzirisa.

Data Hunhu & Yakaderera-kodhi

Rimwe basa rakaitwa neiyo yakaderera-code chishandiso paMediascope purojekiti yaive yeData Quality class basa. Chinhu chakakosha pakuitwa kwepombi yekuongorora data yepurojekiti yekambani yekutsvagisa kwaive kushaikwa kwemhedzisiro pakuita uye kumhanya kweiyo huru yekuverenga data. Kuti ukwanise kuronga yakazvimirira data verification inoyerera, iyo yatove yakajairika Apache Airflow yakashandiswa. Sezvo nhanho imwe neimwe yekugadzirwa kwedata yakagadzirira, chikamu chakaparadzana chepombi yeDQ yakatangwa zvakafanana.

Inoonekwa seyakanaka tsika yekutarisa mhando yedata kubva panguva yekutanga kwayo mune yekuongorora papuratifomu. Tine ruzivo nezve metadata, tinogona kutarisa kutevedzera nemamiriro ekutanga kubva panguva iyo ruzivo rwunopinda muchikamu chekutanga - kwete null, zvipingaidzo, makiyi ekunze. Kuita uku kunoitwa zvichibva pamamepu anogadzirwa otomatiki emhuri yemhando yedata muDatagram. Kugadzira kodhi mune iyi kesi zvakare yakavakirwa pamuenzaniso metadata. PaMediascope project, iyo interface yakaitwa nemetadata ye Enterprise Architect chigadzirwa.

Nekubatanidza iyo yakaderera-code chishandiso ne Enterprise Architect, iwo anotevera cheki akagadzirwa otomatiki:

• Kutarisa kuvepo kwe "null" kukosha muminda ine "kwete null" modifier;
• Kutarisa kuvepo kweanodzokorora kiyi yekutanga;
• Kutarisa kiyi yekunze yesangano;
• Kutarisa kusiyana kwetambo zvichienderana neseti yeminda.

Kuti uwane mamwe macheki akaomesesa ekuwanikwa kwedata uye kuvimbika, mepu ine Scala Expression yakagadzirwa, iyo inotora sekuisa yekunze Spark SQL cheki kodhi yakagadzirwa nevaongorori kuZeppelin.

Ehe, kugadzira otomatiki kwecheki kunofanirwa kuwanikwa zvishoma nezvishoma. Mukati megadziriro yeprojekti yakatsanangurwa, izvi zvakatangirwa nematanho anotevera:

• DQ inoshandiswa muzvinyorwa zveZeppelin;
• DQ yakavakwa mumepu;
• DQ muchimiro cheakaparadzana mamepu makuru ane seti yese yecheki yechimwe chinhu chakasiyana;
• Universal parameterized DQ mepu iyo inogamuchira ruzivo nezve metadata uye bhizinesi cheki sekuisa.

Zvichida mukana mukuru wekugadzira parameterized cheki sevhisi kudzikiswa kwenguva inotora kuendesa mashandiro kunharaunda yekugadzira. Macheki emhando nyowani anogona kupfuura iyo yekare maitiro ekuunza kodhi zvisina kunanga kuburikidza nekusimudzira uye kuyedza nharaunda:

• Yese metadata cheki inogadzirwa otomatiki kana modhi yagadziriswa muEA;
• Kuwanikwa kwedata kutariswa (kuona kuvepo kwechero data pane imwe nguva panguva) inogona kugadzirwa zvichibva pane dhairekitori inochengeta inotarisirwa nguva yekuonekwa kwechidimbu chinotevera che data mumamiriro ezvinhu;
• Bhizinesi data yekusimbisa cheki inogadzirwa nevanoongorora muZeppelin zvinyorwa. Kubva ipapo ivo vanotumirwa zvakananga kuDQ module setup matafura munzvimbo yekugadzira.

Iko hakuna njodzi dzekutumira zvakananga zvinyorwa kune kugadzirwa. Kunyangwe paine chikanganiso che syntax, iyo yakanyanya kutityisidzira kutadza kuita cheki imwe chete, nekuti iyo data kuverenga kuyerera uye yemhando yekutarisa kuyerera inoparadzaniswa kubva kune mumwe nemumwe.

Muchidimbu, sevhisi yeDQ iri kushanda zvachose munzvimbo yekugadzira uye yakagadzirira kutanga basa rayo panguva iyo chikamu chinotevera che data chabuda.

Pane mhedziso

Zvakanakira kushandisa yakaderera-code zviri pachena. Vagadziri havafanire kuvandudza application kubva pakutanga. Uye mugadziri akasunungurwa kubva kune mamwe mabasa anoburitsa mhedzisiro nekukurumidza. Kumhanya, zvakare, kunosunungura imwe nguva yekugadzirisa nyaya dzekugadzirisa. Nokudaro, munyaya iyi, unogona kuvimba nemhinduro iri nani uye inokurumidza.

Ehe, yakaderera-code haisi panacea, uye mashiripiti haaitike ega:

• Indasitiri yakaderera-kodhi iri kupfuura nepakati pe "kusimba" nhanho, uye hapana yunifomu yemaindasitiri zviyero;
• Mishonga yakawanda yakaderera-kodhi haisi yekusununguka, uye kuitenga kunofanira kuva nhanho inoziva, iyo inofanira kuitwa nechivimbo chakazara mune zvemari inobatsira yekuishandisa;
• Mazhinji epasi-code mhinduro haagare achishanda zvakanaka neGIT/SVN. Kana kuti ivo vanonetsekana kushandisa kana iyo yakagadzirwa kodhi yakavanzwa;
• Paunenge uchiwedzera dhizaini, zvingave zvakafanira kunatsiridza iyo yakaderera-code mhinduro - iyo, zvakare, inomutsa mhedzisiro ye "kunamatira uye kutsamira" kune mupi weiyo yakaderera-code mhinduro.
• Chiyero chakaringana chekuchengetedza chinogoneka, asi chinoshanda zvakanyanya uye chakaoma kuita mune yakaderera-code system injini. Low-code mapuratifomu anofanirwa kusarudzwa kwete chete pamusimboti wekutsvaga mabhenefiti kubva mukushandisa kwavo. Paunenge uchisarudza, zvakakodzera kubvunza mibvunzo nezve kuwanikwa kwekushanda kwekutonga kwekuwana uye kutumira / kukwira kwedata rekuzivikanwa kusvika padanho reiyo yese IT landscape yesangano.

Zvisinei, kana zvikanganiso zvose zvegadziriro yakasarudzwa zvichizivikanwa kwauri, uye zvikomborero kubva pakushandiswa kwayo, zvisinei, zviri muhuwandu hwakawanda, zvino enda kune kodhi duku pasina kutya. Uyezve, kuchinja kwairi hakudzivisiki - sezvakangoita chero shanduko isingadzivisiki.

Kana mumwe mugadziri ari pasi-code platform achiita basa rake nokukurumidza kupfuura vagadziri vaviri vasina pasi-code, zvino izvi zvinopa kambani kutanga musoro mune zvose. Chikumbaridzo chekupinda mune yakaderera-code mhinduro yakaderera pane ye "tsika" matekinoroji, uye izvi zvine mhedzisiro yakanaka panyaya yekushomeka kwevashandi. Paunenge uchishandisa maturusi e-low-code, zvinokwanisika kukurumidzira kudyidzana pakati pezvikwata zvinoshanda uye kuita sarudzo nekukurumidza nezve kurongeka kwenzira yakasarudzwa yedata sainzi research. Mapuratifomu akaderera anogona kutyaira shanduko yedhijitari yesangano nekuti mhinduro dzinogadzirwa dzinogona kunzwisiswa nevasiri tekinoroji nyanzvi (kunyanya vashandisi vebhizinesi).

Kana iwe uine nguva dzakasimba, dzakaremerwa bhizinesi pfungwa, kushomeka kwehunyanzvi hwetekinoroji, uye iwe unofanirwa kukurumidzira nguva yako yekutengesa, saka yakaderera-kodhi ndiyo imwe nzira yekuzadzisa zvaunoda.

Iko hakuna kuramba kukosha kwemidziyo yekuvandudza yechinyakare, asi kazhinji, kushandisa yakaderera-code mhinduro ndiyo nzira yakanakisa yekuwedzera kugona kwemabasa ari kugadziriswa.

Source: www.habr.com