I-DataHub yoMthombo oVulekileyo: Ukukhangela kweMetadata ye-LinkedIn kunye nePlatifomu yokuFumana

I-DataHub yoMthombo oVulekileyo: Ukukhangela kweMetadata ye-LinkedIn kunye nePlatifomu yokuFumana

Ukufumana idatha oyifunayo ngokukhawuleza kubalulekile kuyo nayiphi na inkampani exhomekeke kwixabiso elikhulu ledatha ukwenza izigqibo eziqhutywa yidatha. Oku akuchaphazeli kuphela imveliso yabasebenzisi bedatha (kubandakanywa abahlalutyi, abaphuhlisi bokufunda koomatshini, izazinzulu zedatha, kunye neenjineli zedatha), kodwa kunempembelelo ngqo kwimveliso yokugqibela exhomekeke kumbhobho wokufunda ngomatshini (ML). Ukongeza, umkhwa wokuphumeza okanye wokwakha amaqonga okufunda ngoomatshini ngokwendalo uphakamisa umbuzo: yeyiphi indlela yakho yangaphakathi yokufumanisa iimpawu, iimodeli, iimethrikhi, iiseti zedatha, njl.

Kweli nqaku siza kuthetha ngendlela esipapashe ngayo umthombo wedatha phantsi kwelayisensi evulekileyo DataHub kukhangelo lwethu lwemethadatha kunye neqonga lokufumanisa, ukuqala kwiintsuku zokuqala zeprojekthi Uphi. I-LinkedIn igcina inguqulelo yayo yeDataHub ngokwahlukileyo kwinguqulelo yomthombo ovulekileyo. Siza kuqala ngokucacisa ukuba kutheni sifuna iindawo ezimbini ezahlukeneyo zophuhliso, emva koko sixoxe ngeendlela zokuqala zokusebenzisa umthombo ovulekileyo we-HowHows kwaye sithelekise inguqulelo yethu yangaphakathi (yemveliso) yeDataHub kunye nenguqulelo GitHub. Siza kwabelana ngeenkcukacha malunga nesisombululo sethu esitsha esizisebenzelayo sokutyhala kunye nokufumana uhlaziyo lomthombo ovulekileyo ukugcina zombini iindawo zokugcina zilungelelaniswa. Okokugqibela, siza kubonelela ngemiyalelo yokuba ungaqalisa njani ukusebenzisa umthombo ovulekileyo weDataHub kwaye sixoxe ngokufutshane ngobume bayo.

I-DataHub yoMthombo oVulekileyo: Ukukhangela kweMetadata ye-LinkedIn kunye nePlatifomu yokuFumana

WhereHows ngoku iDataHub!

Iqela lemetadata lika-LinkedIn libonisiwe ngaphambili DataHub (umlandeli we-HowHows), uphando lwe-LinkedIn kunye neqonga lokufumanisa i-metadata, kunye nezicwangciso ezabelwana ngazo zokuyivula. Kungekudala emva kwesi sibhengezo, sikhuphe inguqulelo ye-alpha yeDataHub kwaye sabelane ngayo noluntu. Ukusukela ngoko, siye sanikela ngokuqhubekayo kwindawo yokugcina kwaye sasebenza nabasebenzisi abanomdla ukongeza ezona mpawu ziceliweyo kunye nokusombulula iingxaki. Ngoku sinovuyo ukubhengeza ukukhululwa ngokusemthethweni DataHub kwiGitHub.

IiNdlela zoMthombo oVulekileyo

WhereHows, i-portal ye-LinkedIn yokuqala yokufumana idatha kunye nalapho ivela khona, yaqala njengeprojekthi yangaphakathi; Iqela lemetadata liyivulile ikhowudi yomthombo ngo-2016. Ukususela ngoko, iqela liye lahlala ligcina iikhowudi ezimbini ezahlukeneyo-enye yomthombo ovulekileyo kwaye enye yokusetyenziswa kwangaphakathi kwe-LinkedIn-njengoko ingezizo zonke iimpawu zemveliso ezenzelwe iimeko zokusetyenziswa kwe-LinkedIn zazisebenza ngokubanzi kubaphulaphuli ababanzi. Ukongeza, i-HowHows inokuxhomekeka kwangaphakathi (iziseko zophuhliso, amathala eencwadi, njl.njl.) engeyiyo imithombo evulelekileyo. Kwiminyaka eyalandelayo, i-HowHows ihambe ngemijikelo emininzi yokuphindaphinda kunye nophuhliso, isenza ukugcina iziseko ezimbini zekhowudi ukuvumelanisa kube ngumngeni omkhulu. Iqela le-metadata lizame iindlela ezahlukeneyo kule minyaka ukuzama ukugcina uphuhliso lwangaphakathi kunye nomthombo ovulekileyo kwi-sync.

Zama kuqala: "Vula umthombo kuqala"

Siqale salandela "umthombo ovulekileyo kuqala" imodeli yophuhliso, apho uninzi lophuhliso lwenzeka kwindawo yokugcina umthombo ovulekileyo kunye notshintsho lwenzelwe ukuthunyelwa kwangaphakathi. Ingxaki ngale ndlela kukuba ikhowudi ihlala ityhalelwa kwi-GitHub kuqala ngaphambi kokuba ihlaziywe ngokupheleleyo ngaphakathi. Kuze kubekho utshintsho oluvela kwindawo yomthombo ovulekileyo kunye nokuthunyelwa kwangaphakathi okutsha kwenziwa, asiyi kufumana nayiphi na imiba yokuvelisa. Kwimeko yokuthunyelwa kakubi, kwakunzima kakhulu ukufumanisa unobangela ngenxa yokuba utshintsho lwenziwa kwiibhetshi.

Ukongeza, le modeli yanciphisa imveliso yeqela xa kusenziwa izinto ezintsha ezifuna uphindaphindo olukhawulezayo, kuba inyanzelise ukuba zonke iinguqu zityhalelwe kuqala kwindawo yokugcina umthombo ovulekileyo emva koko zityhalelwe kwindawo yokugcina yangaphakathi. Ukunciphisa ixesha lokucubungula, ukulungiswa okufunekayo okanye utshintsho lunokwenziwa kwindawo yokugcina yangaphakathi kuqala, kodwa oku kube yingxaki enkulu xa kufikwa ekudibaniseni olo tshintsho lubuyele kwindawo yomthombo ovulekileyo ngenxa yokuba iindawo zokugcina ezimbini zazingaphandle kokuvumelanisa.

Le modeli ilula kakhulu ukuyiphumeza kumaqonga ekwabelwana ngawo, amathala eencwadi, okanye iiprojekthi zeziseko ezingundoqo kunezicelo ezigcweleyo zewebhu. Ukongeza, le modeli ifanelekile kwiiprojekthi eziqala umthombo ovulekileyo ukusuka kusuku lokuqala, kodwa i-HowHows yakhiwe njengesicelo sewebhu sangaphakathi ngokupheleleyo. Kwakunzima kakhulu ukukhupha ngokupheleleyo konke ukuxhomekeka kwangaphakathi, ngoko ke bekufuneka sigcine ifolokhwe yangaphakathi, kodwa ukugcina ifolokhwe yangaphakathi kunye nokuphuhlisa umthombo ovulekileyo kakhulu akuzange kusebenze.

Umzamo wesibini: "Ngaphakathi kuqala"

** Njengomzamo wesibini, siye safudukela kwimodeli yophuhliso "yangaphakathi yokuqala", apho uninzi lophuhliso lwenzeka ngaphakathi kwaye utshintsho lwenziwa rhoqo kwikhowudi yomthombo ovulekileyo. Nangona le modeli ifanelekile kwimeko yethu yokusetyenziswa, ineengxaki zendalo. Ukutyhala ngokuthe ngqo zonke iiyantlukwano kwindawo yokugcina umthombo ovulekileyo kwaye emva koko uzame ukusombulula iingxabano kamva lukhetho, kodwa kuthatha ixesha. Abaphuhlisi kwiimeko ezininzi bazama ukungakwenzi oku ngalo lonke ixesha bephonononga ikhowudi yabo. Ngenxa yoko, oku kuya kwenziwa kancinci rhoqo, kwiibhetshi, kwaye ke kwenza kube nzima kakhulu ukusombulula ukudibanisa ungquzulwano kamva.

Okwesithathu yasebenza!

Ezi nzame zimbini aziphumelelanga zikhankanywe ngasentla zibangele ukuba indawo yokugcina i-HowHows GitHub ihlale iphelelwe lixesha ixesha elide. Iqela liqhubekile nokuphucula iimpawu zemveliso kunye noyilo, ukuze inguqulelo yangaphakathi ye-HowHows ye-LinkedIn ibe phambili ngakumbi kunenguqulelo yomthombo ovulekileyo. Ide ibe negama elitsha- DataHub. Ngokusekelwe kwimizamo yangaphambili engaphumelelanga, iqela lagqiba ekubeni liphuhlise isisombululo esinobunzima, sexesha elide.

Kuyo nayiphi na iprojekthi entsha yomthombo ovulekileyo, iqela le-LinkedIn lomthombo ovulekileyo licebisa kwaye lixhasa imodeli yophuhliso apho iimodyuli zeprojekthi ziphuhliswa ngokupheleleyo kumthombo ovulekileyo. Izinto zakudala eziguqulelweyo zisasazwa kwindawo yokugcina uluntu kwaye emva koko zijongwe kwakhona kwi-LinkedIn ye-artifact yangaphakathi kusetyenziswa. isicelo selayibrari yangaphandle (ELR). Ukulandela le modeli yophuhliso akulunganga kuphela kwabo basebenzisa umthombo ovulekileyo, kodwa kukhokelela kwimodyuli engaphezulu, eyandisiweyo, kunye neplagi yolwakhiwo.

Nangona kunjalo, isicelo esivuthiweyo sokugqibela esifana neDathaHub siya kufuna ixesha elibalulekileyo lokufikelela kweli lizwe. Oku kukwathintela ukuba nokwenzeka kokukhangela okuvulelekileyo umiliselo olusebenza ngokupheleleyo phambi kokuba zonke izinto ezixhomekeke ngaphakathi zifunyanwe ngokupheleleyo. Yiyo loo nto siye saphuhlisa izixhobo ezisinceda ukuba senze igalelo lomthombo ovulekileyo ngokukhawuleza nangeentlungu ezisezantsi. Esi sisombululo sixhamla zombini iqela lemethadatha (umphuhlisi weDataHub) kunye noluntu oluvulekileyo lomthombo. La macandelo alandelayo aza kuxoxa ngale ndlela intsha.

I-Open Source yoPapasho oluzenzekelayo

Indlela yakutshanje yeqela leMetadata kumthombo ovulekileyo weDataHub kukuphuhlisa isixhobo esivumelanisa ngokuzenzekelayo i-codebase yangaphakathi kunye nomthombo ovulekileyo womthombo. Iimpawu zomgangatho ophezulu wale zixhobo zibandakanya:

  1. Vumelanisa ikhowudi ye-LinkedIn ukuya / ukusuka kumthombo ovulekileyo, okufanayo rsync.
  2. Ukuveliswa kwesihloko sephepha-mvume, esifana ne Apache Rat.
  3. Yenza ngokuzenzekelayo iilog zokuzibophelela kwimithombo evulekileyo ukusuka kwiilog zokuzibophelela zangaphakathi.
  4. Thintela utshintsho lwangaphakathi olwaphula umthombo ovulekileyo owakhayo uvavanyo lokuxhomekeka.

La macandelwana alandelayo aza kuphonononga kule misebenzi ikhankanywe ngasentla eneengxaki ezinomdla.

Ungqamaniso lwekhowudi yemvelaphi

Ngokungafaniyo nenguqulelo yomthombo ovulekileyo weDataHub, eyi-GitHub enye yokugcina, inguqulo ye-LinkedIn yeDataHub yindibaniselwano yeendawo zokugcina ezininzi (ezibizwa ngaphakathi. iimveliso ezininzi). I-DataHub interface, imodeli yelayibrari yemethadatha, inkonzo ye-metadata yokugcina i-backend, kunye nemisebenzi yokusasaza ihlala kwiindawo zokugcina ezahlukeneyo kwi-LinkedIn. Nangona kunjalo, ukwenza kube lula kubasebenzisi bomthombo ovulekileyo, sinovimba omnye wenguqulelo yomthombo ovulekileyo weDataHub.

I-DataHub yoMthombo oVulekileyo: Ukukhangela kweMetadata ye-LinkedIn kunye nePlatifomu yokuFumana

Umzobo 1: Ungqamaniso phakathi kweendawo zokugcina LinkedIn DataHub kunye novimba omnye DataHub Vula Umnikezi

Ukuxhasa ulwakhiwo oluzenzekelayo, ukutyhala, kunye nokutsala umsebenzi, isixhobo sethu esitsha sizenza ngokuzenzekelayo imephu yomgangatho wefayile ehambelana nefayile nganye yomthombo. Nangona kunjalo, i-toolkit ifuna uqwalaselo lokuqala kwaye abasebenzisi kufuneka babonelele ngemephu yemodyuli ephezulu njengoko kubonisiwe ngezantsi.

{
  "datahub-dao": [
    "${datahub-frontend}/datahub-dao"
  ],
  "gms/impl": [
    "${dataset-gms}/impl",
    "${user-gms}/impl"
  ],
  "metadata-dao": [
    "${metadata-models}/metadata-dao"
  ],
  "metadata-builders": [
    "${metadata-models}/metadata-builders"
  ]
}

Imephu yenqanaba lemodyuli yi-JSON elula enezitshixo eziziimodyuli ekujoliswe kuzo kwindawo yokugcina umthombo ovulekileyo kwaye amaxabiso luluhlu lweemodyuli zomthombo kwiindawo zokugcina ze-LinkedIn. Nayiphi na imodyuli ekujoliswe kuyo kwindawo yokugcina umthombo ovulekileyo inokondliwa ngalo naliphi na inani leemodyuli zomthombo. Ukubonisa amagama angaphakathi eendawo zokugcina kwiimodyuli zomthombo, sebenzisa uguqulelo lomtya ngesitayela sikaBash. Ukusebenzisa ifayile yemephu yomgangatho wemodyuli, izixhobo zenza ifayile yefayile yemephu ngokuskena zonke iifayile kubalawuli abahambelanayo.

{
  "${metadata-models}/metadata-builders/src/main/java/com/linkedin/Foo.java":
"metadata-builders/src/main/java/com/linkedin/Foo.java",
  "${metadata-models}/metadata-builders/src/main/java/com/linkedin/Bar.java":
"metadata-builders/src/main/java/com/linkedin/Bar.java",
  "${metadata-models}/metadata-builders/build.gradle": null,
}

Inqanaba lefayile lemephu lenziwe ngokuzenzekelayo zizixhobo; nangona kunjalo, inokuhlaziywa ngesandla ngumsebenzisi. Le yimephu ye-1: 1 yefayile ye-LinkedIn yefayile kwifayile kwindawo yokugcina umthombo ovulekileyo. Kukho imigaqo emininzi eyayanyaniswa nolu yilo oluzenzekelayo lonxulumano lwefayile:

  • Kwimeko yeemodyuli ezininzi zemithombo yemodyuli ekujoliswe kuyo kumthombo ovulekileyo, iingxabano zinokuvela, umz. FQCN, ekhoyo kwimodyuli yomthombo engaphezulu kwesinye. Njengesicwangciso sokusombulula impixano, izixhobo zethu azigqibekanga kukhetho "lokugqibela oluphumeleleyo".
  • "null" ithetha ukuba ifayile yemvelaphi ayiyonxalenye yendawo yokugcina umthombo.
  • Emva kokungeniswa komthombo ngamnye ovulekileyo okanye utsalo, le maphu ihlaziywa ngokuzenzekelayo kwaye i-snapshot yenziwe. Oku kuyimfuneko ukuchonga izongezo kunye nokususwa kwikhowudi yomthombo ukususela kwisenzo sokugqibela.

Ukudala iinkuni zokuzibophelela

Iilog zokuzibophelela zonikezelo lwemithombo evulekileyo nazo zenziwa ngokuzenzekelayo ngokudibanisa iilog zokubophelela zogcino lwangaphakathi. Apha ngezantsi kukho isampulu yokuzibophelela kwelog ukubonisa ubume belog yokuzinikela eyenziwe sisixhobo sethu. Isibophelelo sibonisa ngokucacileyo ukuba zeziphi iinguqulelo zemithombo yogcino ezipakishwe kweso sibophelelo kwaye sinika isishwankathelo selogi yokuzinikela. Jonga le ngaphandle zibophelele usebenzisa umzekelo wokwenyani welogi yokuzibophelela eyenziwe yitoolkit yethu.

metadata-models 29.0.0 -> 30.0.0
    Added aspect model foo
    Fixed issue bar

dataset-gms 2.3.0 -> 2.3.4
    Added rest.li API to serve foo aspect

MP_VERSION=dataset-gms:2.3.4
MP_VERSION=metadata-models:30.0.0

Uvavanyo lokuxhomekeka

LinkedIn unayo iziseko zovavanyo lokuxhomekeka, okunceda ukuqinisekisa ukuba utshintsho kwi-multiproduct yangaphakathi ayiphuli indibano ye-multiproducts exhomekeke. Umthombo ovulekileyo wogcino lweDataHub ayiyomveliso emininzi, kwaye ayinakuba luxhomekeke ngokuthe ngqo kuyo nayiphi na imveliso emininzi, kodwa ngoncedo lwesisongelo semveliso emininzi esithatha ikhowudi yomthombo ovulekileyo weDataHub, sisengasebenzisa olu vavanyo lokuxhomekeka. Ngoko ke, naluphi na utshintsho (olunokuthi kamva luvezwe) kuyo nayiphi na imveliso emininzi eyondla umthombo ovulekileyo wogcino lweDathaHub ibangela isiganeko sokwakha kwi-multiproduct yeqokobhe. Ke ngoko, naluphi na utshintsho olusilelayo ukwakha imveliso esongayo luyasilela kuvavanyo ngaphambi kokuba luyenze imveliso yokuqala kwaye lubuyiselwe umva.

Le yindlela eluncedo enceda ukuthintela nakuphi na ukuzibophelela kwangaphakathi okwaphula umthombo ovulekileyo wokwakha kwaye kubhaqwe ngexesha lokuzibophelela. Ngaphandle koku, kuya kuba nzima ukufumanisa ukuba sesiphi na isibophelelo sangaphakathi esibangele uvimba womthombo ovulekileyo ukuba ungaphumeleli, kuba sidibanisa utshintsho lwangaphakathi kwiDataHub yomthombo ovulekileyo wovimba.

Umahluko phakathi komthombo ovulekileyo weDataHub kunye nenguqulelo yethu yemveliso

Ukuza kuthi ga ngoku, sixoxe ngesisombululo sethu sokungqamanisa iinguqulelo ezimbini zeDathaHub zokugcina, kodwa asikachazi izizathu zokuba kutheni sifuna imijelo emibini eyahlukeneyo yophuhliso kwasekuqaleni. Kweli candelo, siya kuluhlula umahluko phakathi kwenguqu yoluntu yeDataHub kunye nenguqu yokuvelisa kwiiseva ze-LinkedIn, kwaye uchaze izizathu zolu mahluko.

Omnye umthombo wokungangqinelani uvela kwinto yokuba inguqulelo yethu yemveliso inokuxhomekeka kwikhowudi engekavuli mthombo, njengeLinkedIn's Offspring (Isakhelo seLinkedIn yokuxhomekeka kwangaphakathi). Inzala isetyenziswa ngokubanzi kwiikhowudi zangaphakathi kuba yindlela ekhethwayo yokulawula uqwalaselo oluguquguqukayo. Kodwa ayingomthombo ovulekileyo; ngoko besidinga ukufumana ezinye iindlela zomthombo ovulekileyo kumthombo ovulekileyo weDataHub.

Zikho nezinye izizathu. Njengoko sisenza izandiso kwimodeli yemethadatha yeemfuno ze-LinkedIn, ezi zandiso ziqhelekile ngokukodwa kwi-LinkedIn kwaye azinakusebenza ngokuthe ngqo kwezinye iimeko. Umzekelo, sineeleyibhile ezithe ngqo kakhulu kwii-ID zabathathi-nxaxheba kunye nezinye iintlobo zemetadata ehambelanayo. Ke, ngoku asizibandakanyi ezi zandiso kwimodeli yemethadatha yomthombo ovulekileyo weDataHub. Njengoko sisebenzisana noluntu kwaye siqonda iimfuno zabo, siya kusebenza kwiinguqulelo eziqhelekileyo zomthombo ovulekileyo wolu lwandiso apho lufuneka khona.

Ukusebenziseka ngokulula kunye nokulungelelaniswa lula kuluntu oluvulekileyo lomthombo luphinde lwaphefumlela ezinye iiyantlukwano phakathi kweenguqulelo ezimbini zeDataHub. Umahluko kwiziseko zokusetyenzwa komsinga ngumzekelo omhle woku. Nangona inguqulelo yethu yangaphakathi isebenzisa isakhelo sokusingathwa komjelo olawulwayo, sikhethe ukusebenzisa i-built-in (i-standalone) inkqubo yokuhambisa inguqulelo yomthombo ovulekileyo ngenxa yokuba igwema ukudala enye isiseko sokuxhomekeka.

Omnye umzekelo womahluko kukuba ne-GMS enye (uGcino lweMetadata Jikelele) ekuphunyezweni komthombo ovulekileyo kune-GMS ezininzi. I-GMA (i-Architecture yeMetadata Jikelele) ligama le-architecture ye-back-end ye-DataHub, kwaye i-GMS igcina i-metadata kumxholo we-GMA. I-GMA lulwakhiwo olubhetyebhetye kakhulu olukuvumela ukuba usasaze ulwakhiwo lwedatha nganye (umzekelo, iiseti zedatha, abasebenzisi, njl.njl.) kwisitoreji sayo semetadata, okanye ugcine ulwakhiwo lwedatha eninzi kwivenkile yemetadata enye nje ukuba ubhaliso luqulethe imephu yedatha I-GMS ihlaziywa. Ukusebenziseka lula, sikhethe umzekelo omnye we-GMS ogcina zonke iintlobo ezahlukeneyo zedatha kwiDataHub yomthombo ovulekileyo.

Uluhlu olupheleleyo lweeyantlukwano phakathi kokuphunyezwa kokubini lunikwe kwitheyibhile engezantsi.

Iziphumo zomkhiqizo
LinkedIn DataHub
Vula uMthombo weDataHub

Ulwakhiwo lweDatha oluxhaswayo
1) Iiseti zedatha 2) Abasebenzisi 3) Iimetriki 4) Iimpawu zeML 5) Iitshathi 6) Iidashbhodi
1) Iiseti zedatha 2) Abasebenzisi

Imithombo yeMetadata eXhasiweyo yeeSeti zedatha
1) Ambry 2) I-Couchbase 3) Dalids 4) I-Espresso 5) HDFS 6) Hive 7) Kafka 8) MongoDB 9) MySQL 10) Oracle 11) Pinot 12) Presto 12) Yiba 13) Teradata 13) Vector 14) yaseVenice
Hive Kafka RDBMS

I-Pub-sub
LinkedIn Kafka
I-Kafka edibeneyo

Ukusasazwa koMsinga
aphethwe
Ifakwe (izimele)

Isitofu esiXhomekekileyo kunye noBumbeko oluDynamic
LinkedIn Offspring
intlakohlaza

Yakha izixhobo
I-Ligradle (Isonga seGradle sangaphakathi seLinkedIn)
Gradlew

CI / CD
CRT (I-LinkedIn's yangaphakathi CI/CD)
TravisCI kwaye Ihabhu yedokodo

IiVenkile zeMetadata
Ukusasazwa kwe-GMS ezininzi: 1) Iseti yedatha ye-GMS 2) Umsebenzisi we-GMS 3) I-Metric GMS 4) Inqaku le-GMS 5) Itshathi/iDashboard ye-GMS
I-GMS enye: 1) Iiseti zedatha 2) Abasebenzisi

Iinkonzo ezincinci kwizikhongozeli zeDocker

Docker yenza lula ukusasazwa kunye nokuhanjiswa kwesicelo nge ukufakwa kwezingxobo. Yonke inxalenye yenkonzo kwiDataHub ngumthombo ovulekileyo, kubandakanywa amacandelo eziseko ezifana neKafka, Elasticsearch, neo4j ΠΈ MySQL, inomfanekiso wayo weDocker. Ukucwangcisa izitya zeDocker esizisebenzisileyo Docker Ukubhala.

I-DataHub yoMthombo oVulekileyo: Ukukhangela kweMetadata ye-LinkedIn kunye nePlatifomu yokuFumana

Umzobo 2: Uyilo lwezakhiwo DataHub *Vula Umnikezi**

Uyakwazi ukubona i-architecture yezinga eliphezulu leDathaHub kumfanekiso ongentla. Ngaphandle kwezinto ezisisiseko, inezikhongozeli ezine ezahlukeneyo zeDocker:

datahub-gms: inkonzo yokugcina imetadata

datahub-frontend: isicelo umdlalo, ukukhonza iDataHub interface.

datahub-mce-umthengi: isicelo Kafka imijelo, esebenzisa isiganeko sokutshintsha imethadatha (MCE) umjelo kunye nokuhlaziya i-metadata store.

datahub-mae-umthengi: isicelo Kafka imijelo, esebenzisa i-metadata audit event stream (MAE) kwaye yenza isalathisi sokukhangela kunye nedatha yegrafu.

Vula umthombo wogcino amaxwebhu kunye iposti yebhlog yeDathaHub ziqulathe iinkcukacha ezithe vetshe malunga nemisebenzi yeenkonzo ezahlukeneyo.

I-CI/CD kwi-DataHub ngumthombo ovulekileyo

Umthombo ovulekileyo wogcino lweDataHub usebenzisa TravisCI ukuhlanganiswa okuqhubekayo kunye Ihabhu yedokodo ukusasazwa rhoqo. Zombini zinokudityaniswa okuhle kweGitHub kwaye kulula ukuseta. Kwiziseko zoncedo ezininzi ezivulelekileyo eziphuhliswe luluntu okanye iinkampani zabucala (umz. Ukudibana), imifanekiso yeDocker yenziwe kwaye ifakwe kwi-Docker Hub ukuze kube lula ukusetyenziswa luluntu. Nawuphi na umfanekiso weDocker ofunyenwe kwiDocker Hub unokusetyenziswa ngokulula ngomyalelo olula docker tsala.

Ngokuzibophelela ngakunye kwiDataHub evulekileyo yokugcina umthombo, yonke imifanekiso yeDocker yakhiwe ngokuzenzekelayo kwaye ibekwe kwi-Docker Hub enethegi "yamva nje". Ukuba i-Docker Hub iqwalaselwe nabanye ukubiza amasebe abonisa rhoqo, zonke iithegi kwindawo yokugcina umthombo ovulekileyo nazo zikhutshwe ngamagama ethegi ahambelanayo kwiDocker Hub.

Ukusebenzisa i-DataHub

Ukumisela i-DataHub ilula kakhulu kwaye inamanyathelo amathathu alula:

  1. Vala indawo yokugcina umthombo ovulekileyo kwaye uqhube zonke izitya ze-Docker kunye ne-docker-compose usebenzisa i-docker-compose script enikiweyo ukuqala ngokukhawuleza.
  2. Khuphela idatha yesampula enikezelweyo kwindawo yokugcina usebenzisa isixhobo somgca womyalelo onikwe kwakhona.
  3. Bhrawuza iDataHub kwisikhangeli sakho.

Ilandelwa ngokukhutheleyo Gitter incoko nayo iqwalaselwe imibuzo ekhawulezayo. Abasebenzisi banokwenza imiba ngokuthe ngqo kwindawo yokugcina i-GitHub. Okona kubaluleke kakhulu, siyayamkela kwaye siyayixabisa yonke impendulo kunye neengcebiso!

Izicwangciso zekamva

Okwangoku, zonke iziseko okanye i-microservice yomthombo ovulekileyo weDataHub yakhiwe njengesitya seDocker, kwaye yonke inkqubo ilungelelaniswe kusetyenziswa. Dibanisa idokhethi. Ngenxa yokuthandwa kunye nokusasazeka Kubernetes, singathanda kwakhona ukubonelela ngesisombululo esisekwe kwi-Kubernetes kwixesha elizayo elingekude.

Siceba kwakhona ukubonelela ngesisombululo se-turnkey yokuhambisa i-DataHub kwinkonzo yefu yoluntu njenge I-Azure, AWS okanye Ifu leGoogle. Ngenxa yesibhengezo samva nje sokufuduka kwe-LinkedIn ukuya e-Azure, oku kuya kuhambelana nezinto eziphambili zeqela lemethadatha.

Okokugqibela kodwa okungancinci, enkosi kubo bonke abamkeli bokuqala beDataHub kwindawo yomthombo ovulekileyo abaye balinganisela i-DataHub alphas kwaye basinceda ukuba sibone imiba kunye nokuphucula amaxwebhu.

umthombo: www.habr.com

Yongeza izimvo