Buɗe Source DataHub: Binciken Metadata na LinkedIn da Platform Ganewa
Nemo bayanan da kuke buƙata cikin sauri yana da mahimmanci ga kowane kamfani da ya dogara da adadi mai yawa don yanke shawara ta hanyar bayanai. Ba wai kawai wannan yana tasiri tasirin masu amfani da bayanai ba (ciki har da manazarta, masu haɓaka na'ura, masana kimiyyar bayanai, da injiniyoyin bayanai), amma kuma yana da tasiri kai tsaye akan ƙarshen samfuran waɗanda suka dogara da bututun injin koyo (ML). Bugu da ƙari, yanayin aiwatarwa ko gina dandamali na koyon injin yana haifar da tambaya: menene hanyar ku don gano fasali, ƙira, awo, bayanan bayanai, da sauransu.
A cikin wannan labarin za mu yi magana game da yadda muka buga tushen bayanai a ƙarƙashin buɗaɗɗen lasisi a cikin dandalin bincike da gano metadata, farawa daga farkon kwanakin aikin . LinkedIn yana kiyaye nau'in nasa na DataHub daban daga sigar buɗaɗɗen tushe. Za mu fara da bayanin dalilin da yasa muke buƙatar wurare daban-daban na ci gaba guda biyu, sannan mu tattauna hanyoyin farko don amfani da buɗaɗɗen tushen WhereHows kuma kwatanta nau'in DataHub na ciki (samuwar) tare da sigar akan. . Za mu kuma raba cikakkun bayanai game da sabuwar hanyar mu ta atomatik don turawa da karɓar sabuntawar buɗaɗɗen tushe don kiyaye ma'ajiyar duka biyu cikin aiki tare. A ƙarshe, za mu ba da umarni kan yadda ake farawa ta amfani da buɗaɗɗen tushen DataHub kuma a taƙaice tattauna gine-ginensa.
IndaHows yanzu shine DataHub!
An gabatar da ƙungiyar metadata ta LinkedIn a baya (magaji zuwa WhereHows), bincike na LinkedIn da dandalin gano metadata, da shirye-shiryen buɗe shi. Ba da daɗewa ba bayan wannan sanarwar, mun fitar da sigar alpha na DataHub kuma mun raba shi ga al'umma. Tun daga nan, mun ci gaba da ba da gudummawa ga ma'ajiyar kuma muna aiki tare da masu amfani don ƙara abubuwan da aka fi nema da warware matsaloli. Yanzu muna farin cikin sanar da sakin a hukumance .
Buɗe Hanyar Hanyoyi
WhereHows, asalin hanyar sadarwar LinkedIn don nemo bayanai da kuma inda ta fito, ta fara ne azaman aikin cikin gida; ƙungiyar metadata ta buɗe shi . Tun daga wannan lokacin, ƙungiyar ta kasance koyaushe tana kiyaye maɓalli daban-daban guda biyu-ɗaya don buɗe tushen kuma ɗaya don amfanin cikin gida na LinkedIn-kamar ba duk fasalulluka na samfuran da aka haɓaka don shari'o'in amfani da LinkedIn gabaɗaya sun dace da masu sauraro ba. Bugu da ƙari, WhereHows yana da wasu abubuwan dogaro na ciki (kayan aiki, dakunan karatu, da sauransu) waɗanda ba buɗaɗɗen tushe ba. A cikin shekarun da suka biyo baya, IndaHows ya bi ta hanyoyi da yawa da zagayowar ci gaba, yana mai da kiyaye codebases guda biyu cikin daidaitawa babban ƙalubale. Ƙungiyar metadata ta gwada hanyoyi daban-daban tsawon shekaru don ƙoƙarin kiyaye ci gaban ciki da buɗe tushen ci gaba a daidaitawa.
Gwada farko: "Bude tushen farko"
Da farko mun bi tsarin ci gaba na "budewar tushen farko", inda yawancin ci gaba ke faruwa a cikin buɗaɗɗen ma'ajiya kuma ana yin canje-canje don tura ciki. Matsalar wannan hanyar ita ce koyaushe ana tura lambar zuwa GitHub da farko kafin a sake nazarinta gabaɗaya a ciki. Har sai an yi canje-canje daga ma'ajiyar buɗaɗɗen tushe kuma an yi sabon turawar cikin gida, ba za mu sami wasu batutuwan samarwa ba. Idan ba a yi aiki da kyau ba, yana da matukar wahala a iya tantance wanda ya aikata laifin saboda an yi canje-canje a batches.
Bugu da ƙari, wannan ƙirar ta rage yawan haɓakar ƙungiyar yayin haɓaka sabbin fasalolin da ke buƙatar jujjuyawa cikin sauri, tunda ya tilasta duk canje-canje da aka fara tura su zuwa ma'ajiyar buɗaɗɗen tushe sannan tura zuwa ma'ajiyar ciki. Don rage lokacin aiki, ana iya yin gyara ko canji da ake buƙata a cikin ma'ajiyar ciki da farko, amma wannan ya zama babbar matsala idan aka zo haɗa waɗannan canje-canjen zuwa ma'ajiyar buɗaɗɗen tushe saboda ma'ajiyar biyu ba su daidaita ba.
Wannan samfurin ya fi sauƙi don aiwatarwa don dandamali na raba, dakunan karatu, ko ayyukan samar da ababen more rayuwa fiye da cikakkun kayan aikin yanar gizo na al'ada. Bugu da ƙari, wannan ƙirar ya dace don ayyukan da suka fara buɗe tushen daga rana ɗaya, amma WhereHows an gina shi azaman aikace-aikacen gidan yanar gizo gaba ɗaya na ciki. Yana da matukar wahala a cire duk abubuwan dogaro na ciki gaba daya, don haka muna buƙatar kiyaye cokali mai yatsa, amma kiyaye cokali mai yatsa na ciki da haɓaka galibi buɗe tushen bai yi aiki ba.
Ƙoƙari na biyu: "Cikin farko"
**A matsayin ƙoƙari na biyu, mun ƙaura zuwa samfurin ci gaba na "ciki na farko", inda yawancin ci gaba ke faruwa a cikin gida kuma ana yin canje-canje ga lambar tushe akai-akai. Kodayake wannan samfurin ya fi dacewa da yanayin amfaninmu, yana da matsaloli na asali. Tura duk bambance-bambance kai tsaye zuwa wurin buɗaɗɗen tushe sannan kuma ƙoƙarin warware rikice-rikice daga baya zaɓi ne, amma yana ɗaukar lokaci. Masu haɓakawa a mafi yawan lokuta suna ƙoƙarin kada su yi hakan duk lokacin da suka sake duba lambar su. Sakamakon haka, ba za a yi hakan ba akai-akai, a cikin batches, don haka yana da wahala a warware rikice-rikicen haɗuwa daga baya.
A karo na uku ya yi aiki!
Ƙoƙarin yunƙuri biyu da aka gaza ambata a sama sun haifar da ma'ajiyar WhereHows GitHub da ta rage ta zamani na dogon lokaci. Tawagar ta ci gaba da inganta fasalulluka da gine-ginen samfurin, ta yadda sigar ciki ta WhereHows don LinkedIn ya zama mafi ci gaba fiye da buɗaɗɗen sigar. Har ma yana da sabon suna - DataHub. Dangane da yunƙurin da ba a yi nasara ba a baya, ƙungiyar ta yanke shawarar haɓaka mai daidaitawa, mafita na dogon lokaci.
Ga kowane sabon aikin buɗaɗɗen tushe, ƙungiyar buɗe tushen LinkedIn tana ba da shawara da goyan bayan ƙirar ci gaba wanda a cikinsa aka haɓaka samfuran aikin gaba ɗaya a buɗaɗɗen tushe. Ana tura sigar kayan tarihi zuwa ma'ajiyar jama'a sannan a duba su cikin wani kayan tarihi na ciki na LinkedIn ta amfani da . Bin wannan ƙirar haɓaka ba wai kawai yana da kyau ga waɗanda ke amfani da buɗaɗɗen tushe ba, amma har ma yana haifar da ingantaccen tsarin gine-gine na zamani, wanda za'a iya cirewa, da toshewa.
Koyaya, babban aikace-aikacen ƙarshen baya kamar DataHub zai buƙaci babban adadin lokaci don isa wannan jihar. Wannan kuma yana hana yuwuwar buɗe tushen aiwatar da cikakken aiki kafin duk abin dogaro na cikin gida ya kasance cikakke. Shi ya sa muka ƙirƙiro kayan aikin da ke taimaka mana ba da gudummawar buɗaɗɗen tushe cikin sauri kuma tare da ƙarancin zafi. Wannan maganin yana amfana da ƙungiyar metadata (mai haɓaka DataHub) da kuma buɗe tushen al'umma. Sashe na gaba za su tattauna wannan sabuwar hanyar.
Buɗe Tushen Buga Automation
Sabuwar hanyar ƙungiyar Metadata zuwa tushen buɗaɗɗen DataHub shine haɓaka kayan aiki wanda ke daidaita ma'ajin lambar ciki ta atomatik da ma'ajiyar buɗaɗɗen tushe. Babban fasali na wannan kayan aikin sun haɗa da:
- Daidaita lambar LinkedIn zuwa/daga buɗaɗɗen tushe, makamancin haka .
- Ƙirƙirar lasifikan kai, kama da .
- Ƙirƙiri ta atomatik buɗaɗɗen buɗaɗɗen rajistan ayyukan aikata rajistan ayyukan ciki.
- Hana canje-canje na ciki waɗanda ke karya tushen buɗewa da ke ginawa ta .
Ƙasashe masu zuwa za su shiga cikin ayyukan da aka ambata a sama waɗanda ke da matsaloli masu ban sha'awa.
Aiki tare da lambar tushe
Ba kamar buɗaɗɗen tushen DataHub ba, wanda shine ma'ajiyar GitHub guda ɗaya, sigar LinkedIn ta DataHub haɗe ce ta ma'ajiyar ma'auni (wanda ake kira a ciki). ). Ƙididdigar DataHub, ɗakin karatu na samfurin metadata, sabis na baya na ma'ajin metadata, da ayyukan yawo suna zaune a cikin ma'ajiya daban-daban akan LinkedIn. Koyaya, don sauƙaƙawa ga masu amfani da tushen tushe, muna da wurin ajiya guda ɗaya don buɗaɗɗen sigar DataHub.
Hoto 1: Aiki tare tsakanin ma'aji LinkedIn DataHub da rumbun ajiya guda daya DataHub tushen budewa
Don tallafawa ginawa ta atomatik, turawa, da ja da ayyukan aiki, sabon kayan aikin mu yana ƙirƙirar taswirar matakin-fayil ta atomatik daidai da kowane fayil na tushe. Koyaya, kayan aikin kayan aikin yana buƙatar saitin farko kuma dole ne masu amfani su samar da babban taswirar ƙirar ƙirar kamar yadda aka nuna a ƙasa.
{
"datahub-dao": [
"${datahub-frontend}/datahub-dao"
],
"gms/impl": [
"${dataset-gms}/impl",
"${user-gms}/impl"
],
"metadata-dao": [
"${metadata-models}/metadata-dao"
],
"metadata-builders": [
"${metadata-models}/metadata-builders"
]
}Taswirar matakin-module JSON mai sauƙi ne wanda maɓallan su ne maƙallan da aka yi niyya a cikin ma'ajin buɗaɗɗen tushe kuma ƙimar su ne jerin samfuran tushen a cikin ma'ajin LinkedIn. Duk wani nau'in manufa a cikin buɗaɗɗen ma'ajiyar tushe za a iya ciyar da shi ta kowane adadin kayan masarufi. Don nuna sunayen ciki na ma'ajiyar ma'ajiyar bayanai a cikin kayan masarufi, yi amfani in Bash style. Yin amfani da fayil ɗin taswirar matakin-module, kayan aikin suna ƙirƙirar fayil ɗin taswirar matakin-fayil ta hanyar bincika duk fayiloli a cikin kundayen adireshi masu alaƙa.
{
"${metadata-models}/metadata-builders/src/main/java/com/linkedin/Foo.java":
"metadata-builders/src/main/java/com/linkedin/Foo.java",
"${metadata-models}/metadata-builders/src/main/java/com/linkedin/Bar.java":
"metadata-builders/src/main/java/com/linkedin/Bar.java",
"${metadata-models}/metadata-builders/build.gradle": null,
}Ana yin taswirar matakin fayil ta atomatik ta kayan aikin; duk da haka, kuma za a iya sabunta shi da hannu ta mai amfani. Wannan shine taswirar 1:1 na fayil ɗin tushen LinkedIn zuwa fayil a cikin buɗaɗɗen ma'ajiya. Akwai dokoki da yawa masu alaƙa da wannan ƙirƙirar ƙungiyoyin fayil ta atomatik:
- Dangane da nau'ikan tushen tushe da yawa don tsarin manufa a buɗaɗɗen tushe, rikice-rikice na iya tasowa, misali iri ɗaya , data kasance a cikin tsarin tushen fiye da ɗaya. A matsayin dabarun warware rikice-rikice, kayan aikinmu sun saba zuwa zaɓi na "lashe na ƙarshe".
- "null" yana nufin cewa tushen fayil ɗin baya cikin wurin buɗaɗɗen ma'ajin.
- Bayan kowane buɗaɗɗen tushe ko cirewa, ana sabunta wannan taswira ta atomatik kuma ana ƙirƙirar hoto. Wannan yana da mahimmanci don gano ƙari da gogewa daga lambar tushe tun lokacin aikin ƙarshe.
Ƙirƙirar yin rajistan ayyukan
Ƙaddamar da rajistan ayyukan don buɗaɗɗen tushen aikatawa ana kuma haifar da su ta atomatik ta hanyar haɗa rajistan ayyukan ma'ajiyar ciki. A ƙasa akwai samfurin ƙaddamarwa don nuna tsarin tsarin aikin da kayan aikinmu ya samar. Alkawari yana nuna a sarari waɗanne nau'ikan ma'ajin madogaran tushen ne aka tattara a waccan alkawari kuma yana ba da taƙaitaccen tarihin ƙaddamarwa. Duba wannan ta yin amfani da ainihin misali na ƙaƙƙarfan log ɗin da aka samar ta kayan aikin mu.
metadata-models 29.0.0 -> 30.0.0
Added aspect model foo
Fixed issue bar
dataset-gms 2.3.0 -> 2.3.4
Added rest.li API to serve foo aspect
MP_VERSION=dataset-gms:2.3.4
MP_VERSION=metadata-models:30.0.0Gwajin dogaro
LinkedIn yana da , wanda ke taimakawa wajen tabbatar da cewa canje-canje zuwa samfura masu yawa na ciki ba su karya haɗuwar samfuran da aka dogara da su ba. Ma'ajiyar buɗaɗɗen ma'ajin DataHub ba samfura masu yawa ba ne, kuma ba zai iya zama dogaro kai tsaye na kowane samfuri da yawa ba, amma tare da taimakon naɗaɗɗen samfura da yawa wanda ke ɗauko lambar tushe na tushen DataHub, har yanzu muna iya amfani da wannan gwajin dogaro. Don haka, duk wani canji (wanda daga baya za a iya fallasa shi) ga kowane ɗayan samfuran da ke ciyar da buɗaɗɗen tushen ma'ajin DataHub yana haifar da ginin ginin a cikin samfuran harsashi da yawa. Don haka, duk wani canjin da ya kasa gina samfurin kundi ya gaza yin gwaje-gwaje kafin yin ainihin samfurin kuma ana komawa.
Wannan wata hanya ce mai fa'ida wacce ke taimakawa hana duk wani aiki na ciki wanda ya karya ginin tushen buɗaɗɗen kuma gano shi a lokacin ƙaddamarwa. Idan ba tare da wannan ba, zai zama da wahala a tantance wanne alƙawarin ciki ya haifar da ginin ma'adanar tushen buɗe ƙasa, saboda muna daidaita canje-canje na ciki zuwa ma'ajiyar buɗe tushen DataHub.
Bambance-bambance tsakanin buɗaɗɗen tushen DataHub da sigar samarwa mu
Har zuwa wannan batu, mun tattauna mafitarmu don daidaita nau'ikan ma'ajin DataHub guda biyu, amma har yanzu ba mu bayyana dalilan da ya sa muke buƙatar rafukan ci gaba daban-daban guda biyu ba a farkon wuri. A cikin wannan sashe, za mu lissafa bambance-bambance tsakanin sigar jama'a na DataHub da sigar samarwa akan sabar LinkedIn, kuma mu bayyana dalilan waɗannan bambance-bambance.
Ɗaya daga cikin tushen rashin daidaituwa ya samo asali daga gaskiyar cewa nau'in samar da mu yana da dogaro akan lambar da ba ta buɗe tushen ba, kamar Zuriyar LinkedIn (Tsarin allurar dogaro na ciki na LinkedIn). Ana amfani da zuriya ko'ina a cikin codebases saboda ita ce hanyar da aka fi so don sarrafa tsauri mai ƙarfi. Amma ba buɗaɗɗen tushe ba ne; don haka muna buƙatar nemo hanyoyin buɗaɗɗen madadin zuwa tushen buɗaɗɗen DataHub.
Akwai wasu dalilai kuma. Yayin da muke ƙirƙira kari ga ƙirar metadata don buƙatun LinkedIn, waɗannan kari yawanci keɓaɓɓu ne ga LinkedIn kuma ƙila ba za su shafi wasu wurare kai tsaye ba. Misali, muna da takamaiman tambari don ID na ɗan takara da sauran nau'ikan metadata masu dacewa. Don haka, yanzu mun cire waɗannan kari daga ƙirar metadata na buɗe tushen DataHub. Yayin da muke hulɗa da al'umma kuma muna fahimtar bukatunsu, za mu yi aiki a kan buɗaɗɗen tushe gama gari na waɗannan kari a inda ake buƙata.
Sauƙin amfani da sauƙin daidaitawa ga buɗaɗɗen tushen al'umma kuma ya ƙarfafa wasu bambance-bambance tsakanin nau'ikan DataHub guda biyu. Bambance-bambancen kayan aikin sarrafa rafi shine kyakkyawan misali na wannan. Kodayake sigar mu ta ciki tana amfani da tsarin sarrafa rafi da aka sarrafa, mun zaɓi yin amfani da ginanniyar sarrafa rafi (a tsaye) don buɗaɗɗen sigar tushe saboda yana guje wa ƙirƙirar wani dogaro da kayan aikin.
Wani misali na bambancin shine samun GMS guda ɗaya (Babban Shagon Metadata na Gabaɗaya) a cikin aiwatar da buɗaɗɗen tushe maimakon GMS da yawa. GMA (Generalized Metadata Architecture) shine sunan tsarin gine-ginen baya na DataHub, kuma GMS shine ma'ajin metadata a cikin mahallin GMA. GMA tsarin gine-gine ne mai sassauƙa wanda ke ba ka damar rarraba kowane ginin bayanai (misali madaidaitan bayanai, masu amfani, da sauransu) a cikin ma'ajiyar metadata ta kansa, ko adana abubuwan gina bayanai da yawa a cikin ma'ajin metadata guda ɗaya muddin wurin yin rajista yana ɗauke da tsarin taswirar bayanai a ciki. An sabunta GMS. Don sauƙin amfani, mun zaɓi misalin GMS guda ɗaya wanda ke adana duk ginin bayanai daban-daban a cikin buɗaɗɗen tushen DataHub.
An ba da cikakken jerin bambance-bambance tsakanin aiwatarwa biyu a cikin tebur da ke ƙasa.
Product Features
LinkedIn DataHub
Buɗe Source DataHub
Tallafin Bayanan Gina
1) Bayanan bayanai 2) Masu amfani 3) Ma'auni 4) Fasali na ML 5) Charts 6) Dashboards
1) Datasets 2) Masu amfani
Tushen Metadata Masu Goyan bayan don Saitin Bayanai
1) 2) Kofi 3) 4) 5) HDFS 6) Hive 7) Kafka 8) MongoDB 9) MySQL 10) Oracle 11) 12) 12) 13) Taradata 13) Vector 14)
Hive Kafka RDBMS
Pub-sub
Confluent Kafka
Gudanar da Ruwa
gudanar
Ƙunƙwasa (na tsaye)
Dogaro Allurar & Tsananin Kanfigareshan
LinkedIn Zuriyar
Gina Kayan aiki
Ligradle (LinkedIn's na ciki Gradle wrapper)
CI / CD
CRT (LinkedIn na ciki CI/CD)
da kuma
Shagunan Metadata
Rarraba GMS da yawa: 1) GMS Dataset 2) GMS mai amfani 3) Metric GMS 4) Fasalin GMS 5) Chart/Dashboard GMS
GMS guda ɗaya don: 1) Bayanan bayanai 2) Masu amfani
Microservices a cikin kwantena Docker
yana sauƙaƙa tura aikace-aikacen da rarrabawa tare da . Kowane bangare na sabis a cikin DataHub buɗaɗɗe ne, gami da abubuwan abubuwan more rayuwa kamar Kafka, , и , yana da nasa hoton Docker. Don tsara kwantena Docker da muka yi amfani da su .
Hoto 2: Gine-gine DataHub *Bude tushen**
Kuna iya ganin babban matakin gine-gine na DataHub a cikin hoton da ke sama. Bayan abubuwan abubuwan more rayuwa, yana da kwantena Docker guda huɗu daban-daban:
datahub-gms: sabis na ajiya na metadata
datahub-frontend: aikace-aikace , bautar da DataHub interface.
datahub-mce-consumer: aikace-aikace , wanda ke amfani da taron canjin metadata (MCE) rafi kuma yana sabunta ma'ajin metadata.
datahub-mae-consumer: aikace-aikace , wanda ke amfani da rafin taron duba bayanan metadata (MAE) kuma yana ƙirƙirar bayanan bincike da bayanan hoto.
Bude takaddun ma'ajiyar tushe da ya ƙunshi ƙarin cikakkun bayanai game da ayyuka na ayyuka daban-daban.
CI/CD akan DataHub buɗaɗɗen tushe ne
Buɗe tushen ma'ajin DataHub yana amfani don ci gaba da haɗin kai da don ci gaba da turawa. Dukansu suna da haɗin gwiwar GitHub mai kyau kuma suna da sauƙin saitawa. Don mafi yawan buɗaɗɗen kayan aikin da al'umma ko kamfanoni masu zaman kansu suka haɓaka (misali. ), An ƙirƙira hotunan Docker kuma an tura su zuwa Docker Hub don sauƙin amfani da al'umma. Duk wani hoton Docker da aka samu a Docker Hub ana iya amfani dashi cikin sauƙi tare da umarni mai sauƙi .
Tare da kowane sadaukarwa ga ma'ajiyar tushen tushen DataHub, duk hotunan Docker ana gina su ta atomatik kuma ana tura su zuwa Docker Hub tare da alamar "sabon". Idan an saita Docker Hub tare da wasu , duk alamun da ke cikin buɗaɗɗen ma'ajin ma'ajiya kuma ana fitar da su tare da sunaye masu dacewa a Docker Hub.
Amfani da DataHub
mai sauqi ne kuma ya ƙunshi matakai masu sauƙi guda uku:
- Rufe wurin buɗaɗɗen ma'ajiyar tushe kuma gudanar da duk kwantena Docker tare da rubutaccen docker ta amfani da rubutun docker-compose da aka bayar don farawa mai sauri.
- Zazzage samfurin bayanan da aka bayar a cikin ma'ajiyar ta amfani da kayan aikin layin umarni wanda kuma aka bayar.
- Nemo DataHub a cikin burauzar ku.
Ana Bibiyar Rayayye Hakanan an saita don tambayoyi masu sauri. Masu amfani kuma suna iya ƙirƙirar batutuwa kai tsaye a cikin ma'ajin GitHub. Mafi mahimmanci, muna maraba da godiya ga duk amsa da shawarwari!
Shirye-shirye na nan gaba
A halin yanzu, kowane kayan aiki ko microservice don buɗe tushen DataHub an gina shi azaman akwati na Docker, kuma an tsara dukkan tsarin ta amfani da . Ganin shaharar da kuma tartsatsi , Muna kuma so mu samar da tushen tushen Kubernetes a nan gaba.
Har ila yau, muna shirin samar da mafita na juyawa don tura DataHub akan sabis na girgije na jama'a kamar , ko . Idan aka ba da sanarwar ƙaura na LinkedIn zuwa Azure, wannan zai dace da abubuwan da ke cikin ƙungiyar metadata.
A ƙarshe amma ba ƙarami ba, godiya ga duk waɗanda suka fara karɓar DataHub a cikin buɗaɗɗen tushen al'umma waɗanda suka ƙididdige alphas na DataHub kuma suka taimaka mana gano batutuwa da haɓaka takardu.
source: www.habr.com