Lokhu okuqukethwe kuchaza ithuluzi elilula nelishesha kakhulu lokuthola idatha, umsebenzi owubona ku-KDPV. Kuyathakazelisa ukuthi i-whale yakhelwe ukuthi isingathwe kuseva ye-git ekude. Imininingwane ngaphansi kokusikwa.
Indlela Ithuluzi Lokutholwa Kwedatha le-Airbnb Elishintshe Impilo Yami
Ngibe nenhlanhla yokusebenzela ezinkingeni ezithile ezijabulisayo emsebenzini wami: Ngafunda i-thread mathematics ngenkathi ngithola iziqu zami e-MIT, ngasebenza kumamodeli akhuphukayo kanye nephrojekthi yomthombo ovulekile. e-Wayfair, kanye nokusebenzisa amamodeli amasha okuqondiswe kuwo ikhasi eliyisiqalo kanye nokuthuthukiswa kwe-CUPED kwa-Airbnb. Kodwa wonke lo msebenzi awuzange ube muhle - empeleni, ngangivame ukuchitha isikhathi sami esiningi ngicinga, ngifunda futhi ngiqinisekisa idatha. Nakuba lesi kwakuyisimo esiqhubekayo emsebenzini, akuzange kungifikele ukuthi lokhu kube inkinga kwaze kwaba yilapho ngifika ku-Airbnb, lapho kwaxazululwa khona kusetshenziswa ithuluzi lokuthola idatha - .
Ngingayithola kuphi i-{{data}}? Idathaportal.
Isho ukuthini le kholomu? Idathaportal.
Injani i-{{metric}} namuhla? Idathaportal.
Uyini umuzwa wokuphila? IN Idathaportal, mhlawumbe.
Kulungile, unaso isithombe. Ukuthola idatha nokuqonda ukuthi isho ukuthini, ukuthi idalwe kanjani, nokuthi isetshenziswa kanjani konke kuthatha imizuzu, hhayi amahora. Ngingachitha isikhathi sami ngenza iziqondiso ezilula, noma ama-algorithms amasha, (... noma ngiphendule imibuzo engahleliwe mayelana nedatha) kunokuphenya amanothi, ngibhale imibuzo ephindaphindayo ye-SQL, futhi ngikhulume ngozakwethu ku-Slack ukuze ngizame ukuphinda ngidale umongo, omunye umuntu. kakade.
Yini inkinga?
Ngabona ukuthi iningi labangane bami lalingenalo ithuluzi elinjalo. Zimbalwa izinkampani ezifuna ukunikela ngezinsiza ezinkulu ekudaleni nasekugcineni ithuluzi lesikhulumi njengeDathaportal. Futhi nakuba kunezixazululo ezimbalwa zomthombo ovulekile ezitholakalayo, ngokuvamile zenzelwe ukukala, okwenza ukusetha nokugcinwa kube nzima ngaphandle konjiniyela we-DevOps ozinikele. Ngakho nginqume ukudala into entsha.
Umkhomo: ithuluzi lokuthola idatha elilula ngobuwula
Futhi yebo, ngokulula kuze kube sezingeni lobuwula, ngiqonde ukuthi kulula kuze kube sezingeni lobuwula. Umkhomo unezingxenye ezimbili kuphela:
- Umtapo wezincwadi wePython oqoqa imethadatha futhi uyifomethe ku-MarkDown.
- Isixhumanisi somugqa womyalo we-Rust sokusesha le datha.
Ngokombono wengqalasizinda yangaphakathi yokulungiswa, kunamafayela ombhalo amaningi kuphela nohlelo olubuyekeza umbhalo. Yilokho, ngakho-ke ukubamba kuseva ye-git njenge-Github kuyinto encane. Alukho ulimi olusha lombuzo olungafundwa, azikho ingqalasizinda yokuphatha, azikho izipele. I-Git yaziwa yiwo wonke umuntu, ngakho ukuvumelanisa nokusebenzisana kumahhala. Ake sihlolisise ukusebenza .
I-GUI esekelwe ngokugcwele ye-git
Umkhomo uklanyelwe ukubhukuda olwandle lwe-remote git server. Yena okwenziwe ngendlela oyifisayo: chaza okunye ukuxhumana, kopisha umbhalo we-Github Actions (noma ubhale eyodwa yengxenyekazi yakho ye-CI/CD oyikhethayo) futhi uzoba nethuluzi lokuthola idatha elisekelwe kuwebhu ngokushesha. Uzokwazi ukusesha, ukubuka, ukubhala, futhi wabelane ngamaspredishithi akho ngokuqondile ku-Github.
Isibonelo setafula le-stub elakhiwe kusetshenziswa Izenzo ze-Github. Idemo egcwele yokusebenza .
Usesho olusheshayo lwe-CLI lwenqolobane yakho
Umkhomo uphila futhi uphefumula umugqa womyalo, ukuhlinzeka ngosesho olunamandla, lwe-millisecond kuwo wonke amatafula akho. Noma sinezigidi zamatafula, sikwazile ukwenza umkhomo usebenze ngendlela emangalisayo ngokusebenzisa izindlela ezihlakaniphile zokulondoloza isikhashana, kanye nokwakha kabusha ingemuva eRust. Ngeke ubone noma yikuphi ukunensa kokusesha [sawubona, Google DS].
Ukuboniswa komkhomo, sesha amatafula ayisigidi.
Ukubala okuzenzakalelayo kwamamethrikhi [ku-beta]
Enye yezinto engingazithandi kakhulu njengososayensi wedatha usebenzisa imibuzo efanayo kaningi ukuze ahlole ikhwalithi yedatha esetshenziswayo. Umkhomo usekela ikhono lokuchaza amamethrikhi nge-SQL elula ezohlelelwa ukusebenza eduze kwamapayipi akho okukhuhla imethadatha. Chaza ibhulokhi lamamethrikhi ngefomethi ye-YAML ngaphakathi kwetafula le-stub, futhi i-Whale izosebenza ngokuzenzakalelayo kushejuli futhi iphendule imibuzo ebekwe ngaphakathi kwamamethrikhi.
```metrics
metric-name:
sql: |
select count(*) from table
```
Ihlanganiswe ne-Github, le ndlela isho ukuthi umkhomo ungasebenza njengomthombo olula omaphakathi weqiniso wezincazelo zemethrikhi. Umkhomo ugcina ngisho amanani kanye nesitembu sesikhathi kumkhombandlela othi "~/". umkhomo/amamethrikhi" uma ufuna ukwenza uhlobo oluthile lwegrafu noma ucwaningo olujulile.
Ikusasa
Ngemva kokukhuluma nabasebenzisi bezinguqulo zethu zomkhomo ezikhishwe ngaphambilini, sibonile ukuthi abantu badinga ukusebenza okubanzi. Kungani ithuluzi lokusesha ithebula? Kungani kungelona ithuluzi lokusesha amamethrikhi? Kungani ungaqapheli? Kungani kungelona ithuluzi lokubuza le-SQL? Nakuba i-whale v1 ekuqaleni yayihloselwe ukuba ithuluzi elilula elingumngane we-CLI Dataportal/Amundsen, isivele ikhule yaba inkundla ezimele esebenza ngokugcwele, futhi sithemba ukuthi izoba ingxenye ebalulekile yekhithi yamathuluzi yososayensi wedatha.
Uma kukhona ofuna ukukubona ekuthuthukisweni, joyina yethu , vula Izinkinga ngo , noma uthinte ngokuqondile . Sesivele sinenani lezici ezinhle - izifanekiso ze-Jinja, amabhukumaka, izihlungi zosesho, izexwayiso ze-Slack, ukuhlanganiswa kwe-Jupyter, ngisho nedeshibhodi ye-CLI yamamethrikhi - kodwa singakuthanda okufakile kwakho.
isiphetho
Umkhomo uthuthukiswa futhi usekelwa i-Dataframe, isiqalisi engisanda kuba nenjabulo yokusisungula nabanye. Ngenkathi i-whale yakhelwe ososayensi bedatha, i-Dataframe yakhelwe amaqembu esayensi yedatha. Kulabo abafuna ukusebenzelana eduze kakhulu, zizwe ukhululekile , sizokwengeza ohlwini lokulinda.
Futhi ngekhodi yephromo I-HABR, ungathola u-10% owengeziwe esaphulelo esikhonjiswe kusibhengezo.
Izifundo ezengeziwe
Izihloko Ezifakiwe
Source: www.habr.com