Ichi chinyorwa chinotaura nezve yakapusa uye nekukurumidza data yekutsvaga chishandiso, basa raunoona paKDPV. Sezvineiwo, whale yakagadzirirwa kugarwa pane iri kure git server. Tsanangudzo pasi pekucheka.
Mashandisiro Akaita Airbnb Data Discovery Tool Yakachinja Hupenyu Hwangu
Mubasa rangu, ndakave nemufaro wekushanda pane mamwe matambudziko anonakidza: Ndakadzidza flow math ndichiita dhigirii rangu kuMIT, ndakashanda pamhando dzekuwedzera, uye neyakavhurika sosi chirongwa. paWayfair, uye takashandisa mapeji matsva ekunangana nemamodheru uye CUPED kuvandudzwa paAirbnb. Asi basa rose iri rakanga risingamboshamisi β kutaura zvazviri, ndaiwanzopedza nguva yangu yakawanda ndichitsvaga, ndichitsvaga, uye ndichisimbisa data. Kunyangwe ichi chaive chinogara chiripo pabasa, hazvina kuitika kwandiri kuti iyi yaive nyaya kutozosvikira ndasvika kuAirbnb uko yakagadziriswa nechishandiso chekuwana data β .
Ndingawanepi {{data}}? dataportal.
Ko chikamu ichi chinorevei? dataportal.
{{metric}} iri kuita sei nhasi? dataportal.
Chii chinonzi pfungwa yehupenyu? IN dataportal, zvichida.
Zvakanaka, waratidza mufananidzo. Kutsvaga data uye kunzwisisa zvazvinoreva, kuti yakagadzirwa sei uye mashandisiro azvo zvese zvinotora maminetsi mashoma, kwete maawa. Ini ndaigona kupedza nguva yangu ndichidhirowa mhedziso dzakareruka, kana maalgorithms matsva, (... kana kupindura mibvunzo isina kurongeka nezve data), pane kuchera manotsi, kunyora kudzokorora mibvunzo yeSQL, uye kutaura vandinoshanda navo paSlack kuyedza nekugadzira zvakare mamiriro. had.
Dambudziko nderei?
Ndakaona kuti shamwari dzangu zhinji dzaisawana mudziyo wakadaro. Mashoma makambani anoda kupa zviwanikwa zvakakura pakuvaka nekuchengetedza chikuva chepuratifomu seDataportal. Uye kunyangwe paine mashoma akavhurika sosi mhinduro, anowanzo kuve akagadzirirwa kuyera, zvichiita kuti zviome kumisikidza nekuchengetedza pasina akazvipira DevOps injinjini. Saka ndakafunga kugadzira chimwe chinhu chitsva.
Whale: Chishandiso chakapusa chekutsvaga data
Uye hongu, nekupusa kupusa ndinoreva zvakapusa zviri nyore. Whale ine zvikamu zviviri chete:
- Raibhurari yePython inounganidza metadata uye inoigadzira muMarkDown.
- Rust command line interface yekutsvaga kuburikidza neiyi data.
Kubva pakuona kwezvivakwa zvemukati zvekugadzirisa, pane zvakawanda chete zvinyorwa zvinyorwa uye purogiramu inogadzirisa zvinyorwa. Ndizvozvo, saka kubata pane git server seGithub idiki. Hapana mutauro mutsva wemubvunzo wekudzidza, hapana manejimendi manejimendi, hapana backups. Wese munhu anoziva Git, saka syncing uye kubatana ndezvemahara. Ngatitarisei zvakanyanya kushanda .
Yakazara inoratidzwa git-yakavakirwa GUI
Whale yakagadzirirwa kushambira mugungwa reiri kure git server. Iye inogadzirika: tsanangura mamwe maratidziro, tevedzera Github Zviito script (kana nyora imwe yeyako yakasarudzwa CI/CD papuratifomu) uye iwe unenge uine data yekutsvaga yewebhu chishandiso ipapo ipapo. Iwe uchakwanisa kutsvaga, kuona, kunyora uye kugovera maspredishiti ako zvakananga paGithub.
Muenzaniso wetafura ye stub yakagadzirwa uchishandisa Github Zviito. Demo rakazara rekushanda .
Mheni inokurumidza CLI kutsvaga yako repository
Whale inorarama uye inofema pamutsetse wekuraira, ichipa ane simba, millisecond lookups pamatafura ako. Kunyangwe nemamiriyoni ematafura, isu takakwanisa kuita kuti whale iite zvinoshamisa nekushandisa mamwe maitiro ehungwaru caching uye zvakare nekuvakazve backend muRust. Iwe hauzoona kunonoka kutsvaga [mhoro Google DS].
Whale demo, mamirioni ekutarisa tafura.
Kuverengera otomatiki kwemametrics [mubeta]
Chimwe chezvinhu zvandinonyanya kufarira sesainzi wedata ari kumhanyisa mibvunzo yakafanana kakawanda uyezve kungotarisa mhando yedata riri kushandiswa. Whale inotsigira kugona kutsanangura metrics mune yakajeka SQL iyo icharongwa kuti imhanye pamwe nemapaipi ako ekuchenesa metadata. Tsanangura iyo YAML metrics block mukati meiyo stub tafura, uye Whale inomhanya otomatiki pane hurongwa uye inomhanyisa mibvunzo yakaiswa mumametrics.
```metrics
metric-name:
sql: |
select count(*) from table
```
Yakasanganiswa neGithub, nzira iyi inoreva kuti whale inogona kushanda sechinhu chiri nyore chepakati chechokwadi kune metric tsananguro. Whale inotochengetedza kukosha pamwe chete netimestamp mu "~/. whale/metrics" kana iwe uchida kuita mamwe machati kana zvimwe zvakadzama tsvagiridzo.
Ramangwana
Mushure mekutaura nevashandisi vedu vasati vaburitswa mavhezheni ewhale, takaona kuti vanhu vaida kushanda kwakawanda. Sei tafura yekutarisa chishandiso? Wadii kushandisa metrics yekutsvaga? Wadii kutarisa? Sei usiri SQL query execution chishandiso? Nepo whale v1 yakatanga kuumbwa seyakapusa CLI shamwari chishandiso Dataportal/Amundsen, yakatoshanduka kuita chikuva chakamira chakazara, uye tinovimba ichave chikamu chakakosha cheData Scientist's toolkit.
Kana paine chimwe chinhu chaunoda kuona mukuvandudza maitiro, batanidza yedu , open Issues at kana kunyange kubata zvakananga . Tatova neakawanda emhando dzemhando dzinotonhorera - Jinja matemplate, mabhukumaki, mafirita ekutsvaga, Slack chenjedzo, Jupyter kubatanidzwa, kunyangwe CLI dashboard yemametrics - asi isu tingade yako mapindiro.
mhedziso
Whale inogadzirwa uye inochengetwa neDataframe, yekutanga iyo ini nguva pfupi yadarika ndakave nekunakidzwa kwekubatana nevamwe vanhu. Nepo whale ichigadzirirwa masayendisiti edata, Dataframe inogadzirirwa data masayendisiti. Kune avo vanoda kushandira pamwe zvakanyanya, sunungukai isu tichakuwedzera iwe kune yakamirira runyorwa.
Uye nepromo kodhi HABR, unogona kuwana imwe 10% kune dhipatimendi inoratidzwa pabhena.
Mamwe makosi
Featured Articles
Source: www.habr.com