Nkhaniyi ikukamba za chida chosavuta komanso chofulumira kwambiri chopezera deta, ntchito yomwe mukuwona pa KDPV. Chosangalatsa ndichakuti, whale idapangidwa kuti izikhala ndi seva yakutali ya git. Tsatanetsatane pansi pa odulidwa.
Momwe Airbnb's Data Discovery Tool idasinthira Moyo Wanga
M'ntchito yanga, ndakhala ndi mwayi wogwira ntchito pazovuta zina zosangalatsa: Ndinaphunzira masamu othamanga ndikuchita digiri yanga ku MIT, ndinagwira ntchito pazithunzi zowonjezera, komanso ndi polojekiti yotseguka. ku Wayfair, ndikukhazikitsa zitsanzo zatsamba loyambira zatsopano ndi kukonza kwa CUPED pa Airbnb. Koma ntchito yonseyi sinali yosangalatsaβinde, nthawi zambiri ndinkakhala ndikufufuza, kufufuza, ndi kutsimikizira deta. Ngakhale kuti izi zinali zokhazikika kuntchito, sindinazindikire kuti iyi inali vuto mpaka ndinakafika ku Airbnb komwe idathetsedwa ndi chida chotulukira deta β .
Kodi ndingapeze kuti {{data}}? dataportal.
Kodi gawoli likutanthauza chiyani? dataportal.
Kodi {{metric}} ikuyenda bwanji lero? dataportal.
Kodi lingaliro la moyo ndi chiyani? MU dataportal, mwina.
Chabwino, mwapereka chithunzi. Kupeza deta ndikumvetsetsa zomwe zikutanthawuza, momwe zinapangidwira komanso momwe mungagwiritsire ntchito zonse zimatenga mphindi zochepa chabe, osati maola. Nditha kuthera nthawi yanga ndikulemba mfundo zosavuta, kapena ma aligorivimu atsopano, (... .
Vuto ndi chiyani?
Ndinazindikira kuti anzanga ambiri analibe chida choterocho. Ndi makampani ochepa omwe ali okonzeka kupereka chuma chachikulu pomanga ndi kusunga chida cha nsanja monga Dataportal. Ndipo ngakhale pali mayankho angapo otseguka, amakhala opangidwa kuti azitha kukula, zomwe zimapangitsa kuti zikhale zovuta kukhazikitsa ndi kukonza popanda injiniya wodzipereka wa DevOps. Choncho ndinaganiza zopanga china chatsopano.
Whale: Chida chosavuta chotulukira deta
Ndipo inde, mopusa mophweka ndikutanthauza mopusa mophweka. Nangumi ali ndi zigawo ziwiri zokha:
- Laibulale ya Python yomwe imasonkhanitsa metadata ndikuipanga mu MarkDown.
- Rust command line interface posaka deta iyi.
Kuchokera pamawonedwe azinthu zamkati zokonzekera, pali mafayilo ambiri olembera ndi pulogalamu yomwe imasintha malembawo. Ndizomwezo, kotero kuchititsa pa seva ya git ngati Github ndizochepa. Palibe chilankhulo chatsopano choti muphunzire, palibe zoyang'anira, palibe zosunga zobwezeretsera. Aliyense amadziwa Git, kotero kulunzanitsa ndi mgwirizano ndi zaulere. Tiyeni tione mwatsatanetsatane magwiridwe .
GUI yodziwika bwino ya git-based
Nangumi adapangidwa kuti azisambira m'nyanja yakutali ya git seva. Iye zosinthika: fotokozani maulalo ena, lembani zolemba za Github Actions (kapena lembani imodzi ya nsanja yomwe mwasankha ya CI/CD) ndipo mudzakhala ndi chida chotulukira deta nthawi yomweyo. Mudzatha kusaka, kuwona, kulemba ndikugawana masamba anu mwachindunji pa Github.
Chitsanzo cha tebulo la stub lomwe linapangidwa pogwiritsa ntchito Github Actions. Chiwonetsero chathunthu chogwira ntchito .
Mphezi yachangu CLI fufuzani malo anu
Nangumi amakhala ndi moyo ndikupuma pamzere wolamula, ndikupereka mawonekedwe amphamvu, ma millisecond pamatebulo anu. Ngakhale tili ndi matebulo mamiliyoni ambiri, tidatha kupanga anamgumi kuti azigwira ntchito modabwitsa pogwiritsa ntchito njira zanzeru zosungira komanso pomanganso kumbuyo ku Rust. Simuwona kuchedwa kulikonse [moni Google DS].
Chiwonetsero cha Whale, kuyang'ana kwa tebulo miliyoni.
Kuwerengera mokha ma metrics [mu beta]
Chimodzi mwazinthu zomwe ndimakonda kwambiri monga wasayansi wa data akufunsanso zomwezo mobwerezabwereza kuti awone momwe deta ikugwiritsidwa ntchito. Whale imathandizira kutanthauzira ma metrics mu SQL yomveka bwino yomwe idzayendetsedwe ndi mapaipi anu oyeretsa metadata. Tanthauzirani chipika cha ma metrics a YAML mkati mwa tebulo la stub, ndipo Nangumi azingoyendetsa pa ndandanda ndikufunsa mafunso omwe ali muzitsulo.
```metrics
metric-name:
sql: |
select count(*) from table
```
Kuphatikizidwa ndi Github, njira iyi ikutanthauza kuti chinsomba chimatha kukhala gwero losavuta lachowonadi pamatanthauzidwe a metric. Nangumi amasunganso zikhalidwe pamodzi ndi sitampu yanthawi mu "~/. whale/metrics" ngati mukufuna kupanga ma chart kapena kafukufuku wozama.
Zamtsogolo
Titalankhula ndi ogwiritsa ntchito matembenuzidwe athu omwe adatulutsidwa kale, tidazindikira kuti anthu amafunikira magwiridwe antchito ambiri. Chifukwa chiyani chida choyang'ana tebulo? Bwanji osakhala chida chofufuzira ma metrics? Bwanji osayang'anira? Bwanji osagwiritsa ntchito SQL query tool? Ngakhale whale v1 poyambilira idapangidwa ngati chida chosavuta cha CLI Dataportal/Amundsen, yasintha kale kukhala nsanja yodziyimira yokha, ndipo tikukhulupirira kuti ikhala gawo lofunikira la zida za Data Scientist.
Ngati pali china chake chomwe mukufuna kuchiwona pachitukuko, lowani nawo , tsegulani Nkhani pa kapena ngakhale kulumikizana mwachindunji . Tili ndi zinthu zingapo zabwino - ma tempuleti a Jinja, ma bookmark, zosefera zosaka, zidziwitso za Slack, kuphatikiza kwa Jupyter, ngakhale dashboard ya CLI yama metrics - koma tikadakonda zomwe mwalemba.
Pomaliza
Whale imapangidwa ndikusamalidwa ndi Dataframe, chiyambi chomwe posachedwapa ndakhala nacho chisangalalo chogwirizanitsa ndi anthu ena. Ngakhale chinsomba chimapangidwira asayansi a data, Dataframe imapangidwira asayansi a data. Kwa inu omwe mukufuna kuti mugwirizane kwambiri, khalani omasuka tikuwonjezerani pamndandanda wodikirira.
Ndipo ndi promo code HABR, mutha kupeza 10% yowonjezera kuchotsera komwe kwawonetsedwa pachikwangwani.
Maphunziro ambiri
Nkhani Zowonetsedwa
Source: www.habr.com