Kuenderera mberi nedingindira remakwikwi ekudzidza muchina paHabrΓ©, tinoda kusuma vaverengi kune mamwe maviri mapuratifomu. Iwo zvechokwadi haana kukura sekaggle, asi anofanirwa kutariswa.
Ini pachangu, ini handifarire kaggle yakawandisa nekuda kwezvikonzero zvakati:
- chekutanga, makwikwi ikoko anowanzogara kwemwedzi yakati wandei, uye kutora chikamu kunoda kushanda nesimba;
- chechipiri, mhodzi dzeruzhinji (mhinduro dzeruzhinji). Vateveri veKaggle vanopa zano kuvabata nekudzikama kwemamongi eTibetan, asi muchokwadi zvinonyadzisa kana chimwe chinhu chawanga uchishandira kwemwedzi kana maviri chikangoerekana chaiswa mundiro yesirivheri yemunhu wese.
Neraki, makwikwi ekudzidza muchina anoitwa pane mamwe mapuratifomu, uye akati wandei emakwikwi aya achakurukurwa.
| Mutauro wepamutemo: Chirungu, varongi: Yandex, Sberbank, HSE |
Mutauro wepamutemo weRussia, varongi: Mail.ru Group |
| Online Round: Jan 15 - Feb 11, 2019; Pa-Site Yekupedzisira: Kubvumbi 4-6, 2019 |
online - kubva Kukadzi 7 kusvika Kurume 15; offline - kubva Kurume 30 kusvika Kubvumbi 1. |
| Uchishandisa imwe seti yedata nezve chidimbu muLarge Hadron Collider (trajectory, kukurumidza, uye mamwe maparamita akaoma emuviri), tarisa kuti iri muon here kana kuti kwete. Kubva muchirevo ichi, 2 mabasa akaonekwa: - mune imwe chete waifanira kutumira kufanotaura kwako, - uye mune imwe - iyo yakazara kodhi uye modhi yekufembera, uye kuuraya kwacho kwaive pasi pezvirambidzo zvakasimba pakumhanya nguva uye kushandiswa kwendangariro. |
Pamakwikwi eSNA Hackathon, matanda ezvinyorwa anoratidza kubva kumapoka akavhurika mumashandisi enhau ezve Kukadzi-Kurume 2018 akaunganidzwa. Iyo test set ine vhiki yekupedzisira nehafu yaKurume. Chese chinopinda murogi chine ruzivo nezve zvakaratidzwa uye kunaani, uye nemabatiro akaita mushandisi kune izvi zvirimo: zviyeresa, zvakataurwa, hazvina hanya nazvo, kana kuzvivanza kubva pakudya. Izvo zvakakosha zveSNA Hackathon's mabasa ndeyekuisa chinzvimbo chega chega mushandisi wesocial network Odnoklassniki chikafu chake, achisimudza zvakanyanya sezvinobvira izvo zvinyorwa zvinogamuchira "kirasi". Padanho repamhepo, basa rakakamurwa kuita zvikamu zvitatu: 1. chinzvimbo chezvinzvimbo zvinoenderana neakasiyana maitiro ekubatana 2. zvigaro zvenzvimbo zvichibva pamifananidzo yazviri 3. rongedza zvinzvimbo zvinoenderana nezvinyorwa zvazviri |
| Yakaoma tsika metric, chimwe chinhu seROC-AUC | Avhareji ROC-AUC nemushandisi |
| Mibairo yenhanho yekutanga - T-shirts yenzvimbo dzeN, nzira kuenda kune yechipiri nhanho, uko pekugara uye chikafu chakabhadharwa panguva yemakwikwi. Chikamu chechipiri - ??? (Nezvimwe zvikonzero, ini ndakanga ndisipo pamutambo wemubairo uye handina kukwanisa kuziva kuti mibairo yaive yei pakupedzisira). Vakavimbisa malaptops kunhengo dzese dzechikwata chinokunda |
Mibayiro yedanho rekutanga - T-shirts ye100 vatori vechikamu vakanyanya, kuenda kuchikamu chechipiri, uko kuenda kuMoscow, pekugara uye kudya panguva yemakwikwi zvakabhadharwa. Zvakare, kusvika kumagumo kwechikamu chekutanga, mibairo yakaziviswa yeakanakisa mumabasa matatu padanho rekutanga: munhu wese akahwina RTX 3 TI vhidhiyo kadhi! Chikamu chechipiri chaive chikwata chechikwata, zvikwata zvaive nevanhu 2 kusvika ku5, mibairo: 1st nzvimbo - 300 rubles 2st nzvimbo - 200 rubles 3st nzvimbo - 100 rubles jury mubayiro - 100 rubles |
| Boka repamutemo reteregiramu, ~ 190 vatori vechikamu, kutaurirana muChirungu, mibvunzo yaifanira kumirira mazuva akati wandei kuti ipindurwe. | Boka repamutemo muteregiramu, ~ 1500 vatori vechikamu, hurukuro inoshingaira yemabasa pakati pevatori vechikamu nevarongi. |
| Varongi vakapa mhinduro mbiri, dziri nyore uye dzepamberi. Nyore inodiwa isingasviki 16 GB ye RAM, uye ndangariro yepamberi haina kukwana mu16. Panguva imwecheteyo, kutarisa mberi zvishoma, vatori vechikamu vakanga vasingakwanisi kudarika zvakanyanya kugadziriswa kwepamusoro. Pakanga pasina matambudziko pakuparura mhinduro idzi. Izvo zvinofanirwa kucherechedzwa kuti mumuenzaniso wepamberi paive nemhinduro ine zano rekuti ungatanga kupi kugadzirisa mhinduro. | Basic primitive solutions dzakapihwa kune rimwe nerimwe remabasa, ayo aipfuurwa zviri nyore nevatori vechikamu. Mumazuva ekutanga emakwikwi, vatori vechikamu vakasangana nematambudziko akati wandei: chekutanga, iyo data yakapihwa muApache Parquet fomati, uye kwete ese masanganiswa ePython neparquet package yakashanda pasina zvikanganiso. Chechipiri chakaomerwa kwaive kurodha mifananidzo kubva mugore retsamba; parizvino hapana nzira iri nyore yekurodha yakawanda data kamwechete. Nekuda kweizvozvo, matambudziko aya akanonotsa vatori vechikamu kwemazuva mashoma. |
IDAO. Danho rekutanga
Basa racho raive rekuisa muon/non-muon particles zvichienderana nehunhu hwawo. Chinhu chakakosha chebasa iri kwaive kuvapo kwekoramu inorema mune data rekudzidzisa, iro varongi ivo pachavo vakadudzira sekuvimba mumhinduro yemutsara uyu. Dambudziko raive rekuti mitsetse yakati wandei yaive nemaremu asina kunaka.
Mushure mekufunga kwemaminetsi mashoma nezve mutsara une hint (iro zano rakangokwevera pfungwa kune ichi chikamu chehuremu koramu) nekuvaka iyi girafu, isu takasarudza kutarisa 3 sarudzo:
1) shandura chinangwa chemitsara ine huremu husina kunaka (uye uremu zvinoenderana)
2) shandura huremu kune hushoma kukosha kuti hutange kubva ku0
3) usashandise uremu hwetambo
Sarudzo yechitatu yakazova yakaipisisa, asi maviri ekutanga akavandudza chigumisiro, yakanakisisa yaiva sarudzo Nhamba 1, iyo yakabva yatiunza panzvimbo yechipiri ikozvino mubasa rekutanga uye yekutanga mune yechipiri.
Nhanho yedu yaitevera yaive yekuongorora iyo data yezvisina kukosha. Varongi vakatipa data yakakangwa kare, pakanga paine mashoma mashoma asipo, uye akatsiviwa ne -9999.
Tawana hunhu husipo muMatchedHit_{X,Y,Z}[N] uye MatchedHit_D{X,Y,Z}[N] makoramu, uye paingova N=2 kana 3. Sekunzwisisa kwatinoita, zvimwe zvikamu hazvina. pfuura ese mana madetectors, uye akamira angave ari pa4rd kana 3th ndiro. Mashoko aya anga ainewo Lextra_{X,Y}[N] makoramu, ayo anotsanangura zvakafanana neMatchedHit_{X,Y,Z}[N], asi achishandisa imwe nzira yekuwedzera. Idzi fungidziro diki dzinoratidza kuti Lextra_{X,Y}[N] inogona kutsiviwa neukoshi husipo muMatchedHit_{X,Y,Z}[N] (ye X uye Y zvinoronga chete). MatchedHit_Z[N] yakazadzwa zvakanaka neyepakati. Aya manipulations akatibvumira kusvika 4st nzvimbo yepakati mumabasa ese ari maviri.
Tichifunga kuti havana kupa chero chinhu chekukunda danho rekutanga, tingadai takamira ipapo, asi takaenderera mberi, takadhirowa mifananidzo yakanaka uye takauya nezvinhu zvitsva.
Semuyenzaniso, takaona kuti kana tikaronga mharadzano dzechikamu chimwe nechimwe chezvina detector plate, tinokwanisa kuona kuti mapoinzi erimwe nerimwe remahwendefa akaiswa muzvikamu zvishanu zvemarectangles ane aspect ratio ye 5 kusvika 4 uye akananga pa. iyo poindi (5), uye muHamuna mapoinzi murectangle yekutanga.
| Plate nhamba. / rectangle dimensions | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Plate 1 | 500x625 | 1000x1250 | 2000x2500 | 4000x5000 | 8000x10000 |
| Plate 2 | 520x650 | 1040x1300 | 2080x2600 | 4160x5200 | 8320x10400 |
| Plate 3 | 560x700 | 1120x1400 | 2240x2800 | 4480x5600 | 8960x11200 |
| Plate 4 | 600x750 | 1200x1500 | 2400x3000 | 4800x6000 | 9600x12000 |
Taona zvimiro izvi, takawedzera 4 mitsva yechikamu chechikamu chega chega - nhamba yerectangle mairi inopindirana ndiro yega yega.
Takacherechedzawo kuti zvimedu zvaiita sokuti zvinopararira kumativi kubva pakati uye pfungwa yakasimuka kuti iongorore "hutano" hwekupararira uku. Nenzira yakanaka, zvingave zvichibvira kuuya nemhando ye "yakanaka" parabola zvichienderana nenzvimbo yekutora uye kufungidzira kutsauka kubva pairi, asi isu takagumira pachedu kune "yakanaka" mutsara wakarurama. Mushure mekugadzira mitsara yakatwasuka yakadai yepoindi yega yega yekupinda, takakwanisa kuverenga kutsauka kwakajairwa kwenzira yechikamu chimwe nechimwe kubva mumutsara wakatwasuka uyu. Sezvo avhareji yekutsauka kwechinangwa = 1 yaive 152, uye pachinangwa = 0 yaive 390, isu takaongorora chimiro ichi sechakanaka. Uye zvechokwadi, chimiro ichi chakabva chaita kuti ive pamusoro peanonyanya kubatsira.
Isu takafara uye takawedzera kutsauka kwese mana emharadzano mapoinzi ega ega chidimbu kubva kune yakaringana mutsara semamwe maficha mana (uye akashandawo nemazvo).
Zvisungo kune zvinyorwa zvesainzi pamusoro pemusoro wemakwikwi, akapihwa isu nevarongi, akakurudzira pfungwa yekuti isu tiri kure nevekutanga kugadzirisa dambudziko iri uye, pamwe, pane imwe mhando yehunyanzvi software. Mushure mekunge tawana repository pa github uko nzira dzeIsMuonSimple, IsMuon, IsMuonLoose dzakaitwa, takadziendesa kune yedu saiti ine zvidiki zvigadziriso. Nzira dzacho pachadzo dzakanga dziri nyore kwazvo: somuenzaniso, kana simba riri pasi peimwe chikumbaridzo, saka haisi muon, kana zvisina kudaro muon. Mamiriro akareruka akadai sezviri pachena aisakwanisa kupa kuwedzera kwekushandisa gradient kuwedzera, saka takawedzera imwe yakakosha "chinhambwe" pachikumbaridzo. Aya maficha akavandudzwawo zvishoma. Zvichida, nekuongorora nzira dziripo zvakanyanya, zvaive zvichibvira kuwana nzira dzakasimba uye kuwedzera kune zviratidzo.
Pakupera kwemakwikwi, isu takagadzirisa zvishoma "nekukurumidza" mhinduro yedambudziko rechipiri; pakupedzisira, yakasiyana neyekutanga mune anotevera mapoinzi:
- Mumitsara ine huremu husina kunaka chinangwa chakadzoserwa
- Yakazadzwa neukoshi husipo muMatchedHit_{X,Y,Z}[N]
- Yakaderedzwa kudzika kusvika ku7
- Yakaderedzwa mwero wekudzidza kusvika pa0.1 (yaive 0.19)
Nekuda kweizvozvo, takaedza mamwe maficha (asina kunyatsobudirira), akasarudzwa paramita uye akadzidziswa catboost, lightgbm uye xgboost, takaedza kusanganiswa kwakasiyana kwekufungidzira uye tisati tavhura zvakavanzika isu takakunda nechivimbo pabasa rechipiri, uye pane yekutanga isu takanga tiri pakati pe vatungamiri.
Mushure mekuvhura yakavanzika isu taive munzvimbo yegumi yebasa rekutanga uye rechitatu rechipiri. Vatungamiri vese vakasangana, uye kumhanya kwepachivande kwaive kwakakwira kupfuura pabhodhi. Zvinoita sekuti data racho rakanga risina kurongeka zvakanaka (kana semuenzaniso pakanga pasina mitsara ine huremu husina kunaka pachivande) uye izvi zvaitinetsa.
SNA Hackathon 2019 - Zvinyorwa. Danho rekutanga
Basa racho raive rekuisa zvinzvimbo zvevashandisi paOdnoklassniki social network zvichibva pane zvinyorwa zvavaive nazvo; kunze kwechinyorwa, pakanga paine mamwe maitiro mashoma eiyo positi (mutauro, muridzi, zuva uye nguva yekusika, zuva uye nguva yekuona. )
Semaitiro echinyakare ekushanda nemavara, ini ndaizosimbisa maviri sarudzo:
- Kumepu izwi rega rega mune n-dimensional vector nzvimbo zvekuti mazwi akafanana ane mavheji akafanana (verenga zvakawanda mu ), wobva wawana avhareji yezwi rechinyorwa kana kushandisa michina inofunga nezvechinzvimbo chemazwi (CNN, LSTM/GRU).
- Kushandisa mhando dzinogona kushanda nemitsara yese ipapo ipapo. Somuenzaniso, Bert. Mukutaura, nzira iyi inofanira kushanda zviri nani.
Sezvo ichi chaiva chiitiko changu chekutanga nemagwaro, zvingave zvisina kunaka kudzidzisa mumwe munhu, saka ndichazvidzidzisa ini. Aya ndiwo matipi andizozvipa ini pakutanga kwemakwikwi:
- Usati wamhanya kunodzidzisa chimwe chinhu, tarisa data! Pamusoro pezvinyorwa pachazvo, iyo data yaive nemakoramu akati wandei uye zvaikwanisika kudzvanya zvakanyanya kubva kwavari kupfuura zvandakaita. Chinhu chiri nyore kuita kureva chinangwa encoding kune mamwe makoramu.
- Usadzidze kubva kune ese data! Paive nedata rakawanda (inenge miriyoni gumi nemanomwe mitsara) uye zvaive zvisingaite zvachose kushandisa ese kuyedza kufungidzira. Kudzidzira uye preprocessing kwainonoka, uye ini sezviri pachena ndingadai ndakave nenguva yekuyedza mamwe anonakidza mafungiro.
- <Zano rine nharo> Hapana chikonzero chekutsvaga mhondi modhi. Ndakatora nguva yakareba ndichifunga Elmo naBert, ndichitarisira kuti vachabva vandiendesa kunzvimbo yakakwirira, uye somugumisiro ndakashandisa FastText pre-kudzidziswa embeddings yemutauro weRussia. Ini handina kukwanisa kumhanyisa zvirinani naElmo, uye ndakanga ndisati ndave nenguva yekuzvifunga naBert.
- <Zano rine nharo> Hapana chikonzero chekutsvaga imwe mhondi chimiro. Ndichitarisa iyo data, ndakaona kuti yakatenderedza 1 muzana yezvinyorwa haina chaizvo zvinyorwa! Asi pakanga paine zvinongedzo kune zvimwe zviwanikwa, uye ini ndakanyora yakapusa parser yakavhura saiti ndokuburitsa zita uye tsananguro. Zvaiita senge zano rakanaka, asi ndakazotakurwa uye ndikafunga kupatsanura zvese zvinongedzo zvemagwaro ese uye zvakare ndakarasikirwa nenguva yakawanda. Zvese izvi hazvina kupa budiriro yakakosha mumhedzisiro yekupedzisira (kunyangwe ini ndakafunga kubuda, semuenzaniso).
- Classic features work. Isu Google, semuenzaniso, "text features kaggle", verenga nekuwedzera zvese. TF-IDF yakapa kuvandudzwa, sezvakaita zviverengero zvakaita sehurefu hwemavara, mazwi, uye huwandu hwezviratidzo.
- Kana paine DateTime makoramu, zvakakodzera kuapatsanura kuita akati wandei akasiyana maficha (maawa, mazuva evhiki, nezvimwewo). Ndeapi maficha anofanira kujekeswa anofanirwa kuongororwa uchishandisa magirafu / mamwe metrics. Pano, pane chishuwo, ndakaita zvese nemazvo uye ndakasimbisa maficha anodiwa, asi kuongororwa kwakajairwa kwaisazokuvadza (semuenzaniso, sezvatakaita pakupedzisira).
Nekuda kwemakwikwi, ndakadzidzisa imwe keras modhi ine izwi convolution, uye imwe yakavakirwa paLSTM neGRU. Vese vari vaviri vakashandisa pre-yakadzidziswa FastText embeddings yemutauro wechiRussia (ndakaedza akati wandei mamwe embeddings, asi aya ndiwo akashanda zvakanyanya). Mushure mekuenzanisa fungidziro, ndakatora nzvimbo yekupedzisira yechinomwe kubva pavatori vechikamu makumi manomwe nevatanhatu.
Mushure mechikamu chekutanga chakabudiswa , uyo akatora nzvimbo yechipiri (akatora chikamu kunze kwemakwikwi), uye mhinduro yake kusvika kune imwe nhanho yakadzokororwa yangu, asi akaenda mberi nekuda kwemubvunzo-kiyi-kukosha kwekutarisa maitiro.
Chikamu chechipiri OK & IDAO
Zvikamu zvechipiri zvemakwikwi zvakaitika zvakatevedzana, saka ndakafunga kuzvitarisa pamwechete.
Kutanga, ini nechikwata chichangobva kuwanikwa takaguma muhofisi inonakidza yekambani yeMail.ru, uko basa redu raive rekubatanidza mamodheru ematatu kubva padanho rekutanga - zvinyorwa, mifananidzo uye collab. Mazuva mashoma anopfuura maviri akagoverwa kune izvi, izvo zvakazove zvishoma. Muchokwadi, isu takangokwanisa kudzokorora mhedzisiro yedu kubva padanho rekutanga tisina kuwana chero chawawana kubva mukubatanidzwa. Pakupedzisira, takatora nzvimbo yechishanu, asi hatina kukwanisa kushandisa muenzaniso wemavara. Mushure mekutarisa mhinduro dzevamwe vatori vechikamu, zvinoratidzika kuti zvaive zvakakodzera kuedza kuunganidza zvinyorwa uye kuzviwedzera kumuenzaniso wekubatana. Imwe mhedzisiro yedanho iri yaive maonerwo matsva, kusangana uye kutaurirana nevatori vechikamu uye varongi vanotonhorera, pamwe nekushaikwa kwakanyanya kwekurara, izvo zvingave zvakakanganisa mhedzisiro yechikamu chekupedzisira cheIDAO.
Basa paIDAO 2019 Final nhanho yaive yekufanotaura nguva yekumirira yekuraira kweYandex taxi driver paairport. Padanho rechipiri, 2 mabasa = 3 nhandare dzendege dzakaonekwa. Kune imwe neimwe airport, miniti-ne-miniti data pamusoro pehuwandu hwemaodha etekisi kwemwedzi mitanhatu inopihwa. Uye se data rekuyedza, mwedzi unotevera uye miniti-ne-miniti data pamaodha emavhiki maviri apfuura akapihwa. Paiva nenguva shoma (mazuva 3), basa racho rakanga rakanyatsojeka, munhu mumwe chete kubva kuchikwata akauya kumakwikwi - uye somugumisiro, yakanga iri nzvimbo inosuruvarisa kusvika kumagumo. Mazano anonakidza aisanganisira kuedza kushandisa data rekunze: mamiriro ekunze, traffic jam uye Yandex taxi order statistics. Kunyange zvazvo varongi vasina kutaura kuti nhandare dzendege idzi ndedzipi, vatori vechikamu vakawanda vakafunga kuti vaiva Sheremetyevo, Domodedovo uye Vnukovo. Kunyangwe fungidziro iyi yakarambwa mushure memakwikwi, maficha, semuenzaniso, kubva kuMoscow data yemamiriro ekunze akavandudza mhedzisiro pairi kusimbiswa uye pabhodhi.
mhedziso
- Makwikwi eML anotonhorera uye anonakidza! Pano iwe uchawana kushandiswa kwehunyanzvi mukuongororwa kwedata, uye mumamodheru ane hungwaru uye matekiniki, uye kungoita njere kunogamuchirwa.
- ML yatove mutumbi muhombe weruzivo unoratidzika kunge uri kukura zvakanyanya. Ndakazvigadzirira chinangwa chekujairana nenzvimbo dzakasiyana (zviratidzo, mifananidzo, matafura, zvinyorwa) uye ndatoona kuti pane zvakawanda zvekudzidza. Semuenzaniso, mushure memakwikwi aya ndakafunga kudzidza: kubatanidza algorithms, nzira dzepamusoro dzekushanda nemaraibhurari ekusimudzira gradient (kunyanya, kushanda neCatBoost paGPU), capsule network, iyo query-key-value yekutarisisa maitiro.
- Kwete nekaggle chete! Kune mamwe akawanda makwikwi uko zviri nyore kuwana kanenge T-shati, uye kune mikana yakawanda yemamwe mibairo.
- Taura! Patova nenharaunda yakakura mumunda wekudzidza kwemichina uye kuongorora data, kune mapoka ezvinyorwa muteregiramu, slack, uye vanhu vakakomba kubva kuMail.ru, Yandex nemamwe makambani vanopindura mibvunzo nekubatsira vanotanga uye avo vanoenderera mberi nenzira iyi. yezivo.
- Ini ndinopa zano kune wese akafemerwa nepfungwa yapfuura kuti ashanyire - musangano mukuru wakasununguka muMoscow, uchaitika muna May 10-11.
Source: www.habr.com