{"id":53143,"date":"2019-11-24T00:00:00","date_gmt":"2019-11-23T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru"},"modified":"2020-02-18T14:01:00","modified_gmt":"2020-02-18T11:01:00","slug":"ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru","title":{"rendered":"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/ee5728dccb94b849ad3bfd9cf84ad74d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<i>Na z\u00e1klad\u011b m\u00fdch vystoupen\u00ed na Highload++ a DataFest Minsk 2019.<\/i><\/p>\n<p>Pro mnoho lid\u00ed je dnes e-mail neodmyslitelnou sou\u010d\u00e1st\u00ed \u017eivota na internetu. Pomoc\u00ed n\u011bj vedeme obchodn\u00ed korespondenci, uchov\u00e1v\u00e1me r\u016fzn\u00e9 d\u016fle\u017eit\u00e9 informace t\u00fdkaj\u00edc\u00ed se financ\u00ed, rezervac\u00ed hotel\u016f, realizace objedn\u00e1vek a mnohem v\u00edce. V polovin\u011b roku 2018 jsme formulovali produktovou strategii rozvoje e-mailu. Jak\u00fd by m\u011bl b\u00fdt modern\u00ed e-mail?<\/p>\n<p>E-mail mus\u00ed b\u00fdt <b>inteligentn\u00ed<\/b>, tedy pom\u00e1hat u\u017eivatel\u016fm orientovat se v rostouc\u00edm objemu informac\u00ed: filtrovat, strukturovat a poskytovat je co nejpohodln\u011bj\u0161\u00edm zp\u016fsobem. M\u011bl by b\u00fdt <b>u\u017eite\u010dn\u00fd<\/b>, umo\u017e\u0148uj\u00edc\u00ed \u0159e\u0161it r\u016fzn\u00e9 \u00fakoly p\u0159\u00edmo v e-mailov\u00e9 schr\u00e1nce, nap\u0159\u00edklad platit pokuty (funkce, kterou, bohu\u017eel, vyu\u017e\u00edv\u00e1m). A samoz\u0159ejm\u011b, e-mail by m\u011bl zaji\u0161\u0165ovat ochranu informac\u00ed, filtrovat spam a chr\u00e1nit p\u0159ed hackery, tedy b\u00fdt <b>bezpe\u010dn\u00e9<\/b>.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nTyto sm\u011bry ur\u010duj\u00ed \u0159adu kl\u00ed\u010dov\u00fdch \u00fakol\u016f, z nich\u017e mnoh\u00e9 mohou b\u00fdt efektivn\u011b \u0159e\u0161eny pomoc\u00ed strojov\u00e9ho u\u010den\u00ed. Zde jsou p\u0159\u00edklady ji\u017e existuj\u00edc\u00edch funkc\u00ed, vyvinut\u00fdch v r\u00e1mci strategie \u2014 po jednom na ka\u017edou oblast.<\/p>\n<ul>\n<li><b>Smart Reply<\/b>. V e-mailu je funkce chytr\u00e9 odpov\u011bdi. Neuronov\u00e1 s\u00ed\u0165 analyzuje text e-mailu, ch\u00e1pe jeho smysl a \u00fa\u010del, a v d\u016fsledku toho navrhuje t\u0159i nejvhodn\u011bj\u0161\u00ed mo\u017enosti odpov\u011bdi: pozitivn\u00ed, negativn\u00ed a neutr\u00e1ln\u00ed. To pom\u00e1h\u00e1 podstatn\u011b u\u0161et\u0159it \u010das p\u0159i odpov\u011bd\u00edch na e-maily a tak\u00e9 \u010dasto reagovat netradi\u010dn\u011b a z\u00e1bavn\u011b.\n<\/li>\n<li><b>Skupinov\u00e1n\u00ed e-mail\u016f<\/b>, kter\u00e9 se t\u00fdkaj\u00ed objedn\u00e1vek v internetov\u00fdch obchodech. \u010casto nakupujeme na internetu a obchody obvykle mohou pos\u00edlat n\u011bkolik e-mail\u016f na ka\u017edou objedn\u00e1vku. Nap\u0159\u00edklad od AliExpressu, nejv\u011bt\u0161\u00edho servisu, chod\u00ed velmi mnoho e-mail\u016f k jedn\u00e9 objedn\u00e1vce, a zjistili jsme, \u017ee v krajn\u00edm p\u0159\u00edpad\u011b m\u016f\u017ee jejich po\u010det dos\u00e1hnout a\u017e 29. Proto pomoc\u00ed modelu rozpozn\u00e1v\u00e1n\u00ed pojmenovan\u00fdch entit vy\u010dle\u0148ujeme \u010d\u00edslo objedn\u00e1vky a dal\u0161\u00ed informace z textu a skupinujeme v\u0161echny e-maily do jednoho vl\u00e1kna. Tak\u00e9 zobrazujeme z\u00e1kladn\u00ed informace o objedn\u00e1vce v samostatn\u00e9 li\u0161t\u011b, co\u017e usnad\u0148uje pr\u00e1ci s t\u00edmto typem e-mail\u016f.\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/8b37b0bf0c0d152498027bc25d476128.jpg\" style=\"display:block;margin: 0 auto;\" \/>\n<\/li>\n<li><b>Antiphishing<\/b>Phishing je zvl\u00e1\u0161\u0165 nebezpe\u010dn\u00fd typ podvodn\u00fdch e-mail\u016f, pomoc\u00ed kter\u00fdch se zlod\u011bji sna\u017e\u00ed z\u00edskat finan\u010dn\u00ed informace (v\u010detn\u011b \u00fadaj\u016f o bankovn\u00edch kart\u00e1ch u\u017eivatel\u016f) a p\u0159ihla\u0161ovac\u00ed \u00fadaje. Tyto e-maily se maskuj\u00ed jako prav\u00e9 zpr\u00e1vy zaslan\u00e9 slu\u017ebou, v\u010detn\u011b vizu\u00e1ln\u00edho vzhledu. Proto pomoc\u00ed Computer Vision rozpozn\u00e1v\u00e1me loga a styl designu e-mail\u016f velk\u00fdch spole\u010dnost\u00ed (nap\u0159\u00edklad Mail.ru, Sber, Alfa) a zahrnujeme tyto prvky spolu s textem a dal\u0161\u00edmi charakteristikami do na\u0161ich klasifik\u00e1tor\u016f spamu a phishingu.\n<\/li>\n<\/ul>\n<p><\/p>\n<h2>Strojov\u00e9 u\u010den\u00ed<\/h2>\n<p>\nN\u011bco m\u00e1lo o strojov\u00e9m u\u010den\u00ed v e-mailech obecn\u011b. E-mail je vysoce z\u00e1t\u011b\u017eov\u00fd syst\u00e9m: na\u0161imi servery proch\u00e1z\u00ed pr\u016fm\u011brn\u011b 1,5 miliardy e-mail\u016f denn\u011b pro 30 milion\u016f u\u017eivatel\u016f DAU. V\u0161echny pot\u0159ebn\u00e9 funkce a vlastnosti obsluhuje p\u0159ibli\u017en\u011b 30 syst\u00e9m\u016f strojov\u00e9ho u\u010den\u00ed. <\/p>\n<p>Ka\u017ed\u00fd e-mail proch\u00e1z\u00ed cel\u00fdm klasifika\u010dn\u00edm procesem. Nejd\u0159\u00edve odfiltrov\u00e1v\u00e1me spam a nech\u00e1v\u00e1me pouze dobr\u00e9 e-maily. U\u017eivatel\u00e9 \u010dasto nezaznamen\u00e1vaj\u00ed pr\u00e1ci antispamu, proto\u017ee 95\u201399 % spamu se nedostane ani do p\u0159\u00edslu\u0161n\u00e9 slo\u017eky. Rozpozn\u00e1v\u00e1n\u00ed spamu je velmi d\u016fle\u017eitou a nejt\u011b\u017e\u0161\u00ed \u010d\u00e1st\u00ed na\u0161eho syst\u00e9mu, proto\u017ee v oblasti antispamu prob\u00edh\u00e1 neust\u00e1l\u00e1 adaptace mezi ochrann\u00fdmi a \u00fato\u010dn\u00fdmi syst\u00e9my, co\u017e p\u0159in\u00e1\u0161\u00ed trvalou in\u017een\u00fdrskou v\u00fdzvu pro n\u00e1\u0161 t\u00fdm.<\/p>\n<p>D\u00e1le d\u011bl\u00edme e-maily od lid\u00ed a robot\u016f. E-maily od lid\u00ed jsou nejd\u016fle\u017eit\u011bj\u0161\u00ed, proto pro n\u011b nab\u00edz\u00edme funkce jako Smart Reply. E-maily od robot\u016f se d\u011bl\u00ed na dv\u011b \u010d\u00e1sti: transak\u010dn\u00ed \u2014 to jsou d\u016fle\u017eit\u00e9 e-maily od slu\u017eeb, jako jsou potvrzen\u00ed n\u00e1kup\u016f nebo rezervac\u00ed hotel\u016f, financ\u00ed, a informa\u010dn\u00ed \u2014 co\u017e je obchodn\u00ed reklama, slevy. <\/p>\n<p>Domn\u00edv\u00e1me se, \u017ee transak\u010dn\u00ed e-maily maj\u00ed stejnou d\u016fle\u017eitost jako osobn\u00ed korespondence. M\u011bly by b\u00fdt po ruce, proto\u017ee \u010dasto pot\u0159ebujeme rychle naj\u00edt informace o objedn\u00e1vce nebo rezervaci letenky, a ztr\u00e1c\u00edme \u010das jejich hled\u00e1n\u00edm. Proto je pro usnadn\u011bn\u00ed automaticky d\u011bl\u00edme do \u0161esti z\u00e1kladn\u00edch kategori\u00ed: cestov\u00e1n\u00ed, objedn\u00e1vky, finance, j\u00edzdenky, registrace a nakonec pokuty.<\/p>\n<p>Informa\u010dn\u00ed e-maily jsou nejpo\u010detn\u011bj\u0161\u00ed a pravd\u011bpodobn\u011b nejm\u00e9n\u011b d\u016fle\u017eitou skupinou, kter\u00e1 nevy\u017eaduje okam\u017eitou reakci, proto\u017ee pokud u\u017eivatel takov\u00fd e-mail nep\u0159e\u010dte, v jeho \u017eivot\u011b se nic z\u00e1sadn\u00edho nezm\u011bn\u00ed. V na\u0161em nov\u00e9m rozhran\u00ed je svrhujeme do dvou vl\u00e1ken: soci\u00e1ln\u00edch s\u00edt\u00ed a newsletter\u016f, \u010d\u00edm\u017e vizu\u00e1ln\u011b vy\u010dist\u00edme schr\u00e1nku a nech\u00e1me na o\u010d\u00edch pouze d\u016fle\u017eit\u00e9 e-maily.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/175773879daaa29d1542a98070cb7972.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Provoz<\/h3>\n<p>\nVelk\u00e9 mno\u017estv\u00ed syst\u00e9m\u016f p\u0159in\u00e1\u0161\u00ed mnoho obt\u00ed\u017e\u00ed v provozu. V\u017edy\u0165 modely s \u010dasem degraduj\u00ed, stejn\u011b jako jak\u00fdkoli software: zanikaj\u00ed funkce, selh\u00e1vaj\u00ed za\u0159\u00edzen\u00ed a hromad\u00ed se chybov\u00fd k\u00f3d. Nav\u00edc se neust\u00e1le m\u011bn\u00ed \u00fadaje: p\u0159id\u00e1vaj\u00ed se nov\u00e9, transformuje se vzor chov\u00e1n\u00ed u\u017eivatel\u016f atd., tak\u017ee model bez \u0159\u00e1dn\u00e9 podpory bude s \u010dasem fungovat \u010d\u00edm d\u00e1l h\u016f\u0159e. <\/p>\n<p>Nesm\u00edme zapom\u00ednat ani na to, \u017ee \u010d\u00edm hloub\u011bji strojov\u00e9 u\u010den\u00ed pronik\u00e1 do \u017eivota u\u017eivatel\u016f, t\u00edm v\u011bt\u0161\u00ed vliv maj\u00ed na ekosyst\u00e9m, a n\u00e1sledn\u011b t\u00edm v\u00edce finan\u010dn\u00edch ztr\u00e1t nebo zisk\u016f mohou hr\u00e1\u010di na trhu u\u010dinit. Proto se v \u010d\u00edm d\u00e1l v\u011bt\u0161\u00edm po\u010dtu oblast\u00ed hr\u00e1\u010di adaptuj\u00ed na pr\u00e1ci s ML algoritmy (klasick\u00e9 p\u0159\u00edklady - reklama, vyhled\u00e1v\u00e1n\u00ed a ji\u017e zm\u00edn\u011bn\u00fd antispam).<\/p>\n<p>Tak\u00e9 \u00fakoly strojov\u00e9ho u\u010den\u00ed maj\u00ed specifikum: jak\u00e1koliv, i drobn\u00e1 zm\u011bna v syst\u00e9mu m\u016f\u017ee vyvolat spoustu pr\u00e1ce s modelem: pr\u00e1ce s daty, p\u0159e\u0161kolen\u00ed, nasazen\u00ed, co\u017e m\u016f\u017ee trvat t\u00fddny nebo m\u011bs\u00edce. Proto \u010d\u00edm rychleji se m\u011bn\u00ed prost\u0159ed\u00ed, ve kter\u00e9m va\u0161e modely operuj\u00ed, t\u00edm v\u00edce \u00fasil\u00ed vy\u017eaduje jejich podpora. T\u00fdm m\u016f\u017ee vytvo\u0159it celou \u0159adu syst\u00e9m\u016f a radovat se z toho, ale pot\u00e9 m\u016f\u017ee t\u00e9m\u011b\u0159 v\u0161echny zdroje utratit na jejich \u00fadr\u017ebu, ani\u017e by m\u011bl mo\u017enost ud\u011blat n\u011bco nov\u00e9ho. S takovou situac\u00ed jsme se jednou setkali v t\u00fdmu antispamu. A ud\u011blali jsme z\u0159ejm\u00fd z\u00e1v\u011br, \u017ee \u00fadr\u017ebu je t\u0159eba automatizovat.<\/p>\n<h3>Automatizace<\/h3>\n<p>\nCo lze automatizovat? Ve skute\u010dnosti t\u00e9m\u011b\u0159 v\u0161echno. Vyjmenoval jsem \u010dty\u0159i oblasti, kter\u00e9 ur\u010duj\u00ed infrastrukturu strojov\u00e9ho u\u010den\u00ed:<\/p>\n<ul>\n<li>sb\u011br dat;\n<\/li>\n<li>dokon\u010den\u00ed \u0161kolen\u00ed;\n<\/li>\n<li>nasazen\u00ed;\n<\/li>\n<li>testov\u00e1n\u00ed &amp; monitoring.\n<\/li>\n<\/ul>\n<p>\nPokud je prost\u0159ed\u00ed nestabiln\u00ed a neust\u00e1le se m\u011bn\u00ed, pak je cel\u00e1 infrastruktura kolem modelu mnohem d\u016fle\u017eit\u011bj\u0161\u00ed ne\u017e samotn\u00fd model. M\u016f\u017ee to b\u00fdt star\u00fd dobr\u00fd line\u00e1rn\u00ed klasifik\u00e1tor, ale pokud spr\u00e1vn\u011b pod\u00e1me znaky a nastav\u00edme dobrou zp\u011btnou vazbu od u\u017eivatel\u016f, bude fungovat mnohem l\u00e9pe ne\u017e modely typu State-Of-The-Art se v\u0161emi vymo\u017eenostmi.<\/p>\n<h4>Cyklus zp\u011btn\u00e9 vazby<\/h4>\n<p>\nTento cyklus spojuje sb\u011br dat, dolad\u011bn\u00ed a nasazen\u00ed \u2014 v podstat\u011b cel\u00fd cyklus aktualizace modelu. Pro\u010d je to d\u016fle\u017eit\u00e9? Pod\u00edvejte se na graf registrace v e-mailu:<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/f3c98bd49a101754cf299821a40e2b8e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdvoj\u00e1\u0159 strojov\u00e9ho u\u010den\u00ed implementoval model antibota, kter\u00fd br\u00e1n\u00ed bot\u016fm registraci v e-mailu. Graf kles\u00e1 na hodnotu, kde z\u016fst\u00e1vaj\u00ed pouze skute\u010dn\u00ed u\u017eivatel\u00e9. V\u0161e je skv\u011bl\u00e9! Ale po \u010dty\u0159ech hodin\u00e1ch, boti p\u0159izp\u016fsobuj\u00ed sv\u00e9 skripty, a v\u0161e se vrac\u00ed do star\u00fdch kolej\u00ed. V t\u00e9to implementaci str\u00e1vil v\u00fdvoj\u00e1\u0159 m\u011bs\u00edc t\u00edm, \u017ee p\u0159id\u00e1val znaky a doladil model, ale spammer se za \u010dty\u0159i hodiny dok\u00e1zal adaptovat.<\/p>\n<p>Aby to nebylo tak bolestiv\u00e9 a pozd\u011bji se nemuselo v\u0161e p\u0159ed\u011bl\u00e1vat, je t\u0159eba zpo\u010d\u00e1tku myslet na to, jak bude cyklus zp\u011btn\u00e9 vazby vypadat a co budeme d\u011blat, pokud se prost\u0159ed\u00ed zm\u011bn\u00ed. Za\u010dn\u011bme se sb\u011brem dat \u2014 to je palivo pro na\u0161e algoritmy.<\/p>\n<h2>Sb\u011br dat<\/h2>\n<p>\nJe jasn\u00e9, \u017ee modern\u00edm neuronov\u00fdm s\u00edt\u00edm plat\u00ed, \u010d\u00edm v\u00edce dat, t\u00edm l\u00e9pe, a ta vlastn\u011b generuj\u00ed u\u017eivatel\u00e9 produktu. Mohou n\u00e1m pom\u00e1hat u\u017eivatel\u00e9, kte\u0159\u00ed ozna\u010duj\u00ed data, ale nem\u011bli bychom toho zneu\u017e\u00edvat, proto\u017ee u\u017eivatel\u016fm se v ur\u010dit\u00e9m okam\u017eiku p\u0159estane cht\u00edt dotv\u00e1\u0159et va\u0161e modely a p\u0159ejdou na jin\u00fd produkt. <\/p>\n<p>Jednou z nej\u010dast\u011bj\u0161\u00edch chyb (tady odkazuj\u00ed na Andrew Ng) \u2014 p\u0159\u00edli\u0161 siln\u00e1 orientace na metriky na testovac\u00edm datov\u00e9m souboru, a nikoli na zp\u011btnou vazbu od u\u017eivatele, co\u017e je ve skute\u010dnosti hlavn\u00edm m\u011b\u0159\u00edtkem kvality pr\u00e1ce, proto\u017ee vytv\u00e1\u0159\u00edme produkt pro u\u017eivatele. Pokud u\u017eivateli nen\u00ed jasn\u00e9 nebo se mu nel\u00edb\u00ed fungov\u00e1n\u00ed modelu, znamen\u00e1 to, \u017ee v\u0161e je marn\u00e9. <\/p>\n<p>Proto by u\u017eivatel m\u011bl v\u017edy m\u00edt mo\u017enost hlasovat, je t\u0159eba mu d\u00e1t n\u00e1stroj pro zp\u011btnou vazbu. Pokud se domn\u00edv\u00e1me, \u017ee do schr\u00e1nky p\u0159i\u0161el e-mail t\u00fdkaj\u00edc\u00ed se financ\u00ed, je t\u0159eba jej ozna\u010dit jako \u201efinance\u201c a nakreslit tla\u010d\u00edtko, kter\u00e9 u\u017eivatel m\u016f\u017ee stisknout a \u0159\u00edct, \u017ee to nejsou finance.<\/p>\n<h3>Kvalita zp\u011btn\u00e9 vazby<\/h3>\n<p>\nPoj\u010fme si promluvit o kvalit\u011b u\u017eivatelsk\u00e9 zp\u011btn\u00e9 vazby. Za prv\u00e9, vy a u\u017eivatel\u00e9 m\u016f\u017eete m\u00edt r\u016fzn\u00e9 p\u0159edstavy o jednom pojmu. Nap\u0159\u00edklad vy s produktov\u00fdmi mana\u017eery pova\u017eujete finance za bankovn\u00ed dopisy, zat\u00edmco u\u017eivatel si mysl\u00ed, \u017ee dopis od babi\u010dky o d\u016fchodu tak\u00e9 spad\u00e1 do kategorie financ\u00ed. Za druh\u00e9, existuj\u00ed u\u017eivatel\u00e9, kte\u0159\u00ed bezmy\u0161lenkovit\u011b klikaj\u00ed na tla\u010d\u00edtka bez jak\u00e9koli logiky. Za t\u0159et\u00ed, u\u017eivatel m\u016f\u017ee b\u00fdt hluboce na omylu ve sv\u00fdch z\u00e1v\u011brech. V\u00fdrazn\u00fdm p\u0159\u00edkladem z na\u0161\u00ed praxe je implementace klasifik\u00e1toru <noindex><a rel=\"nofollow\" href=\"https:\/\/ru.wikipedia.org\/wiki\/%D0%9D%D0%B8%D0%B3%D0%B5%D1%80%D0%B8%D0%B9%D1%81%D0%BA%D0%B8%D0%B5_%D0%BF%D0%B8%D1%81%D1%8C%D0%BC%D0%B0\">nigersk\u00e9ho spamu<\/a><\/noindex>, velmi vtipn\u00e9ho typu spamu, kdy se u\u017eivateli nab\u00edz\u00ed, aby si vyzvedl n\u011bkolik milion\u016f dolar\u016f od n\u00e1hle nalezen\u00e9ho vzd\u00e1len\u00e9ho p\u0159\u00edbuzn\u00e9ho v Africe. Po zaveden\u00ed tohoto klasifik\u00e1toru jsme zkontrolovali kliknut\u00ed na \"Nen\u00ed spam\" u t\u011bchto e-mail\u016f a uk\u00e1zalo se, \u017ee 80 % z nich je \u0161\u0165avnat\u00fd nigersk\u00fd spam, co\u017e nazna\u010duje, \u017ee u\u017eivatel\u00e9 mohou b\u00fdt extr\u00e9mn\u011b d\u016fv\u011b\u0159iv\u00ed.<\/p>\n<p>A nezapom\u00ednejme, \u017ee na tla\u010d\u00edtka mohou klepat nejen lid\u00e9, ale i r\u016fzn\u00e9 boty, kter\u00e9 se vyd\u00e1vaj\u00ed za prohl\u00ed\u017ee\u010d. Tak\u017ee surov\u00e1 zp\u011btn\u00e1 vazba nen\u00ed vhodn\u00e1 pro u\u010den\u00ed. Co se s t\u011bmito informacemi d\u00e1 d\u011blat?<\/p>\n<p>Pou\u017e\u00edv\u00e1me dva p\u0159\u00edstupy: <\/p>\n<ul>\n<li><b>Zp\u011btn\u00e1 vazba ze spojen\u00e9ho ML<\/b>. Nap\u0159\u00edklad m\u00e1me online syst\u00e9m proti bot\u016fm, kter\u00fd, jak u\u017e jsem zm\u00ednil, p\u0159ij\u00edm\u00e1 rychl\u00e1 rozhodnut\u00ed na z\u00e1klad\u011b omezen\u00e9ho mno\u017estv\u00ed rys\u016f. A m\u00e1me druh\u00fd, pomalej\u0161\u00ed syst\u00e9m, kter\u00fd pracuje ex post. Ten m\u00e1 v\u00edce dat o u\u017eivateli, jeho chov\u00e1n\u00ed atd. V d\u016fsledku toho se p\u0159ij\u00edm\u00e1 nejv\u00edce uv\u00e1\u017een\u00e9 rozhodnut\u00ed, tak\u017ee m\u00e1 vy\u0161\u0161\u00ed p\u0159esnost a \u00faplnost. Rozd\u00edl v \u010dinnosti t\u011bchto syst\u00e9m\u016f lze pou\u017e\u00edt jako data pro u\u010den\u00ed pro prvn\u00ed. Takto se jednodu\u0161\u0161\u00ed syst\u00e9m v\u017edy pokus\u00ed p\u0159ibl\u00ed\u017eit v\u00fdkonu slo\u017eit\u011bj\u0161\u00edho.\n<\/li>\n<li><b>Klasifikace klik\u016f<\/b>. M\u016f\u017eeme jednodu\u0161e klasifikovat ka\u017ed\u00fd klik u\u017eivatele, hodnotit jeho platnost a mo\u017enost vyu\u017eit\u00ed. Takto pracujeme v antispamu pro e-maily, pou\u017e\u00edv\u00e1me znaky u\u017eivatele, jeho historii, charakteristiky odes\u00edlatele, samotn\u00fd text a v\u00fdsledek pr\u00e1ce klasifik\u00e1tor\u016f. Nakonec z\u00edsk\u00e1v\u00e1me automatick\u00fd syst\u00e9m, kter\u00fd validuje zp\u011btnou vazbu od u\u017eivatel\u016f. A proto\u017ee ho mus\u00edme p\u0159eu\u010dovat podstatn\u011b m\u00e9n\u011b \u010dasto, m\u016f\u017ee se jeho pr\u00e1ce st\u00e1t z\u00e1kladem pro v\u0161echny ostatn\u00ed syst\u00e9my. Hlavn\u00edm prioritou v tomto modelu je precision, proto\u017ee tr\u00e9nink modelu na nep\u0159esn\u00fdch datech m\u016f\u017ee m\u00edt z\u00e1va\u017en\u00e9 n\u00e1sledky. \n<\/li>\n<\/ul>\n<p>\nZat\u00edmco \u010dist\u00ed data a p\u0159eu\u010dujeme na\u0161e ML syst\u00e9my, nesm\u00edme zapom\u00ednat na u\u017eivatele, nebo\u0165 pro n\u00e1s jsou tis\u00edce a miliony chyb na grafu statistikou, ale pro u\u017eivatele je ka\u017ed\u00e1 chyba trag\u00e9di\u00ed. Krom\u011b toho, \u017ee u\u017eivatel mus\u00ed n\u011bjak \u017e\u00edt s va\u0161\u00ed chybou v produktu, po zp\u011btn\u00e9 vazb\u011b o\u010dek\u00e1v\u00e1, \u017ee podobn\u00e1 situace se v budoucnu nevyskytn\u011b. Proto je v\u017edy dobr\u00e9 d\u00e1vat u\u017eivatel\u016fm nejen mo\u017enost hlasovat, ale tak\u00e9 upravit chov\u00e1n\u00ed ML syst\u00e9m\u016f, nap\u0159\u00edklad vytv\u00e1\u0159en\u00edm osobn\u00edch heuristik na ka\u017ed\u00fd klik zp\u011btn\u00e9 vazby, v p\u0159\u00edpad\u011b e-mailu by to mohla b\u00fdt mo\u017enost filtrovat podobn\u00e9 e-maily podle odes\u00edlatele a p\u0159edm\u011btu pro tohoto u\u017eivatele.<\/p>\n<p>Tak\u00e9 je pot\u0159eba na z\u00e1klad\u011b n\u011bjak\u00fdch report\u016f nebo kontakt\u016f se supportem polovi\u010dn\u011b automatizovan\u011b nebo ru\u010dn\u011b vylep\u0161ovat model, aby jin\u00ed u\u017eivatel\u00e9 netrp\u011bli podobn\u00fdmi probl\u00e9my.<\/p>\n<h3>Heuristiky pro u\u010den\u00ed<\/h3>\n<p>\nS t\u011bmito heuristikami a \u00fapravami jsou dv\u011b probl\u00e9my. Prvn\u00ed spo\u010d\u00edv\u00e1 v tom, \u017ee st\u00e1le rostouc\u00ed po\u010det \u00faprav je t\u011b\u017ek\u00e9 udr\u017eovat, nemluv\u011b o jejich kvalit\u011b a funk\u010dnosti na del\u0161\u00ed vzd\u00e1lenost. Druh\u00fd probl\u00e9m spo\u010d\u00edv\u00e1 v tom, \u017ee chyba nemus\u00ed b\u00fdt \u010dast\u00e1 a p\u00e1r kliknut\u00ed pro p\u0159eu\u010den\u00ed modelu m\u016f\u017ee b\u00fdt nedostate\u010dn\u00e9. Zd\u00e1lo by se, \u017ee tyto dva nesouvisej\u00edc\u00ed jevy lze v\u00fdrazn\u011b minimalizovat, pokud pou\u017eijeme n\u00e1sleduj\u00edc\u00ed p\u0159\u00edstup.<\/p>\n<ol>\n<li>Vytvo\u0159\u00edme do\u010dasnou \u00fapravu. \n<\/li>\n<li>Sm\u011b\u0159ujeme \u00fadaje z n\u00ed do modelu, kter\u00fd se pravideln\u011b p\u0159eu\u010duje, v\u010detn\u011b z\u00edskan\u00fdch dat. Zde je samoz\u0159ejm\u011b d\u016fle\u017eit\u00e9, aby heuristika m\u011bla vysokou p\u0159esnost, aby nedoch\u00e1zelo ke zhor\u0161en\u00ed kvality dat v tr\u00e9ninkov\u00e9 sad\u011b. \n<\/li>\n<li>Pot\u00e9 nastav\u00edme monitorov\u00e1n\u00ed na aktivaci workaroundu a pokud po n\u011bjak\u00e9m \u010dase workaround ji\u017e nebude aktivov\u00e1n a pln\u011b bude pokryt modelem, m\u016f\u017eeme jej bez obav odstranit. Tato probl\u00e9m se pravd\u011bpodobn\u011b neobjev\u00ed znovu.\n<\/li>\n<\/ol>\n<p>\nTak\u017ee arm\u00e1da workaround\u016f je velmi u\u017eite\u010dn\u00e1. Hlavn\u00ed je, aby jejich pou\u017eit\u00ed bylo do\u010dasn\u00e9, a ne trval\u00e9. <\/p>\n<h2>Dal\u0161\u00ed \u0161kolen\u00ed<\/h2>\n<p>\nDal\u0161\u00ed \u0161kolen\u00ed je proces p\u0159id\u00e1v\u00e1n\u00ed nov\u00fdch dat z\u00edskan\u00fdch na z\u00e1klad\u011b zp\u011btn\u00e9 vazby od u\u017eivatel\u016f \u010di jin\u00fdch syst\u00e9m\u016f a \u0161kolen\u00ed st\u00e1vaj\u00edc\u00edho modelu na t\u011bchto datech. U dal\u0161\u00edho \u0161kolen\u00ed m\u016f\u017ee nastat n\u011bkolik probl\u00e9m\u016f:<\/p>\n<ol>\n<li>Model m\u016f\u017ee jednodu\u0161e nepodporovat dal\u0161\u00ed \u0161kolen\u00ed a u\u010dit se pouze od nuly. \n<\/li>\n<li>N\u011bkde v knize p\u0159\u00edrody nen\u00ed naps\u00e1no, \u017ee dal\u0161\u00ed \u0161kolen\u00ed nutn\u011b zlep\u0161\u00ed kvalitu pr\u00e1ce v produkci. \u010casto se st\u00e1v\u00e1 prav\u00fd opak, tj. je mo\u017en\u00e9 pouze zhor\u0161en\u00ed.\n<\/li>\n<li>Zm\u011bny mohou b\u00fdt nep\u0159edv\u00eddateln\u00e9. To je celkem jemn\u00fd bod, kter\u00fd jsme si uv\u011bdomili. I kdy\u017e nov\u00e1 model v A\/B testu vykazuje podobn\u00e9 v\u00fdsledky jako sou\u010dasn\u00fd, to je\u0161t\u011b neznamen\u00e1, \u017ee bude fungovat identicky. Jejich \u010dinnost se m\u016f\u017ee li\u0161it v n\u011bjak\u00e9m jednom procentu, kter\u00e9 m\u016f\u017ee p\u0159in\u00e9st nov\u00e9 chyby nebo vr\u00e1tit ji\u017e opraven\u00e9 star\u00e9. Se st\u00e1vaj\u00edc\u00edmi chybami si jak my, tak u\u017eivatel\u00e9 ji\u017e um\u00edme poradit, a kdy\u017e se objev\u00ed velk\u00e9 mno\u017estv\u00ed nov\u00fdch chyb, u\u017eivatel tak\u00e9 nemus\u00ed ch\u00e1pat, co se d\u011bje, proto\u017ee o\u010dek\u00e1v\u00e1 p\u0159edv\u00eddateln\u00e9 chov\u00e1n\u00ed.\n<\/li>\n<\/ol>\n<p>\nProto je nejd\u016fle\u017eit\u011bj\u0161\u00ed v dal\u0161\u00edm \u0161kolen\u00ed zaru\u010den\u011b zlep\u0161it model, nebo alespo\u0148 nezhor\u0161ovat jej. <\/p>\n<p>Prvn\u00ed, co m\u011b napadne, kdy\u017e hovo\u0159\u00edme o dal\u0161\u00edm \u0161kolen\u00ed, je p\u0159\u00edstup Aktivn\u00ed u\u010den\u00ed. Co to znamen\u00e1? Nap\u0159\u00edklad klasifik\u00e1tor ur\u010duje, zda zpr\u00e1va pat\u0159\u00ed do financ\u00ed, a kolem jeho hranice rozhodov\u00e1n\u00ed p\u0159id\u00e1v\u00e1me vzorek ozna\u010den\u00fdch p\u0159\u00edklad\u016f. To dob\u0159e funguje nap\u0159\u00edklad v reklam\u011b, kde je zp\u011btn\u00e9 vazby velmi mnoho a model lze \u0161kolit v re\u00e1ln\u00e9m \u010dase. A pokud je zp\u011btn\u00e9 vazby m\u00e1lo, pak dost\u00e1v\u00e1me siln\u011b zkreslen\u00fd vzorek ve srovn\u00e1n\u00ed s produk\u010dn\u00edm rozd\u011blen\u00edm dat, na z\u00e1klad\u011b kter\u00e9ho nelze posoudit chov\u00e1n\u00ed modelu v pr\u016fb\u011bhu provozu.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/fd1b3e07bfaf896dde3248e43537f61a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa\u0161\u00edm skute\u010dn\u00fdm c\u00edlem je zachovat star\u00e9 vzory, ji\u017e zn\u00e1m\u00e9 modely, a z\u00edskat nov\u00e9. D\u016fle\u017eit\u00e1 je kontinuita. Model, kter\u00fd jsme \u010dasto s velk\u00fdmi obt\u00ed\u017eemi uvedli do provozu, u\u017e funguje, tak\u017ee se m\u016f\u017eeme orientovat na jeho v\u00fdkon. <\/p>\n<p>V po\u0161tovn\u00ed slu\u017eb\u011b se pou\u017e\u00edvaj\u00ed r\u016fzn\u00e9 modely: stromy, line\u00e1rn\u00ed modely, neuronov\u00e9 s\u00edt\u011b. Pro ka\u017ed\u00fd vytv\u00e1\u0159\u00edme sv\u016fj vlastn\u00ed algoritmus pro dal\u0161\u00ed u\u010den\u00ed. V procesu dal\u0161\u00edho u\u010den\u00ed z\u00edsk\u00e1v\u00e1me nejen nov\u00e9 \u00fadaje, ale \u010dasto tak\u00e9 nov\u00e9 znaky, kter\u00e9 zohledn\u00edme ve v\u0161ech n\u00ed\u017ee uveden\u00fdch algoritmech.<\/p>\n<h3>Line\u00e1rn\u00ed modely<\/h3>\n<p>\nP\u0159edstavme si, \u017ee m\u00e1me logistickou regresi. Sestavujeme ztr\u00e1tu modelu z n\u00e1sleduj\u00edc\u00edch komponent:<\/p>\n<ul>\n<li>LogLoss na nov\u00fdch \u00fadaj\u00edch;\n<\/li>\n<li>pravideln\u011b upravujeme v\u00e1hy nov\u00fdch znak\u016f (star\u00e9 nem\u011bn\u00edme);\n<\/li>\n<li>u\u010d\u00edme se i na star\u00fdch \u00fadaj\u00edch, abychom zachovali star\u00e9 vzory;\n<\/li>\n<li>a, mo\u017en\u00e1 nejd\u016fle\u017eit\u011bj\u0161\u00ed: zav\u00e1d\u00edme Harmonic Regularization, kter\u00e1 zaru\u010duje, \u017ee v\u00e1hy se \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e star\u00e9ho modelu p\u0159\u00edli\u0161 nezm\u011bn\u00ed.\n<\/li>\n<\/ul>\n<p>\nJeliko\u017e m\u00e1 ka\u017ed\u00e1 komponenta Loss sv\u00e9 koeficienty, m\u016f\u017eeme nal\u00e9zt optim\u00e1ln\u00ed hodnoty pro na\u0161i \u00falohu na kros-validaci nebo na z\u00e1klad\u011b produktov\u00fdch po\u017eadavk\u016f.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/00ae86d23afb780f5e260e61f833d4bd.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Stromy<\/h3>\n<p>\nP\u0159ejdeme k rozhodovac\u00edm strom\u016fm. Vytvo\u0159ili jsme n\u00e1sleduj\u00edc\u00ed algoritmus pro dal\u0161\u00ed u\u010den\u00ed strom\u016f:<\/p>\n<ol>\n<li>Na produkci pracuje les 100\u2014300 strom\u016f, kter\u00fd byl \u043e\u0431\u0443\u0447\u0435\u043d na star\u00e9m datov\u00e9m souboru.\n<\/li>\n<li>Na konci odstran\u00edme M = 5 kus\u016f a p\u0159id\u00e1me 2M = 10 nov\u00fdch, kter\u00e9 byly u\u010den\u00e9 na cel\u00e9m datov\u00e9m souboru, ale s vysokou v\u00e1hou na nov\u00fdch \u00fadaj\u00edch, co\u017e p\u0159irozen\u011b zaru\u010duje inkrement\u00e1ln\u00ed zm\u011bnu modelu.\n<\/li>\n<\/ol>\n<p>\nJe z\u0159ejm\u00e9, \u017ee s \u010dasem se po\u010det strom\u016f zna\u010dn\u011b zvy\u0161uje, a je nutn\u00e9 je pravideln\u011b zmen\u0161ovat, aby se ve\u0161ly do \u010dasov\u00fdch r\u00e1mc\u016f. K tomu pou\u017e\u00edv\u00e1me v\u0161udyp\u0159\u00edtomnou nyn\u00ed Knowledge Distillation (KD). Kr\u00e1tce o principu jeho fungov\u00e1n\u00ed.<\/p>\n<ol>\n<li>M\u00e1me aktu\u00e1ln\u00ed \u201ekomplexn\u00ed\u201c model. Spust\u00edme ho na tr\u00e9ninkov\u00e9m datov\u00e9m souboru a z\u00edsk\u00e1me rozd\u011blen\u00ed pravd\u011bpodobnost\u00ed t\u0159\u00edd na v\u00fdstupu.\n<\/li>\n<li>D\u00e1le u\u010d\u00edme model-\u017e\u00e1ka (model s men\u0161\u00edm po\u010dtem strom\u016f v tomto p\u0159\u00edpad\u011b), aby opakoval v\u00fdsledky pr\u00e1ce modelu, pou\u017e\u00edvaj\u00edc rozd\u011blen\u00ed t\u0159\u00edd jako c\u00edlovou prom\u011bnnou.\n<\/li>\n<li>Je d\u016fle\u017eit\u00e9 poznamenat, \u017ee nikterak nevyu\u017e\u00edv\u00e1me strukturu datov\u00e9 sady, a proto m\u016f\u017eeme pou\u017e\u00edt libovoln\u00e1 data. Samoz\u0159ejm\u011b pou\u017e\u00edv\u00e1me vzorek dat z produk\u010dn\u00edho toku jako tr\u00e9ninkovou sadu pro model studenta. T\u00edmto zp\u016fsobem n\u00e1m tr\u00e9ninkov\u00e1 sada umo\u017e\u0148uje zajistit p\u0159esnost modelu, zat\u00edmco vzorek toku zaru\u010duje podobn\u00fd v\u00fdkon na produk\u010dn\u00edm rozd\u011blen\u00ed, kompenzuj\u00edc odchylku tr\u00e9ninkov\u00e9 sady.\n<\/li>\n<\/ol>\n<p>\n<img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/3fdfcb9b1e5a6fa824226a429afc475a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKombinace t\u011bchto dvou metodik (p\u0159id\u00e1n\u00ed strom\u016f a periodick\u00e9 zmen\u0161ov\u00e1n\u00ed jejich po\u010dtu pomoc\u00ed Knowledge Distillation) zaji\u0161\u0165uje zaveden\u00ed nov\u00fdch vzorc\u016f a plnou kontinuitu.<\/p>\n<p>Pomoc\u00ed KD tak\u00e9 prov\u00e1d\u00edme rozd\u00edln\u00e9 operace s rysy modelu, nap\u0159\u00edklad odstran\u011bn\u00ed rys\u016f a pr\u00e1ci s chyb\u011bj\u00edc\u00edmi hodnotami. V na\u0161em p\u0159\u00edpad\u011b m\u00e1me \u0159adu d\u016fle\u017eit\u00fdch statistick\u00fdch rys\u016f (podle odes\u00edlatel\u016f, textov\u00fdch hash\u016f, URL atd.), kter\u00e9 jsou ulo\u017eeny v datab\u00e1zi, je\u017e m\u00e1 tendenci selh\u00e1vat. Takov\u00e9mu v\u00fdvoji ud\u00e1lost\u00ed model samoz\u0159ejm\u011b nen\u00ed p\u0159ipraven, proto\u017ee v tr\u00e9ninkov\u00e9 sad\u011b nenast\u00e1vaj\u00ed situace selh\u00e1n\u00ed. V podobn\u00fdch p\u0159\u00edpadech kombinujeme techniky KD a augmentace: b\u011bhem tr\u00e9nov\u00e1n\u00ed pro \u010d\u00e1st dat odstra\u0148ujeme nebo nulujeme pot\u0159ebn\u00e9 rysy, zat\u00edmco zna\u010dky (v\u00fdstupy aktu\u00e1ln\u00edho modelu) bereme p\u016fvodn\u00ed, model student se u\u010d\u00ed opakovat toto rozlo\u017een\u00ed.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/5e1f5af9a359646a49f4fe88ffed1025.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u0161imli jsme si, \u017ee \u010d\u00edm v\u00e1\u017en\u011bj\u0161\u00ed manipulace s modely prob\u00edh\u00e1, t\u00edm v\u011bt\u0161\u00ed procento vzorku toku je pot\u0159eba.<\/p>\n<p>Pro odstran\u011bn\u00ed rys\u016f, co\u017e je nejjednodu\u0161\u0161\u00ed operace, sta\u010d\u00ed jen mal\u00e1 \u010d\u00e1st toku, proto\u017ee se m\u011bn\u00ed jen p\u00e1r rys\u016f, a sou\u010dasn\u00fd model se u\u010dil na stejn\u00e9 sad\u011b \u2014 rozd\u00edl je minim\u00e1ln\u00ed. Pro zjednodu\u0161en\u00ed modelu (sn\u00ed\u017een\u00ed po\u010dtu strom\u016f n\u011bkolikr\u00e1t) je u\u017e t\u0159eba 50 na 50. A na chyb\u011bj\u00edc\u00ed d\u016fle\u017eit\u00e9 statistick\u00e9 rysy, kter\u00e9 v\u00e1\u017en\u011b ovliv\u0148uj\u00ed v\u00fdkon modelu, je pot\u0159eba je\u0161t\u011b v\u00edce toku, aby byla vyv\u00e1\u017eena pr\u00e1ce nov\u00e9ho modelu odoln\u00e9ho v\u016f\u010di chyb\u011bj\u00edc\u00edm \u00fadaj\u016fm na v\u0161ech typech e-mail\u016f. <\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/8f3729cfff43be9ecdc47d532daa8d6f.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>FastText<\/h3>\n<p>\nP\u0159ejd\u011bme k FastText. P\u0159ipom\u00edn\u00e1m, \u017ee reprezentace (Embedding) slova se skl\u00e1d\u00e1 ze sou\u010dtu embeddingu samotn\u00e9ho slova a v\u0161ech jeho p\u00edsmenov\u00fdch N-gram, obvykle trigram\u016f. Vzhledem k tomu, \u017ee trigram\u016f m\u016f\u017ee b\u00fdt pom\u011brn\u011b hodn\u011b, pou\u017e\u00edv\u00e1 se Bucket Hashing, tedy transformace cel\u00e9ho prostoru do n\u011bjak\u00e9 fixn\u00ed hash mapy. V\u00fdsledkem je matice vah o rozm\u011brech vnit\u0159n\u00ed vrstvy na po\u010det slov + bucket\u016f. <\/p>\n<p>P\u0159i dolad\u011bn\u00ed se objevuj\u00ed nov\u00e9 rysy: slova a trigramy. V b\u011b\u017en\u00e9m dolad\u011bn\u00ed od Facebooku nedoch\u00e1z\u00ed k ni\u010demu z\u00e1sadn\u00edmu. Dolad\u00ed se pouze star\u00e9 v\u00e1hy s kros-entropi\u00ed na nov\u00fdch datech. Nov\u00e9 rysy tedy nejsou vyu\u017e\u00edv\u00e1ny, tento p\u0159\u00edstup m\u00e1 samoz\u0159ejm\u011b v\u0161echny v\u00fd\u0161e popsan\u00e9 nev\u00fdhody spojen\u00e9 s nep\u0159edv\u00eddatelnost\u00ed modelu v produkci. Proto jsme trochu upravili FastText. P\u0159id\u00e1v\u00e1me v\u0161echny nov\u00e9 v\u00e1hy (slova a trigramy), dolad\u00edme celou matici s kros-entropi\u00ed a p\u0159id\u00e1v\u00e1me harmonickou regularizaci analogicky k line\u00e1rn\u00edmu modeli, kter\u00e1 zaji\u0161\u0165uje nev\u00fdznamnou zm\u011bnu star\u00fdch vah.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/4491639587d1c91ac91d77b587ee0111.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>CNN<\/h3>\n<p>\nSe spletit\u00fdmi s\u00edt\u011bmi je to o n\u011bco slo\u017eit\u011bj\u0161\u00ed. Pokud se v CNN dolad\u00ed posledn\u00ed vrstvy, lze samoz\u0159ejm\u011b pou\u017e\u00edt harmonickou regularizaci a zajistit kontinuitu. Pokud v\u0161ak vy\u017eaduje dolad\u011bn\u00ed cel\u00e9 s\u00edt\u011b, tuto regularizaci ji\u017e nelze aplikovat na v\u0161echny vrstvy. Existuje v\u0161ak mo\u017enost tr\u00e9nov\u00e1n\u00ed komplement\u00e1rn\u00edch embedding\u016f pomoc\u00ed Triplet Loss (<noindex><a rel=\"nofollow\" href=\"https:\/\/arxiv.org\/abs\/1503.03832\">origin\u00e1ln\u00ed \u010dl\u00e1nek<\/a><\/noindex>).<\/p>\n<h4>Triplet Loss<\/h4>\n<p>\nNa p\u0159\u00edkladu \u00fakolu anti-fishing rozebr\u00e1me obecn\u011b Triplet Loss. Bereme na\u0161e logo a pozitivn\u00ed a negativn\u00ed p\u0159\u00edklady loga jin\u00fdch spole\u010dnost\u00ed. Minimalizujeme vzd\u00e1lenost mezi prvn\u00edmi a maximalizujeme vzd\u00e1lenost mezi druh\u00fdmi, d\u011bl\u00e1me to s mal\u00fdm odstupem, abychom zajistili v\u011bt\u0161\u00ed kompaktnost t\u0159\u00edd. <\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/dc8547fa042286798eb5c2f0887c4da6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPokud dolad\u00edme s\u00ed\u0165, m\u011bn\u00ed se n\u00e1m zcela metrick\u00fd prostor a st\u00e1v\u00e1 se absolutn\u011b neslu\u010diteln\u00fdm s p\u0159edchoz\u00edm. To je v\u00e1\u017en\u00fd probl\u00e9m v \u00faloh\u00e1ch pou\u017e\u00edvaj\u00edc\u00edch vektory. Abychom tento probl\u00e9m obeznuli, budeme b\u011bhem tr\u00e9nov\u00e1n\u00ed mixovat star\u00e9 embeddingy.<\/p>\n<p>P\u0159idali jsme nov\u00e1 data do tr\u00e9ninkov\u00e9 sady a znovu tr\u00e9nujeme druhou verzi modelu. V druh\u00e9 f\u00e1zi na\u0161i s\u00ed\u0165 dotr\u00e9nov\u00e1v\u00e1me (finetuning): nejprve se dotr\u00e9nov\u00e1v\u00e1 posledn\u00ed vrstva, pot\u00e9 se odemyk\u00e1 cel\u00e1 s\u00ed\u0165. P\u0159i sestavov\u00e1n\u00ed trojic po\u010d\u00edt\u00e1me pouze \u010d\u00e1st embedding\u016f pomoc\u00ed tr\u00e9novateln\u00e9ho modelu, ostatn\u00ed pomoc\u00ed star\u00e9ho. T\u00edmto zp\u016fsobem b\u011bhem dotr\u00e9nov\u00e1n\u00ed zaji\u0161\u0165ujeme kompatibilitu metrick\u00fdch prostor\u016f v1 a v2. Je to jak\u00e1si forma harmonick\u00e9 regularizace.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/854d4fcc97775b24e6863cc38743cd27.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Architektura jako celek<\/h3>\n<p>\nPokud se pod\u00edv\u00e1me na cel\u00fd syst\u00e9m na p\u0159\u00edkladu antispamu, modely nejsou izolovan\u00e9, ale vz\u00e1jemn\u011b do sebe zapadaj\u00ed. Bereme obr\u00e1zky, text a dal\u0161\u00ed znaky, pomoc\u00ed CNN a Fast Text z\u00edsk\u00e1v\u00e1me embeddings. N\u00e1sledn\u011b se na embeddingy aplikuj\u00ed klasifik\u00e1tory, kter\u00e9 poskytuj\u00ed sk\u00f3re pro r\u016fzn\u00e9 t\u0159\u00eddy (typy zpr\u00e1v, spam, p\u0159\u00edtomnost loga). Sk\u00f3re a znaky u\u017e vstupuj\u00ed do lesa strom\u016f pro kone\u010dn\u00e9 rozhodov\u00e1n\u00ed. Odd\u011blen\u00e9 klasifik\u00e1tory v tomto sch\u00e9matu umo\u017e\u0148uj\u00ed l\u00e9pe interpretovat v\u00fdsledky pr\u00e1ce syst\u00e9mu a c\u00edlen\u011bji dotr\u00e9nov\u00e1vat komponenty v p\u0159\u00edpad\u011b probl\u00e9m\u016f, sp\u00ed\u0161e ne\u017e poskytovat v\u0161echna data do rozhodovac\u00edch strom\u016f v surov\u00e9 podob\u011b.<\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/30effc5ef7398db5f085b6dd415647d3.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV kone\u010dn\u00e9m d\u016fsledku zaji\u0161\u0165ujeme kontinuitu na ka\u017ed\u00e9 \u00farovni. Na doln\u00ed \u00farovni v CNN a Fast Text pou\u017e\u00edv\u00e1me harmonickou regularizaci, pro klasifik\u00e1tory uprost\u0159ed tak\u00e9 harmonickou regularizaci a kalibraci sk\u00f3re pro kompatibilitu rozd\u011blen\u00ed pravd\u011bpodobnosti. No a boostov\u00e1n\u00ed strom\u016f se tr\u00e9nuje inkrement\u00e1ln\u011b nebo pomoc\u00ed Knowledge Distillation.<\/p>\n<p>Obecn\u011b podpora takov\u00e9ho vno\u0159en\u00e9ho syst\u00e9mu strojov\u00e9ho u\u010den\u00ed obvykle p\u0159edstavuje probl\u00e9m, proto\u017ee jak\u00e1koli sou\u010d\u00e1st na doln\u00ed \u00farovni vede k aktualizaci cel\u00e9ho syst\u00e9mu v\u00fd\u0161e. Ale proto\u017ee v na\u0161em nastaven\u00ed se ka\u017ed\u00e1 sou\u010d\u00e1st m\u011bn\u00ed jen nepatrn\u011b a je kompatibiln\u00ed s p\u0159edchoz\u00edm, m\u016f\u017ee se cel\u00fd syst\u00e9m aktualizovat po \u010d\u00e1stech, ani\u017e by bylo nutn\u00e9 znovu u\u010dit celou strukturu, co\u017e umo\u017e\u0148uje udr\u017eovat ji bez v\u00e1\u017en\u00e9ho p\u0159ekro\u010den\u00ed n\u00e1klad\u016f. <\/p>\n<h2>Nasazen\u00ed<\/h2>\n<p>\nRozebrali jsme shroma\u017e\u010fov\u00e1n\u00ed dat a dotr\u00e9nov\u00e1v\u00e1n\u00ed r\u016fzn\u00fdch typ\u016f model\u016f, nyn\u00ed p\u0159ech\u00e1z\u00edme k jejich nasazen\u00ed do produk\u010dn\u00edho prost\u0159ed\u00ed.<\/p>\n<h3>A\/B testov\u00e1n\u00ed<\/h3>\n<p>\nJak jsem ji\u017e zm\u00ednil d\u0159\u00edve, p\u0159i shroma\u017e\u010fov\u00e1n\u00ed dat obvykle z\u00edsk\u00e1v\u00e1me zkreslen\u00fd vzorek, na jeho\u017e z\u00e1klad\u011b nelze posoudit v\u00fdkon modelu v produkci. Proto je p\u0159i nasazen\u00ed modelu nezbytn\u00e9 porovnat ho s p\u0159edchoz\u00ed verz\u00ed, abychom vid\u011bli, jak to skute\u010dn\u011b vypad\u00e1, tedy prov\u00e1d\u011bt A\/B testy. Ve skute\u010dnosti je proces zav\u00e1d\u011bn\u00ed a anal\u00fdzy graf\u016f pom\u011brn\u011b rutinn\u00ed a je skv\u011ble automatizovateln\u00fd. Na\u0161e modely nasazujeme postupn\u011b na 5 %, 30 %, 50 % a nakonec na 100 % u\u017eivatel\u016f, p\u0159i\u010dem\u017e shroma\u017e\u010fujeme v\u0161echny dostupn\u00e9 metriky t\u00fdkaj\u00edc\u00ed se odezvy modelu a zp\u011btn\u00e9 vazby od u\u017eivatel\u016f. V p\u0159\u00edpad\u011b n\u011bjak\u00fdch z\u00e1va\u017en\u00fdch v\u00fdkyv\u016f model automaticky vrac\u00edme zp\u011bt, a v ostatn\u00edch p\u0159\u00edpadech, po z\u00edsk\u00e1n\u00ed dostate\u010dn\u00e9ho po\u010dtu kliknut\u00ed od u\u017eivatel\u016f, rozhodujeme o zv\u00fd\u0161en\u00ed procenta. Nakonec dost\u00e1v\u00e1me nov\u00fd model a\u017e na 50 % u\u017eivatel\u016f \u00fapln\u011b automaticky, a schv\u00e1len\u00ed nasazen\u00ed cel\u00e9 publikum prov\u00e1d\u00ed \u010dlov\u011bk, i kdy\u017e tento krok lze tak\u00e9 automatizovat.<\/p>\n<p>Av\u0161ak proces A\/B testov\u00e1n\u00ed poskytuje prostor pro optimalizaci. Skute\u010dnost je takov\u00e1, \u017ee jak\u00fdkoli A\/B test je dost dlouh\u00fd (v na\u0161em p\u0159\u00edpad\u011b trv\u00e1 od 6 do 24 hodin v z\u00e1vislosti na mno\u017estv\u00ed zp\u011btn\u00e9 vazby), co\u017e jej \u010din\u00ed pom\u011brn\u011b n\u00e1kladn\u00fdm a s omezen\u00fdmi zdroji. Krom\u011b toho je pot\u0159eba dostate\u010dn\u011b vysok\u00e9 procento toku pro test, aby se v podstat\u011b urychlil celkov\u00fd \u010das A\/B testu (z\u00edsk\u00e1n\u00ed statisticky v\u00fdznamn\u00e9ho vzorku pro hodnocen\u00ed metrik na mal\u00e9m procentu m\u016f\u017ee zabrat velmi dlouho), co\u017e \u010din\u00ed po\u010det A\/B slot\u016f velmi omezen\u00fdm. Je jasn\u00e9, \u017ee mus\u00edme do testu za\u0159adit pouze ty nejperspektivn\u011bj\u0161\u00ed modely, kter\u00fdch v pr\u016fb\u011bhu dolad\u011bn\u00ed z\u00edsk\u00e1v\u00e1me opravdu hodn\u011b.<\/p>\n<p>Pro \u0159e\u0161en\u00ed tohoto \u00fakolu jsme vy\u0161kolili samostatn\u00fd klasifik\u00e1tor, kter\u00fd p\u0159edpov\u00edd\u00e1 \u00fasp\u011b\u0161nost A\/B testu. Jako charakteristiky pou\u017e\u00edv\u00e1me statistiku rozhodov\u00e1n\u00ed, Precision, Recall a dal\u0161\u00ed metriky na tr\u00e9novac\u00ed mno\u017ein\u011b, na odd\u011blen\u00e9 a na vzorku z toku. D\u00e1le srovn\u00e1v\u00e1me model s aktu\u00e1ln\u00edm modelem v produkci, s heuristikami, a zohled\u0148ujeme slo\u017eitost modelu. Pou\u017eit\u00edm v\u0161ech t\u011bchto charakteristik klasifik\u00e1tor vy\u0161kolen\u00fd na historii test\u016f hodnot\u00ed kandid\u00e1tsk\u00e9 modely, v na\u0161em p\u0159\u00edpad\u011b jsou to lesy strom\u016f, a rozhoduje, kter\u00fd z nich pustit do A\/B testu. <\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/41b121987209a663075be399e83f5a50.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159i implementaci tento p\u0159\u00edstup v\u00fdrazn\u011b zv\u00fd\u0161il po\u010det \u00fasp\u011b\u0161n\u00fdch A\/B test\u016f.<\/p>\n<h3>Testov\u00e1n\u00ed &amp; monitoring<\/h3>\n<p>\nTestov\u00e1n\u00ed a monitoring, jak podivn\u00e9 se to m\u016f\u017ee zd\u00e1t, ne\u0161kod\u00ed na\u0161emu zdrav\u00ed, naopak, zlep\u0161uj\u00ed ho a zbavuj\u00ed zbyte\u010dn\u00e9ho stresu. Testov\u00e1n\u00ed pom\u00e1h\u00e1 p\u0159edej\u00edt chyb\u00e1m, zat\u00edmco monitoring umo\u017e\u0148uje v\u010dasn\u00e9 odhalen\u00ed probl\u00e9m\u016f, aby se minimalizoval jejich dopad na u\u017eivatele.<\/p>\n<p>Je d\u016fle\u017eit\u00e9 si uv\u011bdomit, \u017ee d\u0159\u00edve \u010di pozd\u011bji dojde k chyb\u011b ve va\u0161\u00ed syst\u00e9mu \u2014 je to spojeno s cyklem v\u00fdvoje jak\u00e9hokoli softwaru. Na za\u010d\u00e1tku v\u00fdvoje syst\u00e9mu se \u010dasto vyskytuje mnoho chyb, dokud se v\u0161e neusad\u00ed a nezavr\u0161uj\u00ed se hlavn\u00ed f\u00e1ze inovac\u00ed. Ale \u010dasem entropie znovu ude\u0159\u00ed a objevuj\u00ed se chyby \u2014 v d\u016fsledku degradace komponent a zm\u011bn dat, o kter\u00fdch jsem se zm\u00ednil na za\u010d\u00e1tku.<\/p>\n<p>R\u00e1d bych zde zd\u016fraznil, \u017ee jak\u00fdkoli syst\u00e9m strojov\u00e9ho u\u010den\u00ed je t\u0159eba posuzovat z pohledu jeho p\u0159\u00ednosu b\u011bhem cel\u00e9ho \u017eivotn\u00edho cyklu. N\u00ed\u017ee na grafu je p\u0159\u00edklad \u010dinnosti syst\u00e9mu, kter\u00fd detekuje vz\u00e1cn\u00fd typ spamu (na grafu \u010d\u00e1ra bl\u00edzko nuly). Jednou vinou \u0161patn\u011b \u043a\u0435\u0448ovan\u00e9ho znaku se syst\u00e9m dostal do pot\u00ed\u017e\u00ed. Bohu\u017eel nebylo monitorov\u00e1n\u00ed na anom\u00e1ln\u00ed chov\u00e1n\u00ed, a tak syst\u00e9m za\u010dal ukl\u00e1dat e-maily do slo\u017eky \u201espam\u201c na hranici rozhodov\u00e1n\u00ed v obrovsk\u00e9m mno\u017estv\u00ed. I kdy\u017e byly n\u00e1sledky napraveny, syst\u00e9m se ji\u017e tolikr\u00e1t zm\u00fdlil, \u017ee se nevyplat\u00ed ani za p\u011bt let. A to je kompletn\u00ed propad\u00e1k z pohledu \u017eivotn\u00edho cyklu modelu. <\/p>\n<p><img decoding=\"async\" alt=\"Exploatace strojov\u00e9ho u\u010den\u00ed v po\u0161t\u011b Mail.ru\" src=\"\/wp-content\/uploads\/2019\/11\/65efd799537ceea3761cabc3a764ba90.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nProto m\u016f\u017ee b\u00fdt takov\u00e1 jednoduch\u00e1 v\u011bc jako monitoring kl\u00ed\u010dov\u00e1 pro \u017eivot modelu. Krom\u011b standardn\u00edch a o\u010dividn\u00fdch metrik sledujeme rozlo\u017een\u00ed odpov\u011bd\u00ed a sk\u00f3re modelu, stejn\u011b jako rozlo\u017een\u00ed hodnot kl\u00ed\u010dov\u00fdch znak\u016f. Pomoc\u00ed KL divergence m\u016f\u017eeme porovn\u00e1vat aktu\u00e1ln\u00ed rozlo\u017een\u00ed s historick\u00fdm nebo hodnoty z A\/B testu s ostatn\u00edm proudem, co\u017e n\u00e1m umo\u017e\u0148uje odhalovat anom\u00e1lie v modelu a v\u010das vracet zm\u011bny.<\/p>\n<p>V\u011bt\u0161inou spou\u0161t\u00edme na\u0161e prvn\u00ed verze syst\u00e9m\u016f pomoc\u00ed jednoduch\u00fdch heuristik nebo model\u016f, kter\u00e9 pozd\u011bji pou\u017e\u00edv\u00e1me k monitorov\u00e1n\u00ed. Nap\u0159\u00edklad monitorujeme NER model ve srovn\u00e1n\u00ed s regul\u00e1rn\u00edmi v\u00fdrazy pro konkr\u00e9tn\u00ed internetov\u00e9 obchody, a pokud pokryt\u00ed klasifik\u00e1toru kles\u00e1 v porovn\u00e1n\u00ed s nimi, zab\u00fdv\u00e1me se p\u0159\u00ed\u010dinami. Je to dal\u0161\u00ed u\u017eite\u010dn\u00e9 vyu\u017eit\u00ed heuristik!<\/p>\n<h2>Z\u00e1v\u011bry<\/h2>\n<p>\nPoj\u010fme si je\u0161t\u011b jednou proj\u00edt kl\u00ed\u010dov\u00e9 my\u0161lenky \u010dl\u00e1nku.<\/p>\n<ul>\n<li><b>Fibd\u00e9k<\/b>. V\u017edy mysl\u00edme na u\u017eivatele: jak bude \u017e\u00edt s na\u0161imi chybami, jak n\u00e1m je m\u016f\u017ee ozn\u00e1mit. Nezapom\u00ednejme, \u017ee u\u017eivatel\u00e9 nejsou zdrojem \u010dist\u00e9 zp\u011btn\u00e9 vazby pro u\u010den\u00ed model\u016f, a je nutn\u00e9 ji filtrovat pomoc\u00ed pomocn\u00fdch ML syst\u00e9m\u016f. Pokud nen\u00ed mo\u017en\u00e9 z\u00edskat sign\u00e1l od u\u017eivatele, hled\u00e1me alternativn\u00ed zdroje zp\u011btn\u00e9 vazby, nap\u0159\u00edklad souvisej\u00edc\u00ed syst\u00e9my. \n<\/li>\n<li><b>Dal\u0161\u00ed \u0161kolen\u00ed<\/b>. Hlavn\u00ed v\u011bc\u00ed je zde kontinuita, proto se op\u00edr\u00e1me o sou\u010dasn\u00fd produk\u010dn\u00ed model. Nov\u00e9 modely vyv\u00edj\u00edme tak, aby se p\u0159\u00edli\u0161 neli\u0161ily od p\u0159edchoz\u00edho modelu d\u00edky harmonick\u00e9 regularizaci a podobn\u00fdm trik\u016fm.<\/li>\n<li><b>Nasazen\u00ed<\/b>. Autodeploy na z\u00e1klad\u011b metrik v\u00fdrazn\u011b zkracuje \u010das pot\u0159ebn\u00fd na nasazen\u00ed model\u016f. Monitoring statistik a rozd\u011blen\u00ed rozhodov\u00e1n\u00ed, po\u010det fale\u0161n\u00fdch pozitiv od u\u017eivatel\u016f je nezbytn\u00fd pro v\u00e1\u0161 klidn\u00fd sp\u00e1nek a produktivn\u00ed v\u00edkendy.\n<\/li>\n<\/ul>\n<p>\nDouf\u00e1m, \u017ee to, co jste si p\u0159e\u010detli, v\u00e1m pom\u016f\u017ee rychleji zlep\u0161ovat va\u0161e ML syst\u00e9my, urychlit jejich uveden\u00ed na trh a u\u010dinit je spolehliv\u011bj\u0161\u00edmi, \u010d\u00edm\u017e sn\u00ed\u017e\u00edte stres z pr\u00e1ce.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/mailru\/blog\/476714\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u043c\u043e\u0438\u0445 \u0432\u044b\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u0439 \u043d\u0430 Highload++ \u0438 DataFest Minsk 2019 \u0433. \u0414\u043b\u044f \u043c\u043d\u043e\u0433\u0438\u0445 \u0441\u0435\u0433\u043e\u0434\u043d\u044f \u043f\u043e\u0447\u0442\u0430 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043d\u0435\u043e\u0442\u044a\u0435\u043c\u043b\u0435\u043c\u043e\u0439 \u0447\u0430\u0441\u0442\u044c\u044e \u0436\u0438\u0437\u043d\u0438 \u0432 \u0441\u0435\u0442\u0438. \u0421 \u0435\u0435 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043c\u044b \u0432\u0435\u0434\u0435\u043c \u0431\u0438\u0437\u043d\u0435\u0441-\u043f\u0435\u0440\u0435\u043f\u0438\u0441\u043a\u0443, \u0445\u0440\u0430\u043d\u0438\u043c \u0432\u0441\u0435\u0432\u043e\u0437\u043c\u043e\u0436\u043d\u0443\u044e \u0432\u0430\u0436\u043d\u0443\u044e \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044e, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u0443\u044e \u0441 \u0444\u0438\u043d\u0430\u043d\u0441\u0430\u043c\u0438, \u0431\u0440\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u043e\u0442\u0435\u043b\u0435\u0439, \u043e\u0444\u043e\u0440\u043c\u043b\u0435\u043d\u0438\u0435\u043c \u0437\u0430\u043a\u0430\u0437\u043e\u0432 \u0438 \u043c\u043d\u043e\u0433\u0438\u043c \u0434\u0440\u0443\u0433\u0438\u043c. \u0412 \u0441\u0435\u0440\u0435\u0434\u0438\u043d\u0435 2018 \u0433\u043e\u0434\u0430 \u043c\u044b \u0441\u0444\u043e\u0440\u043c\u0443\u043b\u0438\u0440\u043e\u0432\u0430\u043b\u0438 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u0443\u044e \u0441\u0442\u0440\u0430\u0442\u0435\u0433\u0438\u044e \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u044f \u043f\u043e\u0447\u0442\u044b. \u041a\u0430\u043a\u043e\u0439 \u0436\u0435 \u0434\u043e\u043b\u0436\u043d\u0430 \u0431\u044b\u0442\u044c [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-53143","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u043c\u043e\u0438\u0445 \u0432\u044b\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u0439 \u043d\u0430.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u042d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f \u0432 \u041f\u043e\u0447\u0442\u0435 Mail.ru | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u043c\u043e\u0438\u0445 \u0432\u044b\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u0439 \u043d\u0430.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-11-23T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:01:00+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Vyu\u017eit\u00ed strojov\u00e9ho u\u010den\u00ed v Po\u0161ty Mail.ru | ProHoster","description":"Na motivy m\u00fdch vystoupen\u00ed na.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u042d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044f \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f \u0432 \u041f\u043e\u0447\u0442\u0435 Mail.ru | ProHoster","og:description":"\u041f\u043e \u043c\u043e\u0442\u0438\u0432\u0430\u043c \u043c\u043e\u0438\u0445 \u0432\u044b\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u0439 \u043d\u0430.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/ekspluatatsiya-mashinnogo-obucheniya-v-pochte-mail-ru","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-11-23T21:00:00+00:00","article:modified_time":"2020-02-18T11:01:00+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"53143","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 06:14:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:30:25","updated":"2026-01-24 06:14:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/53143","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=53143"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/53143\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=53143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=53143"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=53143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}