Apo basa rako nderekugadzira chimwe chinhu chakanaka, haufaniri kutaura zvakawanda pamusoro pazvo, nokuti chigumisiro chiri pamberi pemeso evanhu vose. Asi kana iwe ukadzima zvinyorwa kubva kumafenzi, hapana anozoona basa rako chero bedzi mafenzi achitaridzika zvakanaka kana kusvika wadzima chimwe chinhu chakaipa.
Chero sevhisi yaunogona kusiya mhinduro, ongororo, kutumira meseji kana kurodha mapikicha nekukurumidza kana gare gare inotarisana nedambudziko re spam, hutsotsi uye kunyadzisa. Izvi hazvigoni kudziviswa, asi zvinofanira kugadziriswa.
Zita rangu ndiMikhail, ndinoshanda paboka reAntispam, rinodzivirira vashandisi veYandex mabasa kubva kumatambudziko akadaro. Basa redu hariwanzoonekwa (uye chinhu chakanaka!), Saka nhasi ndichakuudza zvakawanda pamusoro payo. Iwe unozodzidza kana kuenzanisa kusingabatsiri uye nei chokwadi chisiri icho chega chiratidzo chekushanda kwayo. Tichazotaurawo nezvekutuka tichishandisa muenzaniso wekatsi nembwa uye nei dzimwe nguva zvichibatsira "kufunga seanopika."
Masevhisi akawanda ari kuoneka muYandex uko vashandisi vanoburitsa zvirimo. Unogona kubvunza mubvunzo kana kunyora mhinduro muYandex.Q, kurukura nhau dzeyadhi muYandex.District, govana mamiriro emumigwagwa muhurukuro paYandex.Maps. Asi kana vateereri vesevhisi vachikura, inova inokwezva kune scammers uye spammers. Vanouya vozadza zvirevo: vanopa mari iri nyore, shambadza kurapwa kwezvishamiso uye kuvimbisa mabhenefiti emagariro. Nekuda kwevaspammers, vamwe vashandisi vanorasikirwa nemari, nepo vamwe vanorasikirwa nechishuwo chekupedza nguva pabasa rakashata rakazara ne spam.
Uye iri handiro chete dambudziko. Isu tinoedza kwete chete kuchengetedza vashandisi kubva kune scammers, asiwo kugadzira mhepo yakagadzikana yekutaurirana. Kana vanhu vakatarisana nekutuka uye kutuka mumashoko, vanogona kuenda uye havazodzoki. Izvi zvinoreva kuti iwe unofanirwawo kugona kubata nazvo.
Webhu yakachena
Sezvinowanzoitika kwatiri, zvekutanga zvakazvarwa muKutsvaga, muchikamu chinorwisa spam mumibairo yekutsvaga. Makore angangoita gumi apfuura, basa rekusefa zvevakuru zvekutsvaga kwemhuri uye mibvunzo isingade mhinduro kubva muchikamu che18+ chakaonekwa ipapo. Aya ndiwo maitiro ekutanga akataipa maduramazwi ezvinonyadzisira uye kutuka akaonekwa, akazadzikiswa nevanoongorora. Basa guru raive rekuisa zvikumbiro kune izvo zvinotenderwa kuratidza zvevakuru uye pazvisiri. Nokuda kwebasa iri, markup yakaunganidzwa, heuristics yakavakwa, uye mienzaniso yakadzidziswa. Aya ndiwo matangiro akaita mafambisirwo ekutanga ekusefa zvisingadiwe.
Nokufamba kwenguva, UGC (yakagadzirwa nemushandisi) yakatanga kuoneka muYandex - mameseji akanyorwa nevashandisi ivo pachavo, uye Yandex inongoburitsa. Nezvikonzero zvakatsanangurwa pamusoro, mameseji mazhinji haagone kuburitswa pasina kutarisa - moderation yaidiwa. Vakabva vafunga kugadzira sevhisi yaizopa dziviriro kubva kune spam nevanorwisa zvese zveYandex UGC zvigadzirwa uye kushandisa mabudiriro kusefa zvisingadiwe zvirimo muKutsvaga. Basa racho rainzi "Clean Web".
Mabasa matsva uye rubatsiro kubva kune vanosundidzira
Pakutanga, otomatiki akareruka chete aitiitira: masevhisi akatitumira zvinyorwa, uye isu takamhanyisa maduramazwi anonyadzisa, maduramazwi ezvinonyadzisira uye matauriro akajairika paari - vaongorori vakagadzira zvese nemawoko. Asi nekufamba kwenguva, basa racho rakashandiswa mukuwedzera kwehuwandu hwezvigadzirwa zveYandex, uye taifanira kudzidza kushanda nezvinetso zvitsva.
Kazhinji, pachinzvimbo chekuongorora, vashandisi vanoburitsa mabhii asina zvazvinoreva, vachiedza kuwedzera zvavanoita, dzimwe nguva vanoshambadzira kambani yavo muongororo yekambani yemukwikwidzi, uye dzimwe nguva vanongovhiringa masangano uye vonyora muongororo yechitoro chemhuka: " Hove dzakanyatsobikwa!β Zvichida rimwe zuva hungwaru hwekugadzira huchadzidza kunyatso nzwisisa zvinorehwa nechero chinyorwa, asi ikozvino otomatiki dzimwe nguva inokunda zvakanyanya kupfuura vanhu.
Zvakava pachena kuti isu taisakwanisa kuita izvi pasina kutara nemaoko, uye takawedzera danho rechipiri kudunhu redu-kuritumira kuti riongororwe nemaoko nemunhu. Iwo magwaro akadhindwa ayo mugadziri asina kuona chero matambudziko akaiswa ipapo. Iwe unogona kufungidzira zviri nyore chiyero chebasa rakadaro, saka isu hatina kungovimba nevanoongorora, asi zvakare takatora mukana we "uchenjeri hweboka revanhu," ndiko kuti, takatendeukira kune tolokers kuti tibatsirwe. Ndivo vanotibatsira kuona zvakapotsa muchina, uye nekudaro tidzidzise.
Smart caching uye LSH hashing
Rimwe dambudziko ratakasangana naro pakushanda nemacomments raive spam, kana kuti chaizvo, huwandu hwayo nekumhanya kwekupararira. Apo vateereri veYandex.Region vakatanga kukura nokukurumidza, spammers vakauya ikoko. Vakadzidza kunzvenga matauriro enguva dzose nokuchinja zvishoma mashoko. Spam, hongu, yakanga ichiri kuwanikwa uye yakabviswa, asi pachiyero cheYandex, mharidzo isingagamuchirwi yakatumirwa kunyange kwemaminitsi mashanu inogona kuonekwa nemazana evanhu.
Ehe, izvi hazvina kuenderana nesu, uye takagadzira smart text caching yakavakirwa paLSH () Inoshanda seizvi: isu takajairisa zvinyorwa, takabvisa zvinongedzo kubva pazviri uye ndokuzvicheka kuita n-magiramu (kutevedzana kwemavara n). Zvadaro, hashes ye-n-grams yakaverengwa, uye LSH vector yegwaro yakavakwa kubva kwavari. Chiripo ndechekuti magwaro akafanana, kunyangwe akashandurwa zvishoma, akashandurwa kuita mavheti akafanana.
Mhinduro iyi yakaita kuti zvikwanise kushandisa zvakare mitongo yevagadziri uye toloker kune zvinyorwa zvakafanana. Munguva yekurwiswa kwespam, pakarepo meseji yekutanga payakapfuura scan uye ichipinda mucache ine mutongo we "spam", ese mameseji matsva akafanana, kunyangwe akagadziridzwa, akagamuchira mutongo wakafanana uye akadzimwa otomatiki. Gare gare, takadzidza kudzidzisa uye kudzidzisazve otomatiki maclassifiers espam, asi iyi "smart cache" yakagara nesu uye ichiri kutibatsira nguva zhinji.
Good text classifier
Pasina kuwana nguva yekuzorora kubva pakurwisa spam, takaona kuti 95% yezvinyorwa zvedu zvakagadziridzwa nemaoko: vagadziri vanongopindura kana kukanganisa, uye mazhinji ezvinyorwa zvakanaka. Isu tinotakura vachenesi avo mumakumi mapfumbamwe neshanu kubva mu95 vanopa chiyero "Zvese zvakanaka". Ini ndaifanira kuita basa risina kujairika - kugadzira classifiers yezvakanaka zvemukati, nerombo rakanaka markup yaive yaungana panguva iyi.
Wekutanga classifier aiita seizvi: isu lemmatize chinyorwa (kuderedza mazwi kune chimiro chavo chekutanga), kanda kunze zvese zvekuwedzera zvikamu zvekutaura uye kushandisa pre-yakagadzirirwa "duramazwi rezvakanaka lemmas". Kana mazwi ese ari murugwaro ari "akanaka", saka chinyorwa chose hachina chero kukanganisa. Pamasevhisi akasiyana, nzira iyi yakapa kubva pa25 kusvika 35% otomatiki yemanyorero markup. Ehe, nzira iyi haina kunaka: zviri nyore kusanganisa mazwi akati wandei uye kuwana chirevo chinogumbura, asi zvakatibvumira kukurumidza kusvika padanho rakanaka re automation uye yakatipa nguva yekudzidzisa mamwe mamodheru akaomarara.
Idzi dzinotevera vhezheni dzeakanaka mavara classifiers dzakatosanganisira mutsara modhi, miti yesarudzo, uye misanganiswa yavo. Kuti titarise hutsinye uye kutuka, semuenzaniso, tinoedza iyo BERT neural network. Zvakakosha kuti ubate zvinorehwa neshoko muchirevo chechinyorwa uye kubatana kwemazwi kubva mumitsara yakasiyana, uye BERT inoita basa rakanaka reizvi. (Nenzira, munguva pfupi yapfuura vashandi veNhau , kuti teknolojia inoshandiswa sei kune basa risingaenzaniswi - kutsvaga zvikanganiso mumisoro.) Somugumisiro, zvaive zvichibvira kugadzirisa kusvika ku90% yekuyerera, zvichienderana nebasa.
Kururama, kukwana uye kukurumidza
Kuti ugadzirise, iwe unofanirwa kunzwisisa kuti ndezvipi zvinounzwa nevamwe otomatiki classifiers, shanduko mavari, uye kana mhando yekutarisa yemanyorero iri kudzikisira. Kuti tiite izvi, tinoshandisa precision uye yeuka metrics.
Kurongeka ndicho chikamu chemitongo chaiyo pakati pemitongo yese pamusoro pezvakashata. Iyo yakakwirira yekururama, zvishoma zvinyorwa zvenhema. Kana iwe usingateereri kururamisa, saka muchirevo iwe unogona kubvisa zvose spam uye zvinonyangadza, uye pamwe chete navo hafu yemashoko akanaka. Kune rumwe rutivi, kana iwe uchivimba chete nekururama, ipapo teknolojia yakanakisisa ichava iyo isingabati chero munhu zvachose. Naizvozvo, pane zvakare chiratidzo chekuzara: mugove wezvakaonekwa zvakaipa zvemukati pakati pehuwandu hwehuwandu hwehuipi hwakaipa. Aya ma metrics maviri anoenzanisa mumwe nemumwe kunze.
Kuyera, tinoyedza rwizi rwese rwuri kuuya rwesevhisi yega yega uye tinopa zvemukati masampula kune vaongorori kuti vaongorore nyanzvi uye kuenzanisa nemhinduro dzemuchina.
Asi pane chimwe chiratidzo chinokosha.
Ndakanyora pamusoro apa kuti mharidzo isingagamuchirwi inogona kuonekwa nemazana evanhu kunyange mumaminitsi mashanu. Saka tinoverenga kuti kangani patakaratidza vanhu zvakaipa tisati tazvivanza. Izvi zvakakosha nekuti hazvina kukwana kushanda nemazvo - iwe unofanirwawo kushanda nekukurumidza. Uye patakavaka dziviriro pakupika, takanzwa zvizere.
Antimatism uchishandisa muenzaniso wekati nembwa
Kadimburiko kerimbo. Vamwe vanogona kutaura kuti zvinyadzi uye kutuka hazvina ngozi semanongedzo akaipa, uye kwete seanotsamwisa se spam. Asi isu tinoedza kuchengetedza mamiriro akasununguka ekutaurirana kwemamiriyoni evashandisi, uye vanhu havafarire kudzokera kunzvimbo dzavanotukwa. Hazvisi pasina kuti kurambidzwa kwekutuka uye kutuka kunotsanangurwa mumitemo yenharaunda dzakawanda, kusanganisira paHabrΓ©. Asi isu takasiyana.
Kutuka maduramazwi haakwanise kubata nehupfumi hwese hwemutauro weRussia. Pasinei nokuti kune midzi ina chete yemhiko, kubva kwavari iwe unogona kuita nhamba isingaverengeki yemashoko asingagoni kubatwa neipi injini dzenguva dzose. Pamusoro pazvo, unogona kunyora chikamu chezwi mukuturikira, kutsiva mavara nemasanganiswa akafanana, kurongazve mavara, kuwedzera asterisk, nezvimwewo. Dzimwe nguva, pasina mamiriro, hazvigoneke kuona kuti mushandisi aireva shoko rekutuka. Tinoremekedza mitemo yaHabr, saka ticharatidza izvi kwete nemienzaniso mhenyu, asi nekatsi nembwa.
βMutemo,β katsi yakadaro. Asi isu tinonzwisisa kuti katsi yakataura izwi rakasiyana ...
Takatanga kufunga nezve "zvisinganzwisisike kufananidza" maalgorithms eduramazwi redu uye nezve preprocessing zvine hungwaru: takapa kududzira, nzvimbo dzakanamirwa uye nyora dzemutauro pamwe chete, takatsvaga mapatani uye takanyora zvirevo zvakasiyana pazviri. Iyi nzira yakaunza mhedzisiro, asi kazhinji yakaderedza kurongeka uye haina kupa kukwana kwaidiwa.
Ipapo takasarudza "kufunga sevanopika." Takatanga kuunza ruzha mu data isu pachedu: isu takarongazve mavara, takagadzira typos, kutsiva mavara ane zviperengo zvakafanana, zvichingodaro. Mucherechedzo wekutanga weizvi wakatorwa nekushandisa dictionaries mat kune makuru makuru ezvinyorwa. Kana ukatora mutsara mumwe woumonyorora nenzira dzakawanda, unopedzisira wava nemitsara yakawanda. Nenzira iyi unogona kuwedzera sampu yekudzidzira makumi enguva. Chakanga chasara kwaive kudzidzisa padziva rinozobuda imwe yakawedzera kana kushoma modhi yakangwara yaifunga nezve mamiriro.
Kwakasira kutaura nezvesarudzo yekupedzisira. Tichiri kuyedza nzira dzedambudziko iri, asi isu tinogona kutoona kuti yakapfava yekufananidzira convolutional network yezvikamu zvakati wandei inokunda maduramazwi uye injini dzenguva dzose: zvinokwanisika kuwedzera zvese kurongeka nekurangarira.
Ehe, isu tinonzwisisa kuti pachagara paine nzira dzekunzvenga kunyangwe iyo yakanyanyisa otomatiki, kunyanya kana nyaya yacho iine njodzi: nyora nenzira yekuti muchina benzi haunzwisise. Pano, sekurwisa spam, chinangwa chedu hachisi chekubvisa mukana chaiwo wekunyora chimwe chinhu chinonyadzisa; basa redu nderekuona kuti mutambo hauna kukodzera kenduru.
Kuvhura mukana wekugovana maonero ako, kutaurirana uye kutaura hakuna kuoma. Zvakanyanya kuoma kuwana akachengeteka, akasununguka mamiriro uye neruremekedzo kubata vanhu. Uye pasina izvi hapazove nebudiriro yechero nharaunda.
Source: www.habr.com