Uma umsebenzi wakho uwukudala into enhle, akudingeki ukhulume kakhulu ngakho, ngoba umphumela uphambi kwamehlo wonke umuntu. Kodwa uma usula imibhalo ezicingweni, akekho ozobona umsebenzi wakho inqobo nje uma uthango lubukeka luhloniphekile noma uze usule okuthile okungalungile.
Noma iyiphi isevisi lapho ungashiya khona amazwana, ubuyekeze, uthumele umlayezo noma ulayishe izithombe ngokushesha noma kamuva ibhekana nenkinga yogaxekile, ukukhwabanisa namanyala. Lokhu angeke kugwenywe, kodwa kufanele kubhekwane nakho.
Igama lami nginguMikhail, ngisebenza eqenjini le-Antispam, elivikela abasebenzisi bezinsizakalo ze-Yandex ezinkingeni ezinjalo. Umsebenzi wethu awuvamile ukuqashelwa (futhi lokho kuyinto enhle!), ngakho-ke namuhla ngizokutshela okwengeziwe ngawo. Uzofunda lapho ukulinganisela kungenamsebenzi nokuthi kungani ukunemba kungesona ukuphela kwenkomba yokusebenza kwayo. Sizophinde sikhulume ngenhlamba sisebenzisa isibonelo samakati nezinja nokuthi kungani kusiza ngezinye izikhathi “ukucabanga njengomuntu ofungayo.”
Izinsizakalo eziningi ziyavela ku-Yandex lapho abasebenzisi bashicilela khona okuqukethwe kwabo. Ungabuza umbuzo noma ubhale impendulo ku-Yandex.Q, uxoxe ngezindaba zasegcekeni e-Yandex.District, wabelane ngezimo zethrafikhi ezingxoxweni ku-Yandex.Maps. Kodwa lapho izethameli zesevisi zikhula, iba yinhle kubakhwabanisi nakuma-spammers. Bafika bagcwalise ukuphawula: banikela ngemali elula, bakhangise ngemithi eyisimangaliso futhi bathembise izinzuzo zomphakathi. Ngenxa yogaxekile, abanye abasebenzisi balahlekelwa yimali, kuyilapho abanye belahlekelwa isifiso sokuchitha isikhathi enkonzweni engcolile egcwele ugaxekile.
Futhi lokhu akuyona inkinga kuphela. Asizami nje kuphela ukuvikela abasebenzisi kubakhohlisi, kodwa futhi nokudala isimo esinethezekile sokuxhumana. Uma abantu bebhekana nenhlamba nenhlamba emazwaneni, bangase bahambe bangaphinde babuye. Lokhu kusho ukuthi nawe udinga ukuthi ukwazi ukubhekana nalokhu.
Iwebhu ehlanzekile
Njengoba kuvame ukuba njalo ngathi, ukuthuthukiswa kokuqala kwazalwa Kusesho, engxenyeni elwa nogaxekile emiphumeleni yosesho. Cishe eminyakeni eyishumi edlule, umsebenzi wokuhlunga okuqukethwe kwabantu abadala oseshweni lomndeni kanye nemibuzo ebingadingi izimpendulo ezivela esigabeni sabangu-18+ uvele lapho. Yile ndlela ezavela ngayo izichazamazwi zokuqala ezithayishwe ngesandla ze-porn nenhlamba, zagcwaliswa ngabahlaziyi. Umsebenzi omkhulu bekuwukuhlukanisa izicelo zibe yilezo lapho okwamukelekayo ukukhombisa okuqukethwe kwabantu abadala nalapho kungamukelekile khona. Kulo msebenzi, i-markup yaqoqwa, ama-heuristics akhiwa, futhi amamodeli aqeqeshwa. Yile ndlela intuthuko yokuqala yokuhlunga okuqukethwe okungafunwa yavela ngayo.
Ngokuhamba kwesikhathi, i-UGC (okuqukethwe okukhiqizwa umsebenzisi) yaqala ukuvela ku-Yandex - imilayezo ebhalwe abasebenzisi ngokwabo, futhi i-Yandex ishicilela kuphela. Ngenxa yezizathu ezichazwe ngenhla, imilayezo eminingi ayikwazanga ukushicilelwa ngaphandle kokubheka - ukulinganisela bekudingeka. Babe sebenquma ukudala isevisi ezonikeza ukuvikeleka kogaxekile nabahlaseli bayo yonke imikhiqizo ye-Yandex UGC futhi basebenzise ukuthuthukiswa ukuze bahlunge okuqukethwe okungafunwa Kusesho. Isevisi yayibizwa ngokuthi “Iwebhu Ehlanzekile”.
Imisebenzi emisha nosizo oluvela kubaphushi
Ekuqaleni, i-automation elula kuphela eyasisebenzela: izinsizakalo zasithumelela imibhalo, futhi sasebenzisa izichazamazwi zamanyala, izichazamazwi ze-porn nezinkulumo ezivamile kuzo - abahlaziyi bahlanganisa yonke into ngesandla. Kodwa ngokuhamba kwesikhathi, isevisi yasetshenziswa enanini elandayo lemikhiqizo ye-Yandex, futhi kwakudingeka sifunde ukusebenza ngezinkinga ezintsha.
Imvamisa, esikhundleni sokubuyekezwa, abasebenzisi bashicilela iqoqo lezinhlamvu ezingenangqondo, bezama ukukhulisa impumelelo yabo, kwesinye isikhathi bakhangisa inkampani yabo ekubuyekezweni kwenkampani esincintisana nabo, futhi kwesinye isikhathi bavele badideke izinhlangano futhi babhale isibuyekezo ngesitolo sezilwane: “ Izinhlanzi eziphekwe kahle!” Mhlawumbe ngolunye usuku ubuhlakani bokwenziwa buzofunda ukuqonda kahle incazelo yanoma yimuphi umbhalo, kodwa manje i-automation ngezinye izikhathi ibhekana nokubi kunabantu.
Kwaba sobala ukuthi sasingeke sikwenze lokhu ngaphandle kokumaka ngesandla, futhi sengeza isigaba sesibili kumjikelezo wethu—siwuthumela ukuze sihlolwe ngumuntu. Leyo mibhalo eshicilelwe lowo ohlukanisa ngezigaba angazange azibone izinkinga yayifakwe lapho. Ungakwazi ukucabanga kalula ngesilinganiso somsebenzi onjalo, ngakho-ke asizange sithembele kuphela kubahloli, kodwa futhi sasebenzisa "ukuhlakanipha kwesixuku," okungukuthi, saphendukela ku-tolokers ukuze sithole usizo. Yibo abasisizayo ukuthi sibone ukuthi umshini uphuthelwe ini, futhi ngalokho sikufundise.
I-Smart caching kanye ne-LSH hashing
Enye inkinga esihlangabezane nayo lapho sisebenza ngamazwana kwakungugaxekile, noma ngokunembe kakhulu, ivolumu yabo nesivinini sokusabalala. Lapho izethameli ze-Yandex.Region ziqala ukukhula ngokushesha, abakwa-spammers beza lapho. Bafunda ukweqa izinkulumo ezivamile ngokushintsha kancane umbhalo. Ugaxekile, vele, usatholakele futhi ususiwe, kodwa esikalini se-Yandex, umlayezo ongamukeleki othunyelwe ngisho nemizuzu emi-5 ungabonwa ngamakhulu abantu.
Vele, lokhu akuzange kusifanele, futhi senze ukugcinwa kwesikhashana kombhalo okuhlakaniphile ngokusekelwe ku-LSH (). Isebenza kanje: senze umbhalo ube ujwayelekile, sisuse izixhumanisi kuwo futhi sisike sibe ama-n-grams (ukulandelana kwezinhlamvu n). Okulandelayo, ama-hashes we-n-grams abalwa, futhi i-vector ye-LSH yedokhumenti yakhiwe kusuka kubo. Iphuzu liwukuthi imibhalo efanayo, noma ngabe ishintshwe kancane, yaguqulwa yaba ama-vector afanayo.
Lesi sixazululo senze kwaba nokwenzeka ukuphinda kusetshenziswe izinqumo zabahlukanisi bezigaba kanye nama-toloker emibhalweni efanayo. Ngesikhathi sokuhlaselwa kogaxekile, ngokushesha nje lapho umlayezo wokuqala udlula ukuskena futhi ufaka inqolobane ngesinqumo "sogaxekile", yonke imilayezo emisha efanayo, ngisho naleyo eguquliwe, ithole isigwebo esifanayo futhi yasuswa ngokuzenzakalelayo. Kamuva, safunda ukuqeqesha nokuqeqesha kabusha ngokuzenzakalelayo abahlaluli bezigaba zogaxekile, kodwa le “nqolobane ehlakaniphile” yahlala nathi futhi isasisiza njalo.
Isihlukanisi sombhalo esihle
Ngaphandle kokuba nesikhathi sokuphumula ekulweni nogaxekile, sibonile ukuthi u-95% wokuqukethwe kwethu kunganyelwa mathupha: abahlukanisa izigaba basabela kuphela lapho bephulwa khona, futhi imibhalo eminingi mihle. Silayisha abahlanzi okuthi emacaleni angama-95 kwayi-100 banikeze isilinganiso "Konke KULUNGILE". Kwadingeka ngenze umsebenzi ongajwayelekile - ukwenza abahlukanisi bokuqukethwe okuhle, ngenhlanhla umaki okwanele wawunqwabelene ngalesi sikhathi.
Isihlukanisi sokuqala sasibukeka kanje: sinciphisa umbhalo (sehlisa amagama abe yisimo sawo sokuqala), silahle zonke izingxenye zenkulumo ezisizayo futhi sisebenzise “isichazamazwi se-lemmas enhle” esilungiselelwe kusengaphambili. Uma wonke amagama embhalweni “elungile”, umbhalo wonke awuqukethe noma yikuphi ukwephulwa. Ezinkonzweni ezihlukene, le ndlela yanikeza ngokushesha kusuka ku-25 kuya ku-35% okuzenzakalelayo kwemakhaphu okwenziwa ngesandla. Yiqiniso, le ndlela ayilungile: kulula ukuhlanganisa amagama amaningana angenacala futhi uthole isitatimende esicasula kakhulu, kodwa wasivumela ukuba sifinyelele ngokushesha izinga elihle lokuzenzekelayo futhi kwasinika isikhathi sokuqeqesha amamodeli ayinkimbinkimbi.
Izinguqulo ezilandelayo zezigaba ezinhle zombhalo sezivele zihlanganisa amamodeli alayini, izihlahla zezinqumo, nezinhlanganisela zazo. Ukumaka ukudelela nokuthuka, isibonelo, sizama inethiwekhi ye-BERT yemizwa. Kubalulekile ukuqonda incazelo yegama elikumongo kanye nokuxhumana phakathi kwamagama avela emishweni eyahlukene, futhi i-BERT iwenza kahle umsebenzi walokhu. (Ngalokho, ozakwethu bakamuva abavela ku-News , ukuthi ubuchwepheshe busetshenziselwa kanjani umsebenzi ongewona ojwayelekile - amaphutha okuthola kuzihloko.) Ngenxa yalokho, kwakungenzeka ukuzenzekelayo kuze kufike ku-90% wokugeleza, kuye ngesevisi.
Ukunemba, ukuphelela kanye nesivinini
Ukuze uthuthuke, udinga ukuqonda ukuthi yiziphi izinzuzo ezilethwa abahlukanisi bezigaba abazenzakalelayo abathile, izinguquko kuzo, nokuthi izinga lokuhlola mathupha liyehlisa yini isithunzi. Ukuze senze lokhu, sisebenzisa ukunemba nokukhumbula amamethrikhi.
Ukunemba ingxenye yezinqumo ezifanele phakathi kwazo zonke izinqumo mayelana nokuqukethwe okubi. Ukuphakama kokunemba, kuyancipha imibono engamanga. Uma unganaki ukunemba, khona-ke ngombono ungasusa wonke ugaxekile namanyala, kanye nawo uhhafu wemilayezo emihle. Ngakolunye uhlangothi, uma uthembele kuphela ekunembeni, khona-ke ubuchwepheshe obuhle kakhulu buyoba yibo obungabambi muntu nhlobo. Ngakho-ke, kukhona futhi inkomba yokuphelela: isabelo sokuqukethwe okubi okuhlonziwe phakathi kwengqikithi yevolumu yokuqukethwe okubi. Lawa mamethrikhi amabili ayalingana.
Ukuze silinganise, senza isampula konke ukusakaza okungenayo kwesevisi ngayinye futhi sinikeze amasampula okuqukethwe kubahloli ukuze bahlolwe ngochwepheshe futhi baqhathanise nezixazululo zemishini.
Kodwa kunenye inkomba ebalulekile.
Ngibhale ngenhla ukuthi umlayezo ongamukelekile ungabonwa ngamakhulu abantu ngisho nangemizuzu emi-5. Ngakho sibala ukuthi kukangakhi sibonisa abantu okuqukethwe okubi ngaphambi kokuthi sikufihle. Lokhu kubalulekile ngoba akwanele ukusebenza kahle - udinga nokusebenza ngokushesha. Futhi lapho sakha ukuzivikela ngokumelene nenhlamba, sakuzwa ngokugcwele.
I-Antimatism isebenzisa isibonelo samakati nezinja
Ukwehla kwezwi elincane. Abanye bangase bathi amanyala nokuhlambalaza akuyona ingozi njengezixhumanisi ezinonya, futhi akucasulanga njengogaxekile. Kodwa silwela ukugcina izimo ezinethezekile zokuxhumana ezigidini zabasebenzisi, futhi abantu abathandi ukubuyela ezindaweni lapho bethukwa khona. Akuyona ize ukuthi ukuvinjelwa kwenhlamba nokuhlambalaza kuchazwe emithethweni yemiphakathi eminingi, kuhlanganise noHabré. Kodwa siyaphuma.
Izichazamazwi ezinhlamba azikwazi ukubhekana nakho konke ukunotha kolimi lwesiRashiya. Naphezu kweqiniso lokuthi kunezimpande ezine eziyinhloko zenhlamba, kuzo ungenza inombolo engenakubalwa yamagama angenakubanjwa yinoma yiziphi izinjini ezivamile. Ngaphezu kwalokho, ungabhala ingxenye yegama ekuguquleleni amagama, ushintshe izinhlamvu ngezinhlanganisela ezifanayo, uhlele kabusha izinhlamvu, wengeze izinkanyezi, njll. Ngezinye izikhathi, ngaphandle komongo, empeleni akunakwenzeka ukunquma ukuthi umsebenzisi ubeqonde igama lenhlamba. Siyayihlonipha imithetho kaHabr, ngakho-ke lokhu ngeke sikubonise ngezibonelo ezibukhoma, kodwa ngamakati nezinja.
“Umthetho,” kusho ikati. Kodwa siyaqonda ukuthi ikati lisho elinye igama...
Saqala ukucabanga ngama-algorithms "okufanisa okungaqondakali" esichazamazwi sethu kanye nokucubungula kusengaphambili okuhlakaniphile: sinikeze ukuguqulwa kwamaletha, sinamathisele izikhala nezimpawu zokuloba ndawonye, sabheka amaphethini futhi sabhala izinkulumo ezivamile ezihlukene kuzo. Le ndlela ilethe imiphumela, kodwa ngokuvamile inciphisa ukunemba futhi ayizange inikeze ukuphelela okufunayo.
Sabe sesinquma “ukucabanga njengabantu abafungayo.” Saqala ukwethula umsindo kudatha ngokwethu: sahlela kabusha izinhlamvu, sakha ama-typos, sashintsha izinhlamvu zapelwa ngendlela efanayo, njalo njalo. Umaki wokuqala walokhu uthathwe ngokusebenzisa izichazamazwi ze-mat kuqoqo elikhulu lemibhalo. Uma uthatha umusho owodwa uwusonte ngezindlela eziningi, ugcina unemisho eminingi. Ngale ndlela ungakwazi ukwandisa isampula yokuqeqesha izikhathi eziyishumi. Okwakusele kwakuwukuqeqesha echibini eliwumphumela imodeli ethile ehlakaniphile noma engaphansi eyayicabangela umongo.
Kusesekuseni kakhulu ukuthi singakhuluma ngesinqumo sokugcina. Sisazama izindlela zale nkinga, kodwa sesiyabona ukuthi inethiwekhi elula ye-convolutional engokomfanekiso yezendlalelo ezimbalwa idlula kakhulu izichazamazwi nezinjini ezivamile: kungenzeka ukwandisa kokubili ukunemba nokukhumbula.
Yiqiniso, siyaqonda ukuthi zizohlala zikhona izindlela zokudlula ngisho ne-automation ethuthuke kakhulu, ikakhulukazi uma indaba iyingozi kakhulu: bhala ngendlela yokuthi umshini oyisiphukuphuku ngeke uqonde. Lapha, njengasekulweni nogaxekile, inhloso yethu akukona ukuqeda amandla okubhala into engcolile; umsebenzi wethu ukuqinisekisa ukuthi umdlalo awulifanele ikhandlela.
Ukuvula ithuba lokwabelana ngombono wakho, ukuxhumana nokuphawula akunzima. Kunzima kakhulu ukufeza izimo eziphephile, ezinethezekile kanye nokuphathwa ngenhlonipho kwabantu. Futhi ngaphandle kwalokhu ngeke kube khona ukuthuthuka kwanoma yimuphi umphakathi.
Source: www.habr.com