{"id":72898,"date":"2020-03-05T20:42:08","date_gmt":"2020-03-05T17:42:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame"},"modified":"2020-03-05T20:42:08","modified_gmt":"2020-03-05T17:42:08","slug":"kak-otkryt-kommentarii-i-ne-potonut-v-spame","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","title":{"rendered":"Si t\u2019i hapni komentet pa u mbytur n\u00eb spam","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Si
\n
\nKur puna jote \u00ebsht\u00eb krijimi i di\u00e7kaje t\u00eb bukur, nuk ka shum\u00eb p\u00ebr t\u00eb th\u00ebn\u00eb rreth saj, sepse rezultati \u00ebsht\u00eb para gjith\u00eb atyre q\u00eb e shohin. Por, n\u00ebse ti fshih shkrimet nga mur\u00ebt, puna jote nuk v\u00ebrehet derisa mur\u00ebt t\u00eb duken t\u00eb past\u00ebr ose derisa t\u00eb fshish di\u00e7ka q\u00eb nuk duhet.<\/p>\n

\u00c7do sh\u00ebrbim ku mund t\u00eb l\u00ebsh nj\u00eb koment, nj\u00eb rishikim, t\u00eb d\u00ebrgosh mesazhe ose t\u00eb ngarkosh imazhe, p\u00ebrballohet me problemin e spam-it, mashtrimeve dhe fjal\u00ebve ofendues. Kjo nuk mund t\u00eb shmanget, por duhet t\u00eb luftohet. <\/p>\n

M\u00eb quajn\u00eb Mikhail, punoj n\u00eb ekipin e Antispoamit, i cili mbron p\u00ebrdoruesit e sh\u00ebrbimeve t\u00eb Yandex nga k\u00ebto probleme. Puna jon\u00eb shpesh nuk vihet re (dhe \u00ebsht\u00eb mir\u00eb!), prandaj sot do t\u00eb flas m\u00eb shum\u00eb p\u00ebr t\u00eb. Do t\u00eb m\u00ebsoni n\u00eb cilat raste moderimi \u00ebsht\u00eb i pamjaftuesh\u00ebm dhe pse sakt\u00ebsia nuk \u00ebsht\u00eb treguesi i vet\u00ebm i efektivitetit t\u00eb tij. Po ashtu, do t\u00eb diskutojm\u00eb p\u00ebr fjalorin e keq duke p\u00ebrdorur shembuj nga macet dhe qent\u00eb dhe p\u00ebrse ndonj\u00ebher\u00eb \u00ebsht\u00eb e dobishme t\u00eb \"mendonim si nj\u00eb ofendues\".
\n<\/a><\/noindex>
\nN\u00eb Yandex po shfaqen gjithnj\u00eb e m\u00eb shum\u00eb sh\u00ebrbime ku p\u00ebrdoruesit publikojn\u00eb p\u00ebrmbajtjen e tyre. Mund t\u00eb b\u00ebsh nj\u00eb pyetje ose t\u00eb shkruash nj\u00eb p\u00ebrgjigje n\u00eb Yandex.Q, t\u00eb diskutosh p\u00ebr lajmet e lagjes n\u00eb Yandex.Rajoni, t\u00eb ndash gjendjen e trafikut n\u00eb bisedat n\u00eb Yandex.Hartat. Por, kur audienca e sh\u00ebrbimit rritet, ai b\u00ebhet t\u00ebrheq\u00ebs p\u00ebr mashtruesit dhe spamator\u00ebt. Ata vijn\u00eb dhe mbushin komentet: ofrojn\u00eb fitime t\u00eb lehta, reklamojn\u00eb mjete t\u00eb mrekullueshme dhe premtojn\u00eb ndihma sociale. P\u00ebr shkak t\u00eb spamator\u00ebve, disa p\u00ebrdorues humbasin para, nd\u00ebrsa t\u00eb tjer\u00eb humbasin d\u00ebshir\u00ebn p\u00ebr t\u00eb kaluar koh\u00eb n\u00eb nj\u00eb sh\u00ebrbim t\u00eb papast\u00ebr, t\u00eb mbushur me spam.<\/p>\n

Dhe kjo nuk \u00ebsht\u00eb problemi i vet\u00ebm. Ne synojm\u00eb jo vet\u00ebm t\u00eb mbrojm\u00eb p\u00ebrdoruesit nga mashtruesit, por edhe t\u00eb krijojm\u00eb nj\u00eb atmosfer\u00eb komode p\u00ebr bised\u00eb. N\u00ebse njer\u00ebzit n\u00eb komente p\u00ebrballen me fjal\u00eb ofenduese dhe sharje, ata kan\u00eb nj\u00eb probabilitet t\u00eb lart\u00eb t\u00eb largohen dhe t\u00eb mos kthehen m\u00eb. K\u00ebshtu q\u00eb kjo duhet gjithashtu t\u00eb luftohet. <\/p>\n

Web i Past\u00ebr<\/h3>\n

\nSi ndodh shpesh, zhvillimet e para lind\u00ebn n\u00eb K\u00ebrkim, n\u00eb at\u00eb pjes\u00eb q\u00eb lufton me spam n\u00eb rezultatet e k\u00ebrkimit. Dhjet\u00eb vjet m\u00eb par\u00eb, aty u shfaq nj\u00eb detyr\u00eb p\u00ebr filtrimin e p\u00ebrmbajtjeve t\u00eb rritur p\u00ebr k\u00ebrkime familjare dhe p\u00ebr pyetje q\u00eb nuk parashtrojn\u00eb p\u00ebrgjigje nga kategoria 18+. K\u00ebshtu u krijuan fjalor\u00ebt e par\u00eb, t\u00eb shkruar me dor\u00eb, p\u00ebr pornografin\u00eb dhe fjalorin e fjal\u00ebve t\u00eb neveritshme, t\u00eb cil\u00ebt i mbushnin analist\u00ebt. Detyra kryesore ishte klasifikimi i pyetjeve n\u00eb ato q\u00eb lejohej t\u00eb shfaqisnin p\u00ebrmbajtje t\u00eb rritur dhe ato q\u00eb nuk lejohej. P\u00ebr k\u00ebt\u00eb detyr\u00eb u grumbulluan sh\u00ebnime, u nd\u00ebrtuan heuristika, u stjuan modele. K\u00ebshtu u krijuan zhvillimet e para p\u00ebr filtrimin e p\u00ebrmbajtjeve t\u00eb pad\u00ebshiruara.<\/p>\n

Me kalimin e koh\u00ebs, n\u00eb Yandex filloi t\u00eb shfaqej UGC (p\u00ebrmbajtje e krijuar nga p\u00ebrdoruesit) \u2014 mesazhe q\u00eb i shkruajn\u00eb vet\u00eb p\u00ebrdoruesit, dhe Yandex thjesht i publikonte ato. P\u00ebr arsyet e p\u00ebrmendura m\u00eb sip\u00ebr, shum\u00eb mesazhe nuk mund t\u00eb publikohej pa nj\u00eb shqyrtim \u2014 k\u00ebrkohej moderimi. At\u00ebher\u00eb u vendos t\u00eb krijohej nj\u00eb sh\u00ebrbim q\u00eb do t\u00eb ofronte mbrojtje nga spam-i dhe keqb\u00ebr\u00ebsit p\u00ebr t\u00eb gjitha produktet UGC t\u00eb Yandex-it dhe p\u00ebrdorte zhvillimet p\u00ebr filtrimin e p\u00ebrmbajtjeve t\u00eb pad\u00ebshiruara n\u00eb K\u00ebrkim. Sh\u00ebrbimi u quajt \"Webi i Past\u00ebr\". <\/p>\n

Detyra t\u00eb reja dhe ndihma e toloker\u00ebve<\/h3>\n

\nFillimisht, ne kishim vet\u00ebm automatizim t\u00eb thjesht\u00eb: sh\u00ebrbimet na d\u00ebrgonin ne tekstet, dhe ne i kalonim ato p\u00ebrmes fjalor\u00ebve t\u00eb fjal\u00ebve t\u00eb neveritshme, fjalor\u00ebve t\u00eb pornografis\u00eb dhe shprehjeve t\u00eb rregullta \u2014 analist\u00ebt e p\u00ebrgatitnin gjith\u00e7ka manualisht. Por sh\u00ebrbimi me kalimin e koh\u00ebs u p\u00ebrdor n\u00eb num\u00ebr gjithnj\u00eb e m\u00eb t\u00eb madh produktesh Yandex, dhe na duhej t\u00eb m\u00ebsojm\u00eb t\u00eb punojm\u00eb me probleme t\u00eb reja. <\/p>\n

Shpesh, n\u00eb vend q\u00eb t\u00eb japin nj\u00eb vler\u00ebsim, p\u00ebrdoruesit publikojn\u00eb nj\u00eb grup pa lidhje shkronjash, duke u p\u00ebrpjekur t\u00eb fitojn\u00eb nj\u00eb arritje p\u00ebr veten e tyre, her\u00eb pas here reklamojn\u00eb kompanin\u00eb e tyre n\u00eb vler\u00ebsimet p\u00ebr kompanit\u00eb konkurrente, dhe ndonj\u00ebher\u00eb thjesht ngat\u00ebrrojn\u00eb organizatat dhe n\u00eb nj\u00eb vler\u00ebsim p\u00ebr nj\u00eb dyqan p\u00ebr kafsh\u00eb shkrijn\u00eb: \"Peshku i gatuar shk\u00eblqye!\". Ndoshta nj\u00eb dit\u00eb inteligjenca artificiale do t\u00eb m\u00ebsoj\u00eb t\u00eb kuptoj\u00eb plot\u00ebsisht sensin e \u00e7do teksti, por tani automatizimi shpesh nuk funksionon m\u00eb mir\u00eb se njeriu. <\/p>\n

U b\u00eb e qart\u00eb se pa nj\u00eb markup manual k\u00ebtu nuk do t\u00eb kishim zgjidhje, k\u00ebshtu q\u00eb i shtuam konturit ton\u00eb nj\u00eb shkall\u00eb t\u00eb dyt\u00eb - d\u00ebrgimin p\u00ebr kontroll manual nga nj\u00eb njeri. Aty shkonin ato tekste t\u00eb publikuara p\u00ebr t\u00eb cilat klasifikatori nuk kishte par\u00eb probleme. Masat e nj\u00eb detyre t\u00eb till\u00eb mund t\u2019i imagjinoni leht\u00ebsisht, prandaj ne nuk iu besuam vet\u00ebm assessor\u00ebve, por gjithashtu p\u00ebrfituam nga \"men\u00e7uria e turm\u00ebs\", dmth. k\u00ebrkuam ndihm\u00ebn e toloker\u00ebve. Pik\u00ebrisht ata na ndihmojn\u00eb t\u00eb zbulojm\u00eb at\u00eb q\u00eb mahniti makina dhe k\u00ebshtu e m\u00ebsojn\u00eb at\u00eb. <\/p>\n

Keshimi inteligjent dhe hashing LSH<\/h3>\n

\nNj\u00eb tjet\u00ebr problem me t\u00eb cilin u p\u00ebrball\u00ebm gjat\u00eb pun\u00ebs me komentet ishte spam, p\u00ebr t\u00eb qen\u00eb m\u00eb sakt\u00eb, volumi dhe shpejt\u00ebsia e p\u00ebrhapjes s\u00eb tij. Kur audienca e Yandex.Rajon filloi t\u00eb rritet shpejt, atje erdh\u00ebn spamer\u00ebt. Ata m\u00ebsuan t\u00eb kalonin rregullat, duke ndryshuar pak tekstin. Sigurisht, spam-in e gjenim dhe e fshim\u00eb, por n\u00eb mas\u00ebn e Yandex-it, nj\u00eb mesazh i papranuesh\u00ebm i publikuar edhe p\u00ebr 5 minuta mund t\u00eb shihte qindra njer\u00ebz. <\/p>\n

\"Si
\n
\nKjo sigurisht q\u00eb nuk na k\u00ebnaqte, prandaj ne krijuam nj\u00eb keshim inteligjent t\u00eb teksteve n\u00eb baz\u00eb t\u00eb LSH (hashimi sensitivity-lokalisht<\/a><\/noindex>). Kjo funksionon si\u00e7 vijon: ne normalizuam tekstin, e hiqnim lidhjet dhe e ndanim at\u00eb n\u00eb n-grama (s\u00ebkuence prej n-character). M\u00eb pas, llogaritnim hash-\u00ebt e n-gramave, dhe sipas tyre nd\u00ebrtonim vektorin LSH t\u00eb dokumentit. Kuptimi \u00ebsht\u00eb se tekstet e ngjashme, edhe n\u00ebse ato jan\u00eb ndryshuar pak, shnd\u00ebrrohen n\u00eb vektora t\u00eb ngjash\u00ebm. <\/p>\n

Ky zgjidhje na lejojti t\u00eb ri-shfryt\u00ebzojm\u00eb verdiktet e klasifikator\u00ebve dhe toloker\u00ebve p\u00ebr tekste t\u00eb ngjashme. Gjat\u00eb nj\u00eb sulmi me spam, sapo mesazhi i par\u00eb kalonte kontrollin dhe hynte n\u00eb kesh me verdiktin \"spam\", t\u00eb gjitha mesazhet e reja t\u00eb ngjashme, edhe t\u00eb modifikuara, merrnin t\u00eb nj\u00ebjtin verdikt dhe fshiheshin automatikisht. M\u00eb von\u00eb, ne m\u00ebsuam t\u00eb m\u00ebsojm\u00eb dhe t\u00eb ri-m\u00ebsojm\u00eb automatikisht klasifikator\u00ebt e spam, por ky \"kesh inteligjent\" na ka mbetur dhe akoma na ndihmon shpesh.<\/p>\n

Klasifikatori i teksteve t\u00eb mira<\/h3>\n

\nPas arriti t\u00eb pushojm\u00eb nga lufta me spamin, ne kuptuam se 95% e p\u00ebrmbajtjes ton\u00eb moderohen manualisht: klasifikuesit reagojn\u00eb vet\u00ebm ndaj shkeljeve, ndersa shumica e teksteve jan\u00eb t\u00eb mira. Ne ngarkojm\u00eb toloker\u00ebt, t\u00eb cil\u00ebt n\u00eb 95 raste nga 100 japin vler\u00ebsim \"Gjith\u00e7ka n\u00eb rregull\". Duhej t\u00eb merreshim me nj\u00eb pun\u00eb t\u00eb pazakont\u00eb - t\u00eb krijonim klasifikues p\u00ebr p\u00ebrmbajtjen e mir\u00eb, fatmir\u00ebsisht kemi akumuluar mjaft sh\u00ebnime gjat\u00eb k\u00ebtij koh\u00eb.<\/p>\n

Klasifikuesi i par\u00eb dukej k\u00ebshtu: lematizojm\u00eb tekstin (i kthejm\u00eb fjal\u00ebt n\u00eb form\u00ebn fillestare), heqim t\u00eb gjitha pjes\u00ebt e fjal\u00ebve sh\u00ebrbyes dhe aplikojm\u00eb nj\u00eb \"fjalor t\u00eb fjal\u00ebve t\u00eb mira\" t\u00eb p\u00ebrgatitur m\u00eb par\u00eb. N\u00ebse n\u00eb tekst t\u00eb gjitha fjal\u00ebt jan\u00eb \"t\u00eb mira\", at\u00ebher\u00eb edhe teksti n\u00eb t\u00ebr\u00ebsi nuk p\u00ebrmban shkelje. N\u00eb sh\u00ebrbime t\u00eb ndryshme, ky qasje menj\u00ebher\u00eb dha nga 25 deri n\u00eb 35% automatizim t\u00eb sh\u00ebnimeve manuale. Natyrisht, ky qasje nuk \u00ebsht\u00eb i p\u00ebrsosur: nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb kombinosh disa fjal\u00eb t\u00eb pafajshme dhe t\u00eb marr\u00ebsh nj\u00eb shprehje shum\u00eb ofenduese, por na lejoi t\u00eb arrijm\u00eb shpejt nj\u00eb nivel t\u00eb mir\u00eb t\u00eb automatizimit dhe na dha koh\u00eb p\u00ebr t\u00eb trajnuar modele m\u00eb t\u00eb komplikuara.<\/p>\n

Versionet e ardhshme t\u00eb klasifikuesve p\u00ebr tekstet e mira p\u00ebrfshinin tashm\u00eb edhe modele linjare, edhe pem\u00eb vendimtar\u00ebsh, dhe kombinimet e tyre. P\u00ebr sh\u00ebnimin e fyerjeve dhe ofendimeve, ne, p\u00ebr shembull, provuam rrjetin neuro BERT. K\u00ebtu \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb kap\u00ebsh kuptimin e fjal\u00ebs n\u00eb kontekst dhe marr\u00ebdh\u00ebnien e fjal\u00ebve nga fraza t\u00eb ndryshme, dhe BERT e b\u00ebn k\u00ebt\u00eb mjaft mir\u00eb. (P\u00ebr mendim, koleg\u00ebt tan\u00eb nga Lajmet, si e aplikojn\u00eb teknologjin\u00eb p\u00ebr nj\u00eb detyr\u00eb jo standarde - gjetjen e gabimeve n\u00eb tituj.) Si rezultat, arrit\u00ebm t\u00eb automatizojm\u00eb deri n\u00eb 90% t\u00eb fluksit n\u00eb var\u00ebsi t\u00eb sh\u00ebrbimit. treguam<\/a><\/noindex>Sakt\u00ebsia, plotesia dhe shpejt\u00ebsia <\/p>\n

P\u00ebr t\u00eb u zhvilluar, \u00ebsht\u00eb e nevojshme t\u00eb kuptohet se \u00e7far\u00eb dobi sjellin klasifikuesit automatik\u00eb, ndryshimet n\u00eb to, si dhe n\u00ebse cil\u00ebsia e kontrollimeve manuale degradohet. P\u00ebr k\u00ebt\u00eb, ne aplikojm\u00eb treguesit e sakt\u00ebsis\u00eb dhe plot\u00ebsis\u00eb.<\/h3>\n

\nP\u00ebr t\u00eb u zhvilluar, duhet t\u00eb kuptojm\u00eb se cila \u00ebsht\u00eb dobi sjellin klasifikuesit automatike, ndryshimet n\u00eb to, si dhe n\u00ebse nuk po degradohet cil\u00ebsia e kontrollit manual. P\u00ebr k\u00ebt\u00eb ne p\u00ebrdorim treguesit e sakt\u00ebsis\u00eb dhe plot\u00ebsis\u00eb. <\/p>\n

Sakt\u00ebsia \u00ebsht\u00eb p\u00ebrqindja e verdikteve t\u00eb sakta n\u00eb mesin e t\u00eb gjitha verdikteve mbi p\u00ebrmbajtjen e keqe. Sa m\u00eb e lart\u00eb t\u00eb jet\u00eb sakt\u00ebsia, aq m\u00eb pak jan\u00eb alarmet false. N\u00ebse nuk e monitorojm\u00eb sakt\u00ebsin\u00eb, n\u00eb teori mund t\u00eb fshijm\u00eb gjith\u00eb spam-in dhe fjal\u00ebt p\u00ebrbuz\u00ebse, dhe s\u00eb bashku me to gjysm\u00ebn e mesazheve t\u00eb mira. Nga ana tjet\u00ebr, n\u00ebse mb\u00ebshtetemi vet\u00ebm n\u00eb sakt\u00ebsi, teknologjia m\u00eb e mir\u00eb do t\u00eb ishte ajo q\u00eb nuk kap ask\u00ebnd. Prandaj ka edhe nj\u00eb tregues tjet\u00ebr, plot\u00ebsin\u00eb: p\u00ebrqindja e p\u00ebrmbajtjes s\u00eb keqe t\u00eb zbuluar n\u00eb mesin e gjith\u00eb sasis\u00eb s\u00eb p\u00ebrmbajtjes s\u00eb keqe. K\u00ebto dy metrika balancojn\u00eb nj\u00ebra-tjetr\u00ebn. <\/p>\n

P\u00ebr matjen e saj, ne marrim mostra nga i gjith\u00eb fluksi i hyrjes p\u00ebr \u00e7do sh\u00ebrbim dhe ia dor\u00ebzojm\u00eb p\u00ebrmbajtjet asessor\u00ebve p\u00ebr vler\u00ebsim ekspert dhe krahasim me vendimet e makin\u00ebs. <\/p>\n

Por ka edhe nj\u00eb tregues tjet\u00ebr t\u00eb r\u00ebnd\u00ebsish\u00ebm. <\/p>\n

M\u00eb sip\u00ebr kam shkruar se nj\u00eb mesazh i papranuesh\u00ebm mund t\u00eb shihet nga qindra njer\u00ebz brenda 5 minutash. Prandaj ne llogarisim se sa her\u00eb arrit\u00ebm t\u00eb tregojm\u00eb p\u00ebrmbajtjen e keqe p\u00ebr para se ta fshijm\u00eb at\u00eb. Kjo \u00ebsht\u00eb e r\u00ebnd\u00ebsishme, sepse nuk mjafton t\u00eb punosh mir\u00eb - duhet t\u00eb punosh gjithashtu shpejt. Dhe kur po ndreqnim mbrojtjen nga fjal\u00ebt p\u00ebrbuz\u00ebse, e ndjem\u00eb k\u00ebt\u00eb plot\u00ebsisht. <\/p>\n

Antifjal\u00eb p\u00ebr shembullin e maceve dhe qenve<\/h3>\n

\nNj\u00eb shpjegim i vog\u00ebl. Disa mund t\u00eb thon\u00eb se fjal\u00ebt e pahijshme dhe ofendimet nuk jan\u00eb aq t\u00eb rrezikshme sa lidhjet e d\u00ebmshme dhe nuk jan\u00eb aq t\u00eb bezdisshme si spam. Por ne p\u00ebrpiqemi t\u00eb mb\u00ebshtesim kushte t\u00eb rehatshme p\u00ebr bised\u00eb p\u00ebr miliona p\u00ebrdorues, dhe njer\u00ebzit nuk p\u00eblqejn\u00eb t\u00eb kthehen atje ku ofendohen. Nuk \u00ebsht\u00eb rast\u00ebsi q\u00eb ndalimi i fjal\u00ebve p\u00ebrbuz\u00ebse dhe ofendimeve \u00ebsht\u00eb i p\u00ebrcaktuar n\u00eb rregullat e shum\u00eb komuniteteve, p\u00ebrfshir\u00eb dhe Habrin. Por u t\u00ebrhoq\u00ebm. <\/p>\n

Fjalor\u00ebt e fjal\u00ebve p\u00ebrbuz\u00ebse nuk p\u00ebrballen me gjith\u00eb pasurin\u00eb e gjuh\u00ebs ruse. Megjith\u00ebse ka vet\u00ebm kat\u00ebr rr\u00ebnj\u00eb kryesore t\u00eb fjal\u00ebve p\u00ebrbuz\u00ebse, prej tyre mund t\u00eb krijosh nj\u00eb num\u00ebr t\u00eb panum\u00ebrt fjal\u00ebsh, q\u00eb nuk i kap dot me asnj\u00eb shprehje t\u00eb rregullt. P\u00ebr m\u00eb tep\u00ebr, mund t\u00eb shkruash pjes\u00eb t\u00eb fjal\u00ebs n\u00eb transliterim, t\u00eb z\u00ebvend\u00ebsosh shkronjat me kombinime t\u00eb ngjashme, t\u00eb p\u00ebrzjes\u00ebsh shkronjat, t\u00eb shtosh yje etj. Disa her\u00eb \u00ebsht\u00eb e pamundur t\u00eb p\u00ebrcaktosh se \u00e7far\u00eb donte t\u00eb thoshte p\u00ebrdoruesi pa kontekst. Ne respektuam rregullat e Habr\u00ebs, prandaj do ta demonstrojm\u00eb k\u00ebt\u00eb jo me shembuj t\u00eb v\u00ebrtet\u00eb, por me mace dhe qen. <\/p>\n

\"Si
\n
\n\u00abMiau\u00bb, - tha macja. Por ne e kuptojm\u00eb se macja tha nj\u00eb fjal\u00eb tjet\u00ebr...<\/p>\n

Fillojm\u00eb t\u00eb mendojm\u00eb p\u00ebr algoritmet e 'p\u00ebrputhjes fuzzy' t\u00eb fjalorit ton\u00eb dhe p\u00ebr nj\u00eb paraprocesim m\u00eb inteligjent: p\u00ebrgatit\u00ebm transliterimin, bashkuam hap\u00ebsirat dhe pik\u00ebzimin, k\u00ebrkuam modele dhe shkruam shprehje t\u00eb ve\u00e7anta p\u00ebr to. Ky qasje dha rezultate, por shpesh ulte sakt\u00ebsin\u00eb, duke mos ofruar plot\u00ebsin\u00eb e d\u00ebshiruar.<\/p>\n

K\u00ebshtu, ne vendos\u00ebm t\u00eb 'mendojm\u00eb si ata q\u00eb p\u00ebrdorin fjal\u00eb t\u00eb ndaluara'. Filluam t\u00eb shtojm\u00eb zhurm\u00ebn n\u00eb t\u00eb dh\u00ebna: rregullojm\u00eb shkronjat, gjenerojm\u00eb gabime shkrimi, z\u00ebvend\u00ebsojm\u00eb shkronjat me ato t\u00eb ngjashme dhe k\u00ebshtu me radh\u00eb. Disa sh\u00ebnime fillestare p\u00ebr k\u00ebt\u00eb mor\u00ebm duke aplikuar fjalor\u00ebt e fjal\u00ebve t\u00eb ndaluara n\u00eb korpus t\u00eb m\u00ebdha teksti. N\u00ebse marrim nj\u00eb fjali dhe e deformojm\u00eb n\u00eb disa m\u00ebnyra, mund t\u00eb kemi shum\u00eb fjali. K\u00ebshtu mund t\u00eb rrisim numrin e mostrave p\u00ebr trajnimin n\u00eb dhjet\u00ebra her\u00eb. Ishte e nevojshme vet\u00ebm t\u00eb trajnohej nj\u00eb model m\u00eb inteligjent, q\u00eb merr parasysh kontekstin, me pool-in e marzhit t\u00eb marr\u00eb. <\/p>\n

\"Si
\n
\n\u00cbsht\u00eb akoma her\u00ebt t\u00eb flasim p\u00ebr zgjidhjen p\u00ebrfundimtare. Ne ende po eksperimentojm\u00eb me qasje t\u00eb ndryshme p\u00ebr k\u00ebt\u00eb problem, por tashm\u00eb po shohim se nj\u00eb rrjet i thjesht\u00eb konvolucional simbolesh nga disa shtresa \u00ebsht\u00eb duksh\u00ebm m\u00eb i mir\u00eb se fjalor\u00ebt dhe shprehjet e rregullta: mund t\u00eb rritet si sakt\u00ebsia ashtu edhe plot\u00ebsia.<\/p>\n

Natyrisht, kuptojm\u00eb se gjithmon\u00eb do t\u00eb ekzistojn\u00eb m\u00ebnyra p\u00ebr t\u00eb kap\u00ebrcyer edhe automatik\u00ebn m\u00eb t\u00eb avancuar, sidomos kur ajo \u00ebsht\u00eb kaq e emocionuese: t\u00eb shkruash n\u00eb nj\u00eb m\u00ebnyr\u00eb q\u00eb makina e budalla\u00e7k \u00ebsht\u00eb e paaft\u00eb ta kuptoj\u00eb. K\u00ebtu, ashtu si n\u00eb luft\u00ebn kund\u00ebr spam-it, q\u00ebllimi yn\u00eb nuk \u00ebsht\u00eb t\u00eb eliminojm\u00eb mund\u00ebsin\u00eb p\u00ebr t\u00eb shkruar ndonj\u00eb gj\u00eb t\u00eb pajustifikuar, por t\u00eb sigurojm\u00eb q\u00eb loja t\u00eb mos vlej\u00eb gjith\u00e7ka.<\/p>\n

T\u00eb hap\u00ebsh mund\u00ebsin\u00eb p\u00ebr t\u00eb ndar\u00eb mendimet e tua, p\u00ebr t\u00eb komunikuar dhe komentuar \u2014 nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb. M\u00eb e v\u00ebshtir\u00eb \u00ebsht\u00eb t\u00eb arrish kushte t\u00eb sigurta, komode dhe nj\u00eb respekt t\u00eb nd\u00ebrsjell\u00eb ndaj njer\u00ebzve. Dhe pa k\u00ebt\u00eb, nuk do t\u00eb ket\u00eb zhvillim t\u00eb \u00e7do komuniteti.
\n
Burimi: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041a\u043e\u0433\u0434\u0430 \u0442\u0432\u043e\u044f \u0440\u0430\u0431\u043e\u0442\u0430 \u2014 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0442\u043e-\u0442\u043e \u043a\u0440\u0430\u0441\u0438\u0432\u043e\u0435, \u043e \u043d\u0435\u0439 \u043c\u043e\u0436\u043d\u043e \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e \u043d\u0435 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u0442\u044c, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u0443 \u0432\u0441\u0435\u0445 \u043f\u0435\u0440\u0435\u0434 \u0433\u043b\u0430\u0437\u0430\u043c\u0438. \u0410 \u0432\u043e\u0442 \u0435\u0441\u043b\u0438 \u0442\u044b \u0441\u0442\u0438\u0440\u0430\u0435\u0448\u044c \u043d\u0430\u0434\u043f\u0438\u0441\u0438 \u0441 \u0437\u0430\u0431\u043e\u0440\u043e\u0432, \u0442\u0432\u043e\u044e \u0440\u0430\u0431\u043e\u0442\u0443 \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0437\u0430\u043c\u0435\u0447\u0430\u0435\u0442, \u043f\u043e\u043a\u0430 \u0437\u0430\u0431\u043e\u0440\u044b \u0432\u044b\u0433\u043b\u044f\u0434\u044f\u0442 \u043f\u0440\u0438\u043b\u0438\u0447\u043d\u043e \u0438\u043b\u0438 \u043f\u043e\u043a\u0430 \u0442\u044b \u043d\u0435 \u0441\u043e\u0442\u0440\u0451\u0448\u044c \u0447\u0442\u043e-\u043d\u0438\u0431\u0443\u0434\u044c \u043d\u0435 \u0442\u043e. \u041b\u044e\u0431\u043e\u0439 \u0441\u0435\u0440\u0432\u0438\u0441, \u0433\u0434\u0435 \u043c\u043e\u0436\u043d\u043e \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0439, \u043e\u0442\u0437\u044b\u0432, \u043e\u0442\u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 \u0438\u043b\u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":72899,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-72898","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Si si hapin komentet dhe nuk fundoseni n\u00eb spam | ProHoster","description":"Kur kur pun\u00ebn t\u00ebnde \u00ebsht\u00eb t\u00eb krijosh di\u00e7ka.","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043e\u0442\u043a\u0440\u044b\u0442\u044c \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0438 \u0438 \u043d\u0435 \u043f\u043e\u0442\u043e\u043d\u0443\u0442\u044c \u0432 \u0441\u043f\u0430\u043c\u0435 | ProHoster","og:description":"\u041a\u043e\u0433\u0434\u0430 \u0442\u0432\u043e\u044f \u0440\u0430\u0431\u043e\u0442\u0430 \u2014 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0442\u043e-\u0442\u043e.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-05T17:42:08+00:00","article:modified_time":"2020-03-05T17:42:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"72898","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 18:43:27","updated":"2022-09-29 22:58:18","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/72898","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=72898"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/72898\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/72899"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=72898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=72898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=72898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}