{"id":72898,"date":"2020-03-05T20:42:08","date_gmt":"2020-03-05T17:42:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame"},"modified":"2020-03-05T20:42:08","modified_gmt":"2020-03-05T17:42:08","slug":"kak-otkryt-kommentarii-i-ne-potonut-v-spame","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","title":{"rendered":"Si t\u00eb hapni komentet dhe t\u00eb mos fundoseni n\u00eb spam","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Si
\n
\nKur kur pun\u00eb e jotja \u00ebsht\u00eb t\u00eb krijosh di\u00e7ka t\u00eb bukur, nuk ka nevoj\u00eb t\u00eb flas\u00ebsh shum\u00eb p\u00ebr t\u00eb, sepse rezultati \u00ebsht\u00eb p\u00ebrpara t\u00eb gjith\u00ebve. Por n\u00ebse fshin shkrimet nga gardhet, askush nuk e v\u00ebren pun\u00ebn t\u00ebnde, derisa gardhet t\u00eb duken mir\u00eb ose derisa t\u00eb fshish di\u00e7ka t\u00eb gabuar.<\/p>\n

\u00c7do sh\u00ebrbim ku mund t\u00eb l\u00ebsh nj\u00eb koment, rishikim, t\u00eb d\u00ebrgosh nj\u00eb mesazh apo t\u00eb ngarkosh figura, p\u00ebrball\u00ebt p\u00ebr nj\u00eb moment me problemin e spam-it, mashtrimit dhe fjal\u00ebve t\u00eb pahijshme. Kjo nuk mund t\u00eb shmanget, por duhet t\u00eb luftohet. <\/p>\n

M\u00eb quajn\u00eb Mikhail, punoj n\u00eb ekipin e Antispamit, i cili mbron p\u00ebrdoruesit e sh\u00ebrbimeve t\u00eb Yandex-it nga probleme t\u00eb tilla. Puna jon\u00eb r\u00ebndom nuk \u00ebsht\u00eb e dukshme (dhe mir\u00eb \u00ebsht\u00eb!), prandaj sot do t\u00eb flas m\u00eb shum\u00eb p\u00ebr t\u00eb. Do t\u00eb m\u00ebsoni se n\u00eb cilat raste moderimi \u00ebsht\u00eb i kote dhe pse sakt\u00ebsia nuk \u00ebsht\u00eb treguesi i vet\u00ebm i efektivitetit t\u00eb tij. Gjithashtu, do t\u00eb flasim p\u00ebr fjal\u00ebt e pahijshme duke marr\u00eb si shembuj macet dhe qent\u00eb, dhe p\u00ebrse ndonj\u00ebher\u00eb \u00ebsht\u00eb e dobishme t\u00eb \"mendosh si nj\u00eb fjalor i pahijsh\u00ebm\".
\n<\/a><\/noindex>
\nN\u00eb Yandex po shfaqen gjithnj\u00eb e m\u00eb shum\u00eb sh\u00ebrbime ku p\u00ebrdoruesit publikojn\u00eb p\u00ebrmbajtjen e tyre. Mund t\u00eb rivalizosh nj\u00eb pyetje ose t\u00eb shkruash nj\u00eb p\u00ebrgjigje n\u00eb Yandex.Q, t\u00eb diskutosh lajmet e lagjes n\u00eb Yandex.Region, t\u00eb ndash informacion p\u00ebr gjendjen e rrug\u00ebve n\u00eb bisedat n\u00eb Yandex.Maps. Por kur publiku i sh\u00ebrbimit rritet, ai b\u00ebhet t\u00ebrheq\u00ebs p\u00ebr mashtruesit dhe spamer\u00ebt. Ata vijn\u00eb dhe mbushin komentet: ofrojn\u00eb mund\u00ebsi fitimi, reklamojn\u00eb ila\u00e7e t\u00eb mrekullueshme dhe premtojn\u00eb ndihma sociale. Fal\u00eb spam-it disa p\u00ebrdorues humbasin para, kurse t\u00eb tjer\u00eb humbasin d\u00ebshir\u00ebn t\u00eb kalojn\u00eb koh\u00eb n\u00eb nj\u00eb sh\u00ebrbim t\u00eb papast\u00ebrt, t\u00eb mbushur me spam.<\/p>\n

Dhe kjo nuk \u00ebsht\u00eb e vetmja problem. Ne p\u00ebrpiqemi jo vet\u00ebm t\u00eb mbrojm\u00eb p\u00ebrdoruesit nga mashtruesit, por edhe t\u00eb krijojm\u00eb nj\u00eb atmosfer\u00eb t\u00eb rehatshme p\u00ebr komunikim. N\u00ebse njer\u00ebzit p\u00ebrballen me fjal\u00eb t\u00eb pahijshme dhe ofendime n\u00eb komente, ka shum\u00eb gjasa q\u00eb ata t\u00eb largohen dhe t\u00eb mos kthehen m\u00eb. K\u00ebshtu q\u00eb duhet t\u00eb dim\u00eb se si t\u00eb luftojm\u00eb edhe k\u00ebt\u00eb. <\/p>\n

Web i Past\u00ebr<\/h3>\n

\nSi\u00e7 ndodh shpesh, p\u00ebrpjekjet e para lindi n\u00eb K\u00ebrkimin t\u00eb Yandex, n\u00eb at\u00eb pjes\u00eb q\u00eb lufton me spam-in n\u00eb rezultatet. Q\u00eb dhjet\u00eb vjet m\u00eb par\u00eb aty u shfaq p\u00ebrpjekja p\u00ebr filtrimin e p\u00ebrmbajtjeve p\u00ebr t\u00eb rritur, p\u00ebr k\u00ebrkesat q\u00eb nuk pranojn\u00eb p\u00ebrgjigje nga kategoria 18+. K\u00ebshtu lindi fjalori i par\u00eb i p\u00ebrmbajtjeve t\u00eb pahijshme, t\u00eb cilat u plot\u00ebsuan nga analist\u00ebt. Objektivi kryesor ishte klasifikimi i k\u00ebrkesave n\u00eb ato ku \u00ebsht\u00eb e lejueshme t\u00eb shfaqet p\u00ebrmbajtje p\u00ebr t\u00eb rritur dhe ku nuk \u00ebsht\u00eb. P\u00ebr k\u00ebt\u00eb q\u00ebllim u grumbullua sh\u00ebnimi, u nd\u00ebrtuan heuristika, u st\u00ebrvit\u00ebn modelet. K\u00ebshtu lindi p\u00ebrpjekjet p\u00ebr filtrimin e p\u00ebrmbajtjes s\u00eb pad\u00ebshiruar.<\/p>\n

Me kalimin e koh\u00ebs, Yandex filloi t\u00eb integroj\u00eb UGC (p\u00ebrmbajtje e prodhuar nga p\u00ebrdoruesit) \u2014 mesazhe q\u00eb i shkruajn\u00eb vet\u00eb p\u00ebrdoruesit, nd\u00ebrsa Yandex thjesht i publikon. P\u00ebr shkak t\u00eb arsyeve t\u00eb sip\u00ebrp\u00ebrmendura, shum\u00eb mesazhe nuk mund t\u00eb publikohen pa nj\u00eb shqyrtim \u2014 k\u00ebrkohej moderim. At\u00ebher\u00eb u vendos t\u00eb krijohej nj\u00eb sh\u00ebrbim q\u00eb do t\u00eb ofronte mbrojtje nga spam-i dhe keqb\u00ebr\u00ebsit p\u00ebr t\u00eb gjitha produktet UGC t\u00eb Yandex dhe e p\u00ebrdorte p\u00ebrvoj\u00ebn p\u00ebr filtrimin e p\u00ebrmbajtjes s\u00eb pad\u00ebshiruar n\u00eb K\u00ebrkim. Sh\u00ebrbimi u quajt \"Web i Past\u00ebr\". <\/p>\n

Detyra t\u00eb reja dhe ndihma nga punonj\u00ebsit<\/h3>\n

\nFillimisht, ne kishim vet\u00ebm automatik\u00eb t\u00eb thjesht\u00eb: sh\u00ebrbimet na d\u00ebrgonin tekstet, dhe ne i d\u00ebrgonim p\u00ebrmes fjalor\u00ebve t\u00eb fjal\u00ebve t\u00eb pahijshme, fjalor\u00ebve t\u00eb pornografis\u00eb dhe shprehjeve normale \u2014 analist\u00ebt e p\u00ebrgatitnin gjith\u00e7ka manualisht. Por sh\u00ebrbimi me kalimin e koh\u00ebs filloi t\u00eb aplikohej n\u00eb nj\u00eb num\u00ebr gjithnj\u00eb e m\u00eb t\u00eb madh produktesh t\u00eb Yandex, dhe na duhet t\u00eb m\u00ebsojm\u00eb se si t\u00eb p\u00ebrballojm\u00eb me probleme t\u00eb reja. <\/p>\n

Shpesh n\u00eb vend t\u00eb nj\u00eb rishikimi, p\u00ebrdoruesit publikojn\u00eb nj\u00eb grup pa kuptim shkronjash, duke u p\u00ebrpjekur t\u00eb rrisin arritjet e tyre, ndonj\u00ebher\u00eb reklamojn\u00eb kompanin\u00eb e tyre n\u00eb rishikimet p\u00ebr kompanin\u00eb e konkurrentit, e ndonj\u00ebher\u00eb thjesht ngat\u00ebrrojn\u00eb organizatat dhe n\u00eb nj\u00eb rishikim p\u00ebr nj\u00eb dyqan kafsh\u00ebsh shkruajn\u00eb: \"Peshku i p\u00ebrgatitur shk\u00eblqyesh\u00ebm!\". Ndoshta, ndonj\u00ebher\u00eb inteligjenca artificiale do t\u00eb m\u00ebsoj\u00eb t\u00eb kap\u00eb plot\u00ebsisht kuptimin e \u00e7do teksti, por tani automatik\u00eb her\u00eb pas here n\u00eb fakt ndihmon m\u00eb keq se njeriu. <\/p>\n

U b\u00eb e qart\u00eb se pa sh\u00ebnimin manual k\u00ebtu nuk do t\u00eb ishte e mundur, dhe ne integruam nj\u00eb shkall\u00eb t\u00eb dyt\u00eb n\u00eb konturin ton\u00eb \u2014 d\u00ebrgimin p\u00ebr shqyrtim manual nga nj\u00eb njeri. Aty kalonin ato tekste t\u00eb publikuara, n\u00eb lidhje me t\u00eb cilat klasifikuesi nuk shihte probleme. Shkalla e nj\u00eb detyre t\u00eb till\u00eb mund ta imagjinoni leht\u00eb, prandaj nuk u mb\u00ebshtet\u00ebm vet\u00ebm n\u00eb asessor\u00eb, por edhe shfryt\u00ebzuam \"men\u00e7urin\u00eb e turm\u00ebs\", p\u00ebrkat\u00ebsisht, iu drejtuam ndihm\u00ebsve t\u00eb tjer\u00eb p\u00ebr ndihm\u00eb. Ata ndihmojn\u00eb n\u00eb zbulimin e asaj q\u00eb makina ka humbur dhe n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb e m\u00ebsojn\u00eb at\u00eb. <\/p>\n

Kesh i men\u00e7ur dhe hashing LSH<\/h3>\n

\nNj\u00eb tjet\u00ebr problem me t\u00eb cilin u p\u00ebrball\u00ebm gjat\u00eb pun\u00ebs me komentet ishte spamu, konkretisht sasia dhe shpejt\u00ebsia e shp\u00ebrndarjes s\u00eb tij. Kur audienca e Yandex. Rajonit filloi t\u00eb rritet shpejt, t\u00eb par\u00ebt q\u00eb erdh\u00ebn ishin spamers. Ata m\u00ebsuan t\u00eb kalonin p\u00ebrmes rregullave duke ndryshuar pak tekstin. Spam, sigurisht, ende u zbulua dhe u hoq, por n\u00eb shkall\u00ebn e Yandex-it, nj\u00eb mesazh i papranuesh\u00ebm edhe p\u00ebr 5 minuta mund t\u00eb shihej nga qindra njer\u00ebz. <\/p>\n

\"Si
\n
\nSigurisht, ne nuk ishim t\u00eb k\u00ebnaqur me k\u00ebt\u00eb, dhe ne realizuam nj\u00eb keqese t\u00eb men\u00e7ur t\u00eb teksteve mbi baz\u00ebn e LSH (hashimi sensitiv ndaj vendndodhjes<\/a><\/noindex>). Kjo funksionon n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb: ne normalizuam tekstin, heq\u00ebm lidhjet nga t\u00eb dhe e ndan\u00eb at\u00eb n\u00eb n-grama (sekuenca nga n letra). M\u00eb pas llogarit\u00ebm hash-et nga n-gramat dhe nd\u00ebrtuam nj\u00eb vektor LSH t\u00eb dokumentit. Q\u00ebllimi \u00ebsht\u00eb q\u00eb tekste t\u00eb ngjashme, edhe n\u00ebse jan\u00eb ndryshuar pak, t\u00eb shnd\u00ebrrohen n\u00eb vektora t\u00eb ngjash\u00ebm. <\/p>\n

Ky zgjidhje na lejoi t\u00eb rip\u00ebrdorim vendimet e klasifikuesve dhe t\u00eb toloker\u00ebve p\u00ebr tekste t\u00eb ngjashme. N\u00eb rastin e sulmeve me spam, sapo mesazhi i par\u00eb kalonte verifikimin dhe p\u00ebrfundonte n\u00eb keqese me vendim \"spam\", t\u00eb gjitha mesazhet e reja t\u00eb ngjashme, edhe t\u00eb ndryshuara, merrnin t\u00eb nj\u00ebjtin vendim dhe fshiheshin automatikisht. M\u00eb von\u00eb m\u00ebsuam t\u00eb trajnojm\u00eb dhe automatikisht t\u00eb ritrainojm\u00eb klasifikuesit e spamit, por kjo \"keqese e men\u00e7ur\" ka mbetur me ne dhe ende na ndihmon shpesh.<\/p>\n

Klasifikuesi i teksteve t\u00eb mira<\/h3>\n

\nPa vonuar nga lufta me spamin, ne realizuam se 95% e p\u00ebrmbajtjes son\u00eb moderoheshin manualisht: klasifikuesit reagojn\u00eb vet\u00ebm ndaj shkeljeve, nd\u00ebrsa shumica e teksteve jan\u00eb t\u00eb mira. Ne e ngarkojm\u00eb toloker\u00ebt, t\u00eb cil\u00ebt n\u00eb 95 raste nga 100 japin vler\u00ebsimin \"Gj\u00ebja OK\". Duhej t\u00eb merreshim me nj\u00eb pun\u00eb t\u00eb paprekshme - t\u00eb b\u00ebnim klasifikues p\u00ebr p\u00ebrmbajtjen e mir\u00eb, fatmir\u00ebsisht, gjat\u00eb k\u00ebsaj kohe ishin mbledhur mjaft t\u00eb dh\u00ebna.<\/p>\n

Klasifikuesi i par\u00eb dukej k\u00ebshtu: lematisuam tekstin (kthyer n\u00eb form\u00ebn e tij fillestare), heq\u00ebm t\u00eb gjitha pjes\u00ebt flit\u00ebse dhe aplikonim nj\u00eb \"fjalor t\u00eb mire\" p\u00ebr lemma. N\u00ebse n\u00eb tekst t\u00eb gjitha fjal\u00ebt jan\u00eb \"t\u00eb mira\", at\u00ebher\u00eb edhe t\u00ebr\u00ebsisht nuk p\u00ebrmban shkelje. N\u00eb sh\u00ebrbime t\u00eb ndryshme, ky qasje dha menj\u00ebher\u00eb nga 25 deri n\u00eb 35% automatizim t\u00eb vler\u00ebsimeve manuale. Sigurisht, ky qasje nuk \u00ebsht\u00eb perfekt: nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb kompozosh disa fjal\u00eb t\u00eb pafajshme dhe t\u00eb krijosh nj\u00eb shprehje t\u00eb dhunshme, por kjo na lejoi t\u00eb arrijm\u00eb shpejt nj\u00eb nivel t\u00eb mir\u00ebnjohur t\u00eb automatizimit dhe na dha koh\u00eb p\u00ebr t\u00eb trajnuar modele m\u00eb t\u00eb komplikuara.<\/p>\n

Versionet e ardhshme t\u00eb klasifikuesve t\u00eb teksteve t\u00eb mira tashm\u00eb p\u00ebrfshinin modele lineare, pem\u00ebt vendimtare dhe kombinimet e tyre. P\u00ebr vler\u00ebsimin e fjal\u00ebve t\u00eb papranueshme dhe ofenduese, ne, p\u00ebr shembull, po provojm\u00eb rrjetin neural BERT. K\u00ebtu \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb kap\u00ebsh kuptimin e fjal\u00ebs n\u00eb kontekst dhe lidhjen e fjal\u00ebve nga proza t\u00eb ndryshme dhe BERT b\u00ebn mir\u00eb me k\u00ebt\u00eb. (Mesazhuam, koleg\u00ebt tan\u00eb nga Lajmet, si e p\u00ebrdorin teknologjin\u00eb p\u00ebr nj\u00eb detyr\u00eb t\u00eb pazakont\u00eb \u2014 k\u00ebrkimin e gabimeve n\u00eb tituj.) N\u00eb fund arrit\u00ebm t\u00eb automatizojm\u00eb deri n\u00eb 90% t\u00eb fluksit n\u00eb var\u00ebsi t\u00eb sh\u00ebrbimit. fol\u00ebm<\/a><\/noindex>, si si p\u00ebrdor teknologjia p\u00ebr nj\u00eb detyr\u00eb jasht\u00ebzakonshme - gjetjen e gabimeve n\u00eb tituj.) Si rezultat, arrit\u00ebm t\u00eb automatizojm\u00eb deri n\u00eb 90% t\u00eb fluksit n\u00eb var\u00ebsi t\u00eb sh\u00ebrbimit. <\/p>\n

Sakt\u00ebsia, plotesia dhe shpejt\u00ebsia<\/h3>\n

\nP\u00ebr t\u00eb evoluar, duhet t\u00eb kuptohet se cila \u00ebsht\u00eb dobi nga klasifikuesit automatik\u00eb t\u00eb caktuar, ndryshimet n\u00eb to si dhe n\u00ebse cil\u00ebsia e kontrolleve manuale nuk po degradohet. P\u00ebr k\u00ebt\u00eb, ne p\u00ebrdorim treguesit e sakt\u00ebsis\u00eb dhe plotesis\u00eb. <\/p>\n

Sakt\u00ebsia \u00ebsht\u00eb pjesa e vendimeve t\u00eb sakta midis t\u00eb gjith\u00eb vendimeve p\u00ebr p\u00ebrmbajtjen e keqe. Sa m\u00eb e lart\u00eb t\u00eb jet\u00eb sakt\u00ebsia, aq m\u00eb pak alarmin e rrem\u00eb ka. N\u00ebse nuk e ndjekim sakt\u00ebsin\u00eb, n\u00eb teori mund t\u00eb hiqet i gjith\u00eb spami dhe sharjet, s\u00eb bashku me gjysm\u00ebn e mesazheve t\u00eb mira. Nga ana tjet\u00ebr, n\u00ebse mb\u00ebshtetemi vet\u00ebm n\u00eb sakt\u00ebsi, teknologjia m\u00eb e mir\u00eb do t\u00eb jet\u00eb ajo q\u00eb nuk z\u00eb asnj\u00eb. Prandaj ka edhe tregues plotesie: pjesa e p\u00ebrmbajtjes s\u00eb keqe t\u00eb identifikuar midis t\u00eb gjith\u00eb volumit t\u00eb p\u00ebrmbajtjes s\u00eb keqe. K\u00ebto dy metrika balancojn\u00eb nj\u00ebra-tjetr\u00ebn. <\/p>\n

P\u00ebr t\u00eb matur ne provojm\u00eb t\u00eb gjith\u00eb rrjedh\u00ebn e hyrjeve p\u00ebr \u00e7do sh\u00ebrbim dhe i d\u00ebrgojm\u00eb mostrat e p\u00ebrmbajtjes p\u00ebr vler\u00ebsim t\u00eb ekspert\u00ebve dhe krahasim me vendimet e makin\u00ebs. <\/p>\n

Por ka edhe nj\u00eb tregues t\u00eb r\u00ebnd\u00ebsish\u00ebm. <\/p>\n

M\u00eb lart shkrova q\u00eb nj\u00eb mesazh i papranuesh\u00ebm mund t\u00eb shikohet nga qindra njer\u00ebz edhe p\u00ebr 5 minuta. Prandaj ne llogarisim sa her\u00eb kemi arritur t\u00eb tregojm\u00eb njer\u00ebzve p\u00ebrmbajtje t\u00eb keqe p\u00ebrpara se ta fshijm\u00eb. Kjo \u00ebsht\u00eb e r\u00ebnd\u00ebsishme, sepse nuk \u00ebsht\u00eb vet\u00ebm e mjaftueshme t\u00eb punosh me cil\u00ebsi \u2014 duhet gjithashtu t\u00eb punosh shpejt. Dhe kur nd\u00ebrtuam mbrojtjen nga sharjet, k\u00ebt\u00eb e ndjem\u00eb plot\u00ebsisht. <\/p>\n

Antisharjet mbi shembujt e maceve dhe qenve<\/h3>\n

\nNj\u00eb shk\u00ebputje t\u00eb vog\u00ebl lirike. Disa mund t\u00eb thon\u00eb se fyerjet dhe p\u00ebrdorimi i fjal\u00ebve t\u00eb pahijshme nuk jan\u00eb aq t\u00eb rrezikshme sa lidhjet malinje dhe nuk jan\u00eb aq shqet\u00ebsuese sa spami. Por ne p\u00ebrpiqemi t\u00eb mbajm\u00eb kushte komode p\u00ebr komunikimin e miliona p\u00ebrdoruesve, dhe njer\u00ebzit nuk e p\u00eblqejn\u00eb t\u00eb kthehen atje ku ofendohen. Nuk \u00ebsht\u00eb pa q\u00ebllim q\u00eb ndalimi i fjal\u00ebve t\u00eb pahijshme dhe fyerjeve \u00ebsht\u00eb i p\u00ebrfshir\u00eb n\u00eb rregullat e shum\u00eb komuniteteve, duke p\u00ebrfshir\u00eb edhe Habr\u00ebn. Por po shp\u00ebrqendrohemi. <\/p>\n

Fjalor\u00ebt e fjal\u00ebve t\u00eb pahijshme nuk p\u00ebrballen me pasurit\u00eb e gjuh\u00ebs ruse. Megjith\u00ebse ka vet\u00ebm kat\u00ebr rr\u00ebnj\u00eb kryesore t\u00eb fjal\u00ebve t\u00eb pahijshme, prej tyre mund t\u00eb krijohet nj\u00eb num\u00ebr i pafund fjal\u00ebsh q\u00eb nuk i kap asnj\u00eb regex. P\u00ebr m\u00eb tep\u00ebr, mund t\u00eb shkruash nj\u00eb pjes\u00eb t\u00eb fjal\u00ebs me transliterim, t\u00eb z\u00ebvend\u00ebsosh shkronjat me kombinime t\u00eb ngjashme, t\u00eb rregullosh shkronjat, t\u00eb shtosh yje etj. Ndonj\u00ebher\u00eb pa kontekst \u00ebsht\u00eb e pamundur t\u00eb p\u00ebrcaktosh se \u00e7far\u00eb kishte parasysh p\u00ebrdoruesi me fjal\u00ebn e pahijshme. Ne respektojm\u00eb rregullat e Habr\u00ebs, prandaj do ta demonstrojm\u00eb k\u00ebt\u00eb jo me shembuj t\u00eb gjall\u00eb, por me mace dhe qen. <\/p>\n

\"Si
\n
\n\u00abMiau\u00bb, tha macja. Por ne e kuptojm\u00eb se macja tha nj\u00eb fjal\u00eb tjet\u00ebr...<\/p>\n

Kemi filluar t\u00eb mendojm\u00eb p\u00ebr algoritmet e \"nd\u00ebrprerjes s\u00eb pacaktuar\" t\u00eb fjalorit ton\u00eb dhe p\u00ebr nj\u00eb paraprocesim m\u00eb t\u00eb men\u00e7ur: unifikonim transliterimin, bashkonim hap\u00ebsirat dhe pik\u00ebt, k\u00ebrkonim modele dhe shkruanim regex t\u00eb ve\u00e7anta p\u00ebr to. Ky qasje jepte rezultate, por shpesh ulte sakt\u00ebsin\u00eb, duke mos ofruar plot\u00ebsin\u00eb e d\u00ebshiruar.<\/p>\n

At\u00ebher\u00eb vendos\u00ebm \"t\u00eb mendojm\u00eb si p\u00ebrdorues t\u00eb fjal\u00ebve t\u00eb pahijshme\". Filluam t\u00eb krijonim vet\u00eb zhurm\u00eb n\u00eb t\u00eb dh\u00ebna: rregullonim shkronjat, gjeneronim gabime shkrimi, z\u00ebvend\u00ebsonim shkronjat me ato t\u00eb ngjashme dhe k\u00ebshtu me radh\u00eb. Dhanim sh\u00ebnime fillestare p\u00ebr k\u00ebt\u00eb duke aplikuar fjalor\u00ebt e fjal\u00ebve t\u00eb pahijshme n\u00eb korpus t\u00eb m\u00ebdha teksti. N\u00ebse merrni nj\u00eb fjali dhe e deformoni at\u00eb n\u00eb disa m\u00ebnyra, do t\u00eb merrni shum\u00eb fraza. K\u00ebshtu mund t\u00eb rrisim mostrat p\u00ebr m\u00ebsim n\u00eb dhjet\u00ebfish. <\/p>\n

\"Si
\n
\nP\u00ebr zgjidhjen p\u00ebrfundimtare \u00ebsht\u00eb ende her\u00ebt p\u00ebr t\u00eb folur. Ne ende jemi duke eksperimentuar me qasjet ndaj k\u00ebsaj problemi, por tashm\u00eb shohim se nj\u00eb rrjet\u00eb konvencionale e karaktereve me disa shtresa tejkalon ndjesh\u00ebm fjalor\u00ebt dhe regex: arrijm\u00eb t\u00eb rrisim si sakt\u00ebsin\u00eb ashtu edhe plot\u00ebsin\u00eb.<\/p>\n

Sigurisht, ne e kuptojm\u00eb se gjithmon\u00eb do t\u00eb gjejn\u00eb m\u00ebnyra p\u00ebr t\u00eb kaluar edhe sistemin m\u00eb t\u00eb avancuar, ve\u00e7an\u00ebrisht kur b\u00ebhet fjal\u00eb p\u00ebr di\u00e7ka kaq emocionuese: t\u00eb shkruajm\u00eb n\u00eb m\u00ebnyr\u00eb q\u00eb makina t\u00eb mos kuptoj\u00eb. K\u00ebtu, si\u00e7 \u00ebsht\u00eb p\u00ebr luftimin e spamit, nuk \u00ebsht\u00eb q\u00ebllimi yn\u00eb t\u00eb eliminojm\u00eb mund\u00ebsin\u00eb p\u00ebr t\u00eb shkruar di\u00e7ka t\u00eb pahijshme, detyra jon\u00eb \u00ebsht\u00eb t\u00eb b\u00ebjm\u00eb q\u00eb loja t\u00eb mos ia vlej\u00eb.<\/p>\n

T\u00eb hap\u00ebsh mund\u00ebsin\u00eb p\u00ebr t\u00eb ndar\u00eb mendimin tuaj, p\u00ebr t\u00eb komunikuar dhe p\u00ebr t\u00eb komentuar \u2014 nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb. M\u00eb e v\u00ebshtir\u00eb \u00ebsht\u00eb t\u00eb arrihet kushtet e sigurta, komode dhe respektuese p\u00ebr njer\u00ebzit. Pa k\u00ebt\u00eb, nuk do t\u00eb ket\u00eb zhvillim t\u00eb ndonj\u00eb komuniteti.
\n
Burimi: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041a\u043e\u0433\u0434\u0430 \u0442\u0432\u043e\u044f \u0440\u0430\u0431\u043e\u0442\u0430 \u2014 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0442\u043e-\u0442\u043e \u043a\u0440\u0430\u0441\u0438\u0432\u043e\u0435, \u043e \u043d\u0435\u0439 \u043c\u043e\u0436\u043d\u043e \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e \u043d\u0435 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u0442\u044c, \u043f\u043e\u0442\u043e\u043c\u0443 \u0447\u0442\u043e \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u0443 \u0432\u0441\u0435\u0445 \u043f\u0435\u0440\u0435\u0434 \u0433\u043b\u0430\u0437\u0430\u043c\u0438. \u0410 \u0432\u043e\u0442 \u0435\u0441\u043b\u0438 \u0442\u044b \u0441\u0442\u0438\u0440\u0430\u0435\u0448\u044c \u043d\u0430\u0434\u043f\u0438\u0441\u0438 \u0441 \u0437\u0430\u0431\u043e\u0440\u043e\u0432, \u0442\u0432\u043e\u044e \u0440\u0430\u0431\u043e\u0442\u0443 \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0437\u0430\u043c\u0435\u0447\u0430\u0435\u0442, \u043f\u043e\u043a\u0430 \u0437\u0430\u0431\u043e\u0440\u044b \u0432\u044b\u0433\u043b\u044f\u0434\u044f\u0442 \u043f\u0440\u0438\u043b\u0438\u0447\u043d\u043e \u0438\u043b\u0438 \u043f\u043e\u043a\u0430 \u0442\u044b \u043d\u0435 \u0441\u043e\u0442\u0440\u0451\u0448\u044c \u0447\u0442\u043e-\u043d\u0438\u0431\u0443\u0434\u044c \u043d\u0435 \u0442\u043e. \u041b\u044e\u0431\u043e\u0439 \u0441\u0435\u0440\u0432\u0438\u0441, \u0433\u0434\u0435 \u043c\u043e\u0436\u043d\u043e \u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0439, \u043e\u0442\u0437\u044b\u0432, \u043e\u0442\u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0435 \u0438\u043b\u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":72899,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-72898","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Si t\u00eb hapni komentet dhe t\u00eb mos fundoseni n\u00eb spamin | ProHoster","description":"Kur kur punon \u00ebsht\u00eb t\u00eb krijosh di\u00e7ka.","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043e\u0442\u043a\u0440\u044b\u0442\u044c \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0438 \u0438 \u043d\u0435 \u043f\u043e\u0442\u043e\u043d\u0443\u0442\u044c \u0432 \u0441\u043f\u0430\u043c\u0435 | ProHoster","og:description":"\u041a\u043e\u0433\u0434\u0430 \u0442\u0432\u043e\u044f \u0440\u0430\u0431\u043e\u0442\u0430 \u2014 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0447\u0442\u043e-\u0442\u043e.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-otkryt-kommentarii-i-ne-potonut-v-spame","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-05T17:42:08+00:00","article:modified_time":"2020-03-05T17:42:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"72898","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 18:43:27","updated":"2022-09-29 22:58:18","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/72898","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=72898"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/72898\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/72899"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=72898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=72898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=72898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}