Tsiku lililonse, ogwiritsa ntchito mamiliyoni ambiri amabwera pamalo ochezera a pa Intaneti ndikufalitsa mabiliyoni azinthu: kuchokera pazithunzi mpaka makanema ndi zolemba. Odnoklassniki moderation platform imathandizira kuyang'ana ma data ambiri ndikuthana ndi spammers ndi bots.
Gulu la OK moderation lapeza zambiri, popeza lakhala likukonza zida zake kwa zaka 12. Ndikofunikira kuti asamangogawana nawo mayankho omwe adapangidwa kale, komanso kusintha mamangidwe a nsanja yawo kuti agwirizane ndi ntchito zathu zenizeni.
Kuyambira pano, pakufupikitsa, tingotcha nsanja ya OK "platform."
Momwe zonse zimagwirira ntchito
Kusinthana kwa data pakati pa Yula ndi Odnoklassniki kumakhazikitsidwa kudzera Apache Kafka.
Asanaphatikizepo, zolemba zimasinthidwa kuti zichotse zilembo zapadera, zilembo zosinthidwa ndi zinyalala zina. Deta yolandiridwa imagawidwa mu N-grams, yomwe ili ndi hashed. Zotsatira zake ndi ma hashi ambiri apadera. Kufanana kwa malemba kumatsimikiziridwa ndi Mulingo wa Jaccard pakati pa seti ziwiri zotsatira. Ngati kufanana kuli kwakukulu kuposa malire, ndiye kuti malembawo amaphatikizidwa kukhala gulu limodzi. Kuti mufulumizitse kusaka kwamagulu ofanana, MinHash ndi Locality-sensitive hashing amagwiritsidwa ntchito.
Zosankha zingapo zopangira zithunzi za gluing zidapangidwa kuti zizijambula, kuyambira kufananiza zithunzi za pHash mpaka kusaka zobwereza pogwiritsa ntchito neural network.
Njira yotsiriza ndiyo "yoopsa" kwambiri. Kuti aphunzitse chitsanzocho, zithunzi zitatu (N, A, P) zinasankhidwa zomwe N sizifanana ndi A, ndipo P ndi yofanana ndi A (ndi semi-duplicate). Kenako maukonde a neural adaphunzira kupanga A ndi P kukhala pafupi momwe angathere, ndi A ndi N momwe angathere. Izi zimabweretsa zabwino zabodza zochepa poyerekeza ndi kungotenga zoyika kuchokera pa netiweki yophunzitsidwa kale.
Neural network ikalandira zithunzi monga zolowetsa, imapanga vekitala ya N(128)-dimensional kwa aliyense wa iwo ndipo pempho limapangidwa kuti liwone kuyandikira kwa chithunzicho. Kenako, poyambira amawerengedwa pomwe zithunzi zapafupi zimatengedwa ngati zobwereza.
Mtunduwu umatha kupeza mwaluso ma spammers omwe amajambula chinthu chomwecho kuchokera kumakona osiyanasiyana kuti adutse kufananiza kwa pHash.
Chitsanzo cha zithunzi za sipamu zolumikizidwa pamodzi ndi neural network ngati zobwereza.
Pamapeto pake, malonda obwereza amafufuzidwa nthawi imodzi ndi zolemba ndi zithunzi.
Akapangidwa, magulu onse amadutsa mndandanda wa zosefera zokha. Fyuluta iliyonse imapereka zigoli ku gululo: ndizotheka bwanji kuti lili ndi chiwopsezo chomwe fyulutayi imadziwika.
Mwachitsanzo, makina amasanthula kufotokozera mu malonda ndikusankha magulu omwe angakhale nawo. Kenako zimatengera yemwe ali ndi kuthekera kwakukulu ndikufanizitsa ndi gulu lomwe lafotokozedwa ndi wolemba malonda. Ngati sizikufanana, malondawo amaletsedwa pagulu lolakwika. Ndipo popeza ndife okoma mtima komanso owona mtima, timauza wogwiritsa ntchito gulu lomwe akuyenera kusankha kuti malondawo adutse bwino.
Chidziwitso cha kuletsa kwa gulu lolakwika.
Kuphunzira pamakina kumamveka bwino papulatifomu yathu. Mwachitsanzo, ndi chithandizo chake timafufuza mayina ndi mafotokozedwe a katundu woletsedwa ku Russian Federation. Ndipo mitundu ya neural network mosamalitsa "imayang'ana" zithunzizo kuti muwone ngati zili ndi ma URL, mawu a spam, manambala a foni, ndi zidziwitso zomwezo "zoletsedwa".
Nthawi zomwe akufuna kugulitsa chinthu choletsedwa chobisika ngati chovomerezeka, ndipo palibe mawu mumutu kapena kufotokozera, timagwiritsa ntchito ma taging. Pa chithunzi chilichonse, mpaka 11 ma tag osiyanasiyana amatha kuwonjezeredwa omwe amafotokoza zomwe zili pachithunzichi.
Chidziwitso choti wogulitsa ali ndi chinthu chatsopano.
Zotsatira zake, malonda aliwonse "amachulukira" ndi metadata, ena amapangidwa pamene malonda apangidwa (adiresi ya IP ya wolemba, wogwiritsa ntchito, nsanja, geolocation, ndi zina zotero), ndipo zina zonse ndi mphambu zomwe zimaperekedwa ndi fyuluta iliyonse. .
Mwachitsanzo, mukhoza kupanga mzere wa malonda mu gulu la "Mafoni a M'manja" kuchokera kwa ogwiritsa ntchito a Yula omwe amati aku St. Petersburg, koma ma adilesi awo a IP akuchokera ku Moscow kapena mizinda ina.
Chitsanzo cha zotsatsa zotumizidwa ndi wogwiritsa ntchito m'mizinda yosiyanasiyana.
Ngati palibe zilengezo zatsopano zomwe zikufika pagululi, zimachotsedwa pamtima ndipo hashi yake ndi yankho lake zimalembedwa kwa Apache Cassandra.
Pulatifomu ikalandira kutsatsa kwatsopano, imayesa kaye kupeza gulu lofananira pakati pa omwe adapangidwa kale ndikutenga yankho kuchokera pamenepo. Ngati palibe gulu loterolo, nsanja imapita ku Cassandra ndikuyang'ana pamenepo. Mwaipeza? Zabwino, zimagwiritsa ntchito yankho ku tsango ndikutumiza ku Yula. Pali avareji ya zosankha βzobwerezedwaβ 70 tsiku lililonseβ8% ya chiwonkhetso.
Kufotokozera mwachidule
Takhala tikugwiritsa ntchito nsanja ya Odnoklassniki kwa zaka ziwiri ndi theka. Timakonda zotsatira:
Timangoyesa 94% ya zotsatsa zonse patsiku.
Mtengo wowongolera malonda udachepetsedwa kuchokera ku ma ruble 2 mpaka 7 kopecks.
Chifukwa cha chida chokonzekera, tinayiwala za mavuto oyang'anira oyang'anira.
Tidachulukitsa kuchuluka kwa zotsatsa zosinthidwa pamanja ndi nthawi 2,5 ndi owongolera ndi bajeti yofanana. Ubwino wa kuwongolera pamanja wakulanso chifukwa chowongolera zokha, komanso kusinthasintha pafupifupi 0,5% ya zolakwika.
Timaphimba mwachangu mitundu yatsopano ya sipamu ndi zosefera.
Timalumikiza mwachangu madipatimenti atsopano kuti azichita moyenera "Yula Verticals". Kuyambira 2017, Yula adawonjezera Real Estate, Vacancies and Auto verticals.