{"id":89206,"date":"2020-07-20T07:41:57","date_gmt":"2020-07-20T05:41:57","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh"},"modified":"2020-07-20T07:41:57","modified_gmt":"2020-07-20T05:41:57","slug":"chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","title":{"rendered":"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science? Grumbullimi i t\u00eb dh\u00ebnave","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\"><img decoding=\"async\" alt=\"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science? Grumbullimi i t\u00eb dh\u00ebnave\" src=\"\/wp-content\/uploads\/2020\/07\/49fb91fc3e45754e437b2c1fc4fec12f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\nSot \u00ebsht\u00eb 100500 kurse p\u00ebr Data Science dhe \u00ebsht\u00eb e njohur se m\u00eb shum\u00eb para n\u00eb Data Science mund t\u00eb fitohen pik\u00ebrisht nga kurset e Data Science (pse t\u00eb g\u00ebrmosh, kur mund t\u00eb shes\u00ebsh lopata?). Minus i madh i k\u00ebtyre kurseve \u00ebsht\u00eb se ato nuk kan\u00eb asnj\u00eb lidhje me pun\u00ebn reale: askush nuk do t'ju jap\u00eb t\u00eb dh\u00ebna t\u00eb pastra, t\u00eb p\u00ebrpunuara n\u00eb formatin e d\u00ebshiruar. Dhe kur dilni nga kurset dhe filloni t\u00eb zgjidhni nj\u00eb problem t\u00eb v\u00ebrtet\u00eb \u2014 dalin n\u00eb pah shum\u00eb nuanca.<\/p>\n<p>Prandaj fillojm\u00eb nj\u00eb seri sh\u00ebnimesh \"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science\", t\u00eb bazuara n\u00eb ngjarje reale q\u00eb m\u00eb kan\u00eb ndodhur mua, shok\u00ebve dhe koleg\u00ebve t\u00eb mi. Do shqyrtojm\u00eb me raste reale detyrat tipike t\u00eb Data Science: si ndodh n\u00eb t\u00eb v\u00ebrtet\u00eb. T\u00eb fillojm\u00eb sot me problemin e mbledhjes s\u00eb t\u00eb dh\u00ebnave.<\/p>\n<p>Dhe e para n\u00eb t\u00eb cil\u00ebn president\u00ebt hasin, filluan t\u00eb punojn\u00eb me t\u00eb dh\u00ebna reale \u2014 \u00ebsht\u00eb pik\u00ebrisht mbledhja e k\u00ebtyre t\u00eb dh\u00ebnave q\u00eb na interesojn\u00eb. Mesazhi kryesor i k\u00ebtij artikulli:<\/p>\n<blockquote><p><b>Ne sistematikisht n\u00ebnvler\u00ebsojm\u00eb koh\u00ebn, burimet dhe p\u00ebrpjekjet p\u00ebr mbledhjen, pastrimin dhe p\u00ebrgatitjen e t\u00eb dh\u00ebnave.<\/b><\/p><\/blockquote>\n<p>\nDhe m\u00eb kryesorja, do diskutojm\u00eb se \u00e7far\u00eb t\u00eb b\u00ebjm\u00eb q\u00eb t\u00eb mos e lejojm\u00eb k\u00ebt\u00eb.<\/p>\n<p>Sipas vler\u00ebsimeve t\u00eb ndryshme, pastrimi, transformimi, p\u00ebrpunimi i t\u00eb dh\u00ebnave, inxhinieria e ve\u00e7orive etj. z\u00ebn\u00eb 80-90% t\u00eb koh\u00ebs, dhe analiza 10-20%, nd\u00ebrsa praktiksht i gjith\u00eb materiali m\u00ebsimor p\u00ebrqendrohet ekskluzivisht n\u00eb analiz\u00eb.<\/p>\n<p>Le t\u00eb shqyrtojm\u00eb si nj\u00eb shembull tipik nj\u00eb detyr\u00eb analitike t\u00eb thjesht\u00eb n\u00eb tri variante dhe t\u00eb shohim se \u00e7far\u00eb jan\u00eb \"rrethanat njollos\u00ebse\".<\/p>\n<p>Dhe p\u00ebr shembull po ashtu, ne do t\u00eb shqyrtojm\u00eb variacione t\u00eb tilla t\u00eb detyr\u00ebs s\u00eb mbledhjes s\u00eb t\u00eb dh\u00ebnave dhe krahasimit t\u00eb komuniteteve p\u00ebr:<\/p>\n<ol>\n<li>Dy subreddite n\u00eb Reddit<\/li>\n<li>Dy seksione n\u00eb Habr<\/li>\n<li>Dy grupe n\u00eb Odnoklassniki<\/li>\n<\/ol>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Qasja kondicionale n\u00eb teori<\/h2>\n<p>\nHapni faqen dhe lexoni shembuj, n\u00ebse \u00ebsht\u00eb e qart\u00eb, planifikoni disa or\u00eb p\u00ebr leximin, disa or\u00eb p\u00ebr kodin sipas shembujve dhe debug. Shtoni disa or\u00eb p\u00ebr mbledhjen. Shtoni disa or\u00eb p\u00ebr rezerv\u00eb (shumoni me dy dhe shtoni N or\u00eb).<\/p>\n<p><b>Pika kryesore: vler\u00ebsimi temporal bazohet n\u00eb supozime dhe hamendje se sa do t\u00eb zgjas\u00eb.<\/b><\/p>\n<p>T\u00eb filloni analiz\u00ebn e koh\u00ebs \u00ebsht\u00eb e nevojshme q\u00eb t\u00eb b\u00ebni vler\u00ebsimin e parametrave t\u00eb m\u00ebposht\u00ebm p\u00ebr detyr\u00ebn e kushtezuar, t\u00eb p\u00ebrshkruar m\u00eb lart:<\/p>\n<ul>\n<li>Cila \u00ebsht\u00eb madh\u00ebsia e t\u00eb dh\u00ebnave dhe sa prej saj nevojitet fizikisht p\u00ebr t'u mbledhur (*shih m\u00eb posht\u00eb*).<\/li>\n<li>Cili \u00ebsht\u00eb koha e mbledhjes s\u00eb nj\u00eb regjistrimi dhe sa duhet t\u00eb prisni para se t\u00eb mund t\u00eb mbledhni t\u00eb dyt\u00ebn.<\/li>\n<li>T\u00eb krijoni nj\u00eb kod q\u00eb ruan gjendjen dhe fillon rinisjen, kur (e jo n\u00ebse) gjith\u00e7ka d\u00ebshton.<\/li>\n<li>T\u00eb kuptojm\u00eb n\u00ebse na nevojitet autentifikimi dhe t\u00eb parashikojm\u00eb koh\u00ebn p\u00ebr t\u00eb marr\u00eb qasje p\u00ebrmes API.<\/li>\n<li>T\u00eb parashikojm\u00eb numrin e gabimeve si funksion t\u00eb kompleksitetit t\u00eb t\u00eb dh\u00ebnave \u2013 t\u00eb vler\u00ebsojm\u00eb p\u00ebr nj\u00eb detyr\u00eb t\u00eb caktuar: struktura, sa transformime, \u00e7far\u00eb dhe si e nxjerrim.<\/li>\n<li>T\u00eb parashikojm\u00eb gabimet e rrjetit dhe problemet me sjellje jostandarde t\u00eb projektit.<\/li>\n<li>T\u00eb vler\u00ebsojm\u00eb n\u00ebse funksionet e nevojshme jan\u00eb n\u00eb dokumentacion dhe n\u00ebse jo, at\u00ebher\u00eb si dhe sa \u00ebsht\u00eb e nevojshme p\u00ebr nj\u00eb zgjidhje alternative.<\/li>\n<\/ul>\n<p>\nM\u00eb e r\u00ebnd\u00ebsishmja, p\u00ebr t\u00eb vler\u00ebsuar koh\u00ebn \u2013 ju faktikisht duhet t\u00eb shpenzoni koh\u00eb dhe p\u00ebrpjekje p\u00ebr \"zhbllokimin e situat\u00ebs\" \u2013 vet\u00ebm at\u00ebher\u00eb planifikimi juaj do t\u00eb jet\u00eb adekuat. Pra, pavar\u00ebsisht se si ju p\u00ebrpiqen t\u00eb thon\u00eb \"sa koh\u00eb nevojitet p\u00ebr mbledhjen e t\u00eb dh\u00ebnave\" \u2013 merrni koh\u00eb p\u00ebr nj\u00eb analiz\u00eb paraprake dhe argumentoni se sa koh\u00eb do t\u00eb vari\u00ebren n\u00eb var\u00ebsi t\u00eb parametrave real\u00eb t\u00eb detyr\u00ebs.<\/p>\n<p>Dhe tani do t\u00eb ilustrojm\u00eb shembuj konkret\u00eb, ku k\u00ebta parametra do t\u00eb ndryshojn\u00eb.<\/p>\n<p><b>Pika ky\u00e7e: vler\u00ebsimi bazohet n\u00eb analiz\u00ebn e faktor\u00ebve ky\u00e7 q\u00eb ndikojn\u00eb n\u00eb sasin\u00eb dhe kompleksitetin e pun\u00ebs.<\/b><\/p>\n<p>Vler\u00ebsimi, i bazuar n\u00eb supozime \u2013 \u00ebsht\u00eb nj\u00eb qasje e mir\u00eb kur element\u00ebt funksional\u00eb jan\u00eb mjaft t\u00eb vegj\u00ebl dhe nuk ka shum\u00eb faktor\u00eb q\u00eb mund t\u00eb ndikojn\u00eb duksh\u00ebm n\u00eb struktur\u00ebn e detyr\u00ebs. Por n\u00eb rastin e nj\u00eb s\u00ebr\u00eb detyrash n\u00eb Shkenc\u00ebn e t\u00eb Dh\u00ebnave, k\u00ebta faktor\u00eb b\u00ebhen jasht\u00ebzakonisht t\u00eb shumt\u00eb dhe kjo qasje b\u00ebhet e pap\u00ebrshtatshme.<\/p>\n<h2>Krahasimi i komuniteteve n\u00eb Reddit<\/h2>\n<p>\nT\u00eb fillojm\u00eb me rastin m\u00eb t\u00eb thjesht\u00eb (si\u00e7 do t\u00eb rezultoj\u00eb m\u00eb von\u00eb). N\u00eb t\u00eb v\u00ebrtet\u00eb, n\u00ebse jemi t\u00eb sinqert\u00eb, kemi p\u00ebrpara nj\u00eb rast praktikisht ideal, le t\u00eb kontrollojm\u00eb list\u00ebn ton\u00eb t\u00eb kontrollit p\u00ebr kompleksitetin:<\/p>\n<ul>\n<li>Ka nj\u00eb API t\u00eb qart\u00eb, t\u00eb kuptueshme dhe t\u00eb dokumentuar.<\/li>\n<li>Shum\u00eb e thjesht\u00eb dhe m\u00eb e r\u00ebnd\u00ebsishmja fitohet automatikisht nj\u00eb token.<\/li>\n<li>Ka <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/praw-dev\/praw\">python wrapper<\/a><\/noindex> \u2013 me nj\u00eb mori shembujsh.<\/li>\n<li>Komuniteti q\u00eb merret me analiz\u00ebn dhe mbledhjen e t\u00eb dh\u00ebnave n\u00eb Reddit (p\u00ebrfshir\u00eb deri n\u00eb video n\u00eb YouTube q\u00eb shpjegojn\u00eb se si t\u00eb p\u00ebrdorni python wrapper) <noindex><a rel=\"nofollow\" href=\"https:\/\/www.programcreek.com\/python\/example\/86599\/praw.Reddit\">ja p\u00ebr shembull<\/a><\/noindex>.<\/li>\n<li>Metodat q\u00eb na duhen ndoshta ekzistojn\u00eb n\u00eb API. P\u00ebr m\u00eb tep\u00ebr, kodi duket kompakt dhe i past\u00ebr, m\u00eb posht\u00eb \u00ebsht\u00eb nj\u00eb shembull i funksionit q\u00eb mbledh komentet p\u00ebr postimin.<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"python\">def get_comments(submission_id):\n    reddit = Reddit(check_for_updates=False, user_agent=AGENT)\n    submission = reddit.submission(id=submission_id)\n    more_comments = submission.comments.replace_more()\n    if more_comments:\n        skipped_comments = sum(x.count for x in more_comments)\n        logger.debug('Skipped %d MoreComments (%d comments)',\n                     len(more_comments), skipped_comments)\n    return submission.comments.list()\n<\/code><\/pre>\n<p><i>Marr\u00eb nga <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/timendum\/reddit-utilities\/blob\/master\/thread-export.py\">k\u00ebt\u00eb<\/a><\/noindex> p\u00ebrmbledhja e mjeteve t\u00eb p\u00ebrshtatshme p\u00ebr paketimin.<\/i><\/p>\n<p>Megjith\u00ebse kemi rastin m\u00eb t\u00eb mir\u00eb p\u00ebrpara nesh, megjithat\u00eb \u00ebsht\u00eb e nevojshme t\u00eb merret parasysh nj\u00eb s\u00ebr\u00eb faktor\u00ebsh t\u00eb r\u00ebnd\u00ebsish\u00ebm nga jeta reale:<\/p>\n<ul>\n<li>Kufijt\u00eb e API-s\u00eb \u2014 jemi t\u00eb detyruar t\u00eb marrim t\u00eb dh\u00ebnat n\u00eb grupe (t\u00eb presim midis k\u00ebrkesave, etj.).<\/li>\n<li>Koha e mbledhjes \u2014 p\u00ebr nj\u00eb analiz\u00eb t\u00eb plot\u00eb dhe krahasim, do t\u00eb duhen r\u00ebnd\u00ebsisht koh\u00eb vet\u00ebm p\u00ebr k\u00ebrkuesi p\u00ebr t\u00eb kaluar n\u00ebp\u00ebr subredit.<\/li>\n<li>Robot\u00ebt duhet t\u00eb funksionojn\u00eb n\u00eb server \u2014 nuk mund ta filloni thjesht n\u00eb laptop, ta vendosni n\u00eb \u00e7ant\u00eb dhe t\u00eb shkoni p\u00ebr biznes. Prandaj, un\u00eb kam filluar gjith\u00e7ka n\u00eb VPS. Me kodin e zbritjes habrahabr10 mund t\u00eb kurseni edhe 10% t\u00eb \u00e7mimit.<\/li>\n<li>Pamund\u00ebsia fizike p\u00ebr t\u00eb aksesuar disa t\u00eb dh\u00ebna (ato jan\u00eb t\u00eb dukshme p\u00ebr administratoret ose jan\u00eb shum\u00eb t\u00eb v\u00ebshtira p\u00ebr t'u mbledhur) \u2014 duhet ta marrim parasysh, jo t\u00eb gjitha t\u00eb dh\u00ebnat mund t\u00eb mblidhen brenda nj\u00eb kohe t\u00eb arsyeshme.<\/li>\n<li>Gabimet e pun\u00ebs n\u00eb rrjet: punimi me rrjetin \u00ebsht\u00eb e dhimbshme.<\/li>\n<li>K\u00ebto jan\u00eb t\u00eb dh\u00ebna reale \u2014 ato nuk jan\u00eb kurr\u00eb t\u00eb pastra.<\/li>\n<\/ul>\n<p>\nSigurisht, \u00ebsht\u00eb e nevojshme t\u00eb merret parasysh k\u00ebto nuanca n\u00eb zhvillim. Or\u00ebt\/ditat e sakta varen nga p\u00ebrvoja e zhvillimit ose p\u00ebrvoja e pun\u00ebs me detyra t\u00eb ngjashme, megjithat\u00eb ne shohim se k\u00ebtu detyra \u00ebsht\u00eb ekskluzivisht inxhinierike dhe nuk k\u00ebrkon l\u00ebvizje t\u00eb tjera shtes\u00eb p\u00ebr zgjidhje \u2014 gjith\u00e7ka mund t\u00eb vler\u00ebsohet, t\u00eb p\u00ebrshkruhet dhe t\u00eb b\u00ebhet shum\u00eb mir\u00eb.<\/p>\n<h2>Krahasimi i seksioneve t\u00eb Habr\u00ebs<\/h2>\n<p>\nKaloim n\u00eb nj\u00eb rast m\u00eb interesant dhe jo trivial, duke krahasuar rrjedhat dhe\/ose seksionet e Habr\u00ebs.<\/p>\n<p>Le t\u00eb kontrollojm\u00eb list\u00ebn ton\u00eb t\u00eb komplikiave \u2014 k\u00ebtu, p\u00ebr t\u00eb kuptuar \u00e7do pik\u00eb, do t\u00eb duhet t\u00eb eksperimentoni pak me detyr\u00ebn dhe t\u00eb provoni.<\/p>\n<ul>\n<li>Fillimisht mendoni se ka nj\u00eb API, por nuk ka. Po, Habra ka API, por ai nuk \u00ebsht\u00eb n\u00eb dispozicion p\u00ebr p\u00ebrdoruesit (ndoshta as nuk funksionon).<\/li>\n<li>Pastaj thjesht filloni t\u00eb parse html \u2014 \u00abimport requests\u00bb, \u00e7far\u00eb mund t\u00eb shkoj\u00eb keq?<\/li>\n<li>Dhe si mund t\u00eb parse? Qasja m\u00eb e thjesht\u00eb dhe e p\u00ebrdorur shpesh \u00ebsht\u00eb t\u00eb iteroni mbi ID, do t\u00eb theksojm\u00eb se nuk \u00ebsht\u00eb metoda m\u00eb efektive dhe do t\u00eb duhen t\u00eb p\u00ebrpunohen raste t\u00eb ndryshme \u2014 p\u00ebr shembull, densiteti i ID-ve reale n\u00eb mesin e t\u00eb gjith\u00eb atyre q\u00eb ekzistojn\u00eb.\n<p><img decoding=\"async\" alt=\"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science? Grumbullimi i t\u00eb dh\u00ebnave\" src=\"\/wp-content\/uploads\/2020\/07\/2c67fc14f672e92979edd617314210da.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Marr\u00eb nga <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/218607\/\">k\u00ebt\u00eb<\/a><\/noindex> artikuj.<\/i><\/li>\n<li>T\u00eb dh\u00ebnat e pap\u00ebrpunuara, t\u00eb mb\u00ebshtjella n\u00eb HTML sip\u00ebr rrjetit \u2014 jan\u00eb nj\u00eb dhimbje. P\u00ebr shembull, d\u00ebshironi t\u00eb mbledhni dhe ruani vler\u00ebsimin e artikullit: e nxorr\u00ebt score nga html dhe vendos\u00ebt ta ruani si num\u00ebr p\u00ebr p\u00ebrpunim t\u00eb m\u00ebtejsh\u00ebm:\u00a0\n<p>1) int(score) shkakton nj\u00eb gabim: pasi n\u00eb Habr\u00eb minus, si n\u00eb p\u00ebr shembull n\u00eb rreshtin \"\u20135\" \u2014 kjo \u00ebsht\u00eb nj\u00eb tir\u00eb e shkurt\u00ebr, jo shenj\u00eb minus (papritur, apo jo?), prandaj nj\u00eb moment duhet t\u00eb ngrejm\u00eb parsimin n\u00eb jet\u00eb me k\u00ebt\u00eb p\u00ebrmir\u00ebsim t\u00eb tmerrsh\u00ebm.<\/p>\n<pre><code class=\"python\">provoni:\n      score_txt = post.find(class_= \"score\").text.replace(u\"\u2013\", \"-\").replace(u\"+\", \"+\")\n      score = int(score_txt)\n      n\u00ebse kontrollohet_data(data):\n        post_score += score\n<\/code><\/pre>\n<p>\nDatat, pluset dhe minuset mund t\u00eb mos ekzistojn\u00eb fare (si\u00e7 e shohim m\u00eb lart n\u00eb funksionin check_date, dhe k\u00ebshtu ka ndodhur).<\/p>\n<p>2) Simbolet speciale t\u00eb paekranuara \u2014 ato do t\u00eb vijn\u00eb, duhet t\u00eb jeni gati.<\/p>\n<p>3) Strukturat ndryshojn\u00eb var\u00ebsisht nga lloji i postit.<\/p>\n<p>4) Postet e vjetra mund t\u00eb ken\u00eb **struktur\u00eb t\u00eb \u00e7uditshme**.<\/li>\n<li>N\u00eb thelb, procesi i p\u00ebrpunimit t\u00eb gabimeve dhe \u00e7far\u00eb mund ose s'mund t\u00eb ndodh\u00eb do t\u00eb duhet t\u00eb p\u00ebrballeni dhe nuk mund t\u00eb parashikohet me siguri se \u00e7far\u00eb do t\u00eb shkoj\u00eb keq dhe si mund t\u00eb jet\u00eb ndryshe struktura dhe se \u00e7far\u00eb do t\u00eb bjer\u00eb \u2014 ndoshta do t'ju duhet t\u00eb provoni dhe t\u00eb merrni parasysh gabimet q\u00eb jep parser-i.<\/li>\n<li>M\u00eb pas kuptoni se duhet t\u00eb parse n\u00eb disa rrjedha n\u00eb t\u00eb kund\u00ebrt, procesi n\u00eb nj\u00eb t\u00eb vetme do t\u00eb zgjas\u00eb 30+ or\u00eb (ky \u00ebsht\u00eb thjesht koha e ekzekutimit t\u00eb parser-it t\u00eb vet\u00ebm q\u00eb fle dhe nuk bie n\u00ebn asnj\u00eb ndalese). N\u00eb <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/276383\/\">k\u00ebt\u00eb<\/a><\/noindex> artikull, kjo solli nj\u00eb moment t\u00eb ngjash\u00ebm n\u00eb skem\u00ebn e till\u00eb:<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science? Grumbullimi i t\u00eb dh\u00ebnave\" src=\"\/wp-content\/uploads\/2020\/07\/9ad1853f3ee2c9321ee9f1a5e3efea7d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nK\u00ebshtu, lista e kontrollit p\u00ebr v\u00ebshtir\u00ebsin\u00eb:<\/p>\n<ul>\n<li>Puna me rrjetin dhe parse HTML me iterim dhe kalim sipas ID.<\/li>\n<li>Dokumentet kan\u00eb struktur\u00eb t\u00eb heterogjene.<\/li>\n<li>Ka shum\u00eb vende ku kodi mund t\u00eb d\u00ebshtoj\u00eb leht\u00ebsisht.<\/li>\n<li>Nevojitet t\u00eb shkruhet || kodi.<\/li>\n<li>Mungojn\u00eb dokumentacioni i nevojsh\u00ebm, shembujt e kodit dhe\\\/ose komuniteti.<\/li>\n<\/ul>\n<p>\nVler\u00ebsimi kushtetues i koh\u00ebs p\u00ebr k\u00ebt\u00eb detyr\u00eb do t\u00eb jet\u00eb 3-5 her\u00eb m\u00eb i lart\u00eb, sesa p\u00ebr mbledhjen e t\u00eb dh\u00ebnave nga Reddit.<\/p>\n<h2>Krahasimi i grupeve n\u00eb Odnoklassniki<\/h2>\n<p>\nT\u00eb kalojm\u00eb n\u00eb rastin m\u00eb teknologjikisht t\u00eb interesant p\u00ebrve\u00e7 atyre q\u00eb jan\u00eb p\u00ebrmendur. P\u00ebr mua, ai ishte interesant pik\u00ebrisht p\u00ebr faktin se n\u00eb dukje duket mjaft trivial, por nuk \u00ebsht\u00eb krejt\u00ebsisht i till\u00eb \u2014 sapo t\u00eb b\u00ebni nj\u00eb prekje n\u00eb t\u00eb.<\/p>\n<p>Do t\u00eb fillojm\u00eb me list\u00ebn ton\u00eb t\u00eb kontrollit p\u00ebr v\u00ebshtir\u00ebsin\u00eb dhe do t\u00eb theksojm\u00eb se shum\u00eb nga ato do t\u00eb rezultojn\u00eb m\u00eb t\u00eb komplikuara se si duken n\u00eb fillim:<\/p>\n<ul>\n<li>API ekziston, por i mungojn\u00eb pothuajse t\u00eb gjitha funksionet e nevojshme.<\/li>\n<li>P\u00ebr disa funksione duhet t\u00eb k\u00ebrkoni akses me email, dometh\u00ebn\u00eb dh\u00ebnia e aksesit nuk \u00ebsht\u00eb e menj\u00ebhershme.<\/li>\n<li>Dokumentimi \u00ebsht\u00eb jasht\u00ebzakonisht i dob\u00ebt (t\u00eb fillojm\u00eb me faktin se terma rus\u00eb dhe anglishte p\u00ebrzihen kudo, n\u00eb nj\u00eb m\u00ebnyr\u00eb krejt\u00ebsisht t\u00eb paparashikueshme \u2014 ndonj\u00ebher\u00eb duhet thjesht t\u00eb gjesh se \u00e7far\u00eb po k\u00ebrkohet prej teje diku) <noindex><a rel=\"nofollow\" href=\"https:\/\/apiok.ru\/dev\/methods\/rest\/discussions\/discussions.getComments\">funksioni q\u00eb na nevojitet<\/a><\/noindex>.<\/li>\n<li>K\u00ebrkon seanc\u00eb n\u00eb dokumentacion, nd\u00ebrsa n\u00eb praktik\u00eb nuk e p\u00ebrdor at\u00eb \u2014 dhe nuk ka asnj\u00eb m\u00ebnyr\u00eb p\u00ebr t\u00eb kuptuar t\u00eb gjitha nuancat e mod\u00ebs s\u00eb API, p\u00ebrve\u00e7 t\u00eb provohet dhe shpresohet q\u00eb di\u00e7ka do t\u00eb funksionoj\u00eb.<\/li>\n<li>Mungojn\u00eb shembuj dhe komuniteti, pika e vetme e mb\u00ebshtetjes p\u00ebr mbledhjen e informacionit \u00ebsht\u00eb nj\u00eb <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/alternativshik\/python-odnoklassniki\">wrapper<\/a><\/noindex> n\u00eb Python (pa shum\u00eb shembuj p\u00ebrdorimi).<\/li>\n<li>M\u00ebnyra m\u00eb funksionale duket t\u00eb jet\u00eb Selenium, pasi shum\u00eb t\u00eb dh\u00ebna t\u00eb nevojshme jan\u00eb n\u00ebn mbrojtje.<br \/>\n1) Pra, ndodh autorizimi p\u00ebrmes nj\u00eb p\u00ebrdoruesi t\u00eb rrem\u00eb (dhe regjistrimi manual).<\/p>\n<p>2) Megjithat\u00eb, me Selenium nuk ka garantira p\u00ebr funksionimin e sakt\u00eb dhe t\u00eb p\u00ebrs\u00ebritsh\u00ebm (sigurisht n\u00eb rastin me ok.ru).<\/p>\n<p>3) Site Ok.ru p\u00ebrmban gabime JavaScript dhe ndonj\u00ebher\u00eb sillet \u00e7uditsh\u00ebm dhe pa nj\u00eb q\u00ebndrim t\u00eb qart\u00eb.<\/p>\n<p>4) Duhet t\u00eb merremi me paginimin, ngarkimin e elementeve, etj...<\/p>\n<p>5) Gabimet e API q\u00eb jep wrapper-it do t\u00eb duhet t\u00eb p\u00ebrpunohen n\u00eb m\u00ebnyr\u00eb t\u00eb p\u00ebrpikt\u00eb, p\u00ebr shembull, k\u00ebshtu (nj\u00eb cop\u00eb kod eksperimental):<\/p>\n<pre><code class=\"python\">def merr_komentet(args, kontekst, diskutime):\n    pauz\u00eb = 1\n    n\u00ebse args.extract_comments:\n        t\u00eb gjitha_komentet = set()\n#ka kuptim t\u00eb mbash gjurm\u00ebt e diskutimeve t\u00eb p\u00ebrpunuara tashm\u00eb\n        p\u00ebr diskutim n\u00eb tqdm(diskutime): \n            provoni:\n                komentet = merr_komentet_nga_diskutimi_via_api(kontekst, diskutim)\n            p\u00ebrjashto odnoklassniki.api.OdnoklassnikiError si e:\n                n\u00ebse \"NOT_FOUND\" n\u00eb str(e):\n                    komentet = set()\n                p\u00ebrndryshe:\n                    print(e)\n                    bp()\n                    kaloni\n            t\u00eb gjitha_komentet |= komentet\n            time.sleep(pauz\u00eb)\n        kthe t\u00eb gjitha_komentet\n<\/code><\/pre>\n<p>\nGabimi im i preferuar ishte:<\/p>\n<p><code>OdnoklassnikiError(\"Gabim(kodi: 'None', p\u00ebrshkrimi: 'gabim HTTP', metoda: 'discussions.getComments', parametrat: \u2026)\")<\/code><\/p>\n<p>6) N\u00eb fund, mund\u00ebsia Selenium + API duket t\u00eb jet\u00eb mund\u00ebsia m\u00eb e arsyeshme.<\/li>\n<li>Nevoja p\u00ebr ruajtjen e gjendjes dhe rikthimin e sistemit, p\u00ebrpunimi i shum\u00eb gabimeve, p\u00ebrfshir\u00eb sjelljen e paq\u00ebndrueshme t\u00eb site-it \u2014 p\u00ebr m\u00eb tep\u00ebr, k\u00ebto gabime jan\u00eb mjaft t\u00eb v\u00ebshtira p\u00ebr t'u imagjinuar (n\u00ebse nuk jeni profesionist n\u00eb shkruarjen e parser\u00ebve, natyrisht).<\/li>\n<\/ul>\n<p>\nVler\u00ebsimi i kusht\u00ebzuar i koh\u00ebs p\u00ebr k\u00ebt\u00eb detyr\u00eb do t\u00eb jet\u00eb nga 3-5 her\u00eb m\u00eb i lart\u00eb se sa p\u00ebr mbledhjen e t\u00eb dh\u00ebnave nga Habra. Pavar\u00ebsisht se n\u00eb rastin e Habra ne p\u00ebrdorim nj\u00eb qasje ballore me parsing t\u00eb HTML-s\u00eb, n\u00eb rastin e OK ne mund t\u00eb punojm\u00eb me API n\u00eb vendet kritike.<\/p>\n<h2>P\u00ebrfundimet<\/h2>\n<p>\nPavar\u00ebsisht se si mund t\u00eb k\u00ebrkohet vler\u00ebsimi i afateve \"n\u00eb vend\" (ne kemi planifikim sot!), volumi i modulit t\u00eb harduerit t\u00eb procesit t\u00eb t\u00eb dh\u00ebnave \u00ebsht\u00eb praktikisht e pamundur t\u00eb vler\u00ebsohet cil\u00ebsisht pa analizuar parametrat e projektit. <\/p>\n<p>N\u00ebse flasim pak m\u00eb filozofikisht, strategjit\u00eb e vler\u00ebsimit n\u00eb agile i p\u00ebrshtaten mir\u00eb detyrave inxhinierike, por me detyra m\u00eb eksperimentale dhe, n\u00eb nj\u00eb far\u00eb m\u00ebnyre, \"kreative\" dhe k\u00ebrkuese, pra, m\u00eb pak t\u00eb parashikueshme, lindin v\u00ebshtir\u00ebsi, si n\u00eb shembujt q\u00eb analizuam k\u00ebtu. <\/p>\n<p>Sigurisht, mbledhja e t\u00eb dh\u00ebnave \u00ebsht\u00eb nj\u00eb shembull ilustrativ \u2014 zakonisht kjo detyr\u00eb duket jasht\u00ebzakonisht e leht\u00eb dhe teknikisht jo e komplikuar, dhe pik\u00ebrisht n\u00eb detaje k\u00ebtu shpesh fshihet djalli. Dhe pik\u00ebrisht n\u00eb k\u00ebt\u00eb detyr\u00eb \u00ebsht\u00eb e mundur t\u00eb tregohet e gjith\u00eb spektri i mund\u00ebsive se \u00e7far\u00eb mund t\u00eb shkoj\u00eb keq dhe sa shum\u00eb mund t\u00eb zgjatet puna. <\/p>\n<p>N\u00ebse shikoni nj\u00eb her\u00eb p\u00ebr t\u00eb par\u00eb karakteristikat e detyr\u00ebs pa eksperimente t\u00eb tjera, Reddit dhe OK duken t\u00eb ngjashme: jan\u00eb me API, py wrapper, por n\u00eb thelb, ndryshimi \u00ebsht\u00eb gjigand. N\u00ebse gjykohet sipas k\u00ebtyre parametrave, parser-i i Habr\u00ebs duket m\u00eb i komplikuar se OK \u2014 nd\u00ebrsa n\u00eb praktik\u00eb \u00ebsht\u00eb krejt\u00ebsisht e kund\u00ebrta dhe pik\u00ebrisht k\u00ebt\u00eb mund ta zbulojm\u00eb duke kryer eksperimente t\u00eb thjeshta p\u00ebr analiz\u00ebn e parametrave t\u00eb detyr\u00ebs.<\/p>\n<p>Nga p\u00ebrvoja ime, qasja m\u00eb efektive \u00ebsht\u00eb nj\u00eb vler\u00ebsim i p\u00ebraf\u00ebrt i koh\u00ebs q\u00eb do t'ju nevojitet p\u00ebr analiz\u00ebn preliminare dhe eksperimentet e para t\u00eb thjeshta, leximi i dokumentacionit \u2014 ato do t'ju lejojn\u00eb t\u00eb jepni nj\u00eb vler\u00ebsim t\u00eb sakt\u00eb p\u00ebr t\u00eb gjith\u00eb pun\u00ebn. Me terma t\u00eb metodologjis\u00eb popullore agile \u2014 k\u00ebrkoj t\u00eb krijohet nj\u00eb bilet\u00eb p\u00ebr \"vler\u00ebsimin e parametrave t\u00eb detyr\u00ebs\", mbi baz\u00ebn e s\u00eb cil\u00ebs mund t\u00eb jap nj\u00eb vler\u00ebsim p\u00ebr at\u00eb q\u00eb \u00ebsht\u00eb e mundur t\u00eb realizohet brenda \"sprintit\" dhe t\u00eb jap nj\u00eb vler\u00ebsim m\u00eb t\u00eb sakt\u00eb p\u00ebr secil\u00ebn detyr\u00eb.<\/p>\n<p>Prandaj duket se argumenti m\u00eb efektiv \u00ebsht\u00eb ai q\u00eb do t\u00eb tregonte \"specialistit jo teknik\" sa shum\u00eb do t\u00eb varioj\u00eb koha dhe burimet n\u00eb var\u00ebsi t\u00eb parametrave q\u00eb ende duhet t\u00eb vler\u00ebsohen.<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"http:\/\/ruvds.com\/ru-rub?utm_source=habr&amp;utm_medium=article&amp;utm_campaign=param&amp;utm_content=datascience1#order\"><img decoding=\"async\" alt=\"\u00c7far\u00eb mund t\u00eb shkoj\u00eb keq me Data Science? Grumbullimi i t\u00eb dh\u00ebnave\" src=\"\/wp-content\/uploads\/2020\/07\/2ca567a078c8500d37dfcf982e76252c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<br \/>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u0435\u0442 100500 \u043a\u0443\u0440\u0441\u043e\u0432 \u043f\u043e Data Science \u0438 \u0434\u0430\u0432\u043d\u043e \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e, \u0447\u0442\u043e \u0431\u043e\u043b\u044c\u0448\u0435 \u0432\u0441\u0435\u0433\u043e \u0434\u0435\u043d\u0435\u0433 \u0432 Data Science \u043c\u043e\u0436\u043d\u043e \u0437\u0430\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u043d\u043e \u043a\u0443\u0440\u0441\u0430\u043c\u0438 \u043f\u043e Data Science (\u0437\u0430\u0447\u0435\u043c \u043a\u043e\u043f\u0430\u0442\u044c, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0434\u0430\u0432\u0430\u0442\u044c \u043b\u043e\u043f\u0430\u0442\u044b?). \u041e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043c\u0438\u043d\u0443\u0441 \u044d\u0442\u0438\u0445 \u043a\u0443\u0440\u0441\u043e\u0432 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u043e\u043d\u0438 \u043d\u0435 \u0438\u043c\u0435\u044e\u0442 \u043d\u0438\u0447\u0435\u0433\u043e \u043e\u0431\u0449\u0435\u0433\u043e \u0441 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u043e\u0439: \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0434\u0430\u0441\u0442 \u0432\u0430\u043c \u0447\u0438\u0441\u0442\u044b\u0435, \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u0430\u043d\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0432 \u043d\u0443\u0436\u043d\u043e\u043c \u0444\u043e\u0440\u043c\u0430\u0442\u0435. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":89207,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-89206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47\u00c7far\u00eb mund t\u00eb shkoj\u00eb gabim me Data Science? Mbledhja e t\u00eb dh\u00ebnave | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-20T05:41:57+00:00","article:modified_time":"2020-07-20T05:41:57+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"89206","title":null,"description":null,"keywords":null,"keyphrases":{"focus":[],"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:51:09","updated":"2026-08-11 12:50:12","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/89206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=89206"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/89206\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/89207"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=89206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=89206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=89206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}