{"id":89206,"date":"2020-07-20T07:41:57","date_gmt":"2020-07-20T05:41:57","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh"},"modified":"2020-07-20T07:41:57","modified_gmt":"2020-07-20T05:41:57","slug":"chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","title":{"rendered":"\u010co m\u00f4\u017ee pokazi\u0165 Data Science? Zber d\u00e1t","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\"><img decoding=\"async\" alt=\"\u010co m\u00f4\u017ee pokazi\u0165 Data Science? Zber d\u00e1t\" src=\"\/wp-content\/uploads\/2020\/07\/49fb91fc3e45754e437b2c1fc4fec12f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\nDnes existuje mno\u017estvo kurzov Data Science a u\u017e d\u00e1vno je zn\u00e1me, \u017ee najviac pe\u0148az\u00ed v Data Science mo\u017eno zarobi\u0165 pr\u00e1ve predajom kurzov zameran\u00fdch na Data Science (pre\u010do kopa\u0165, ke\u010f m\u00f4\u017eete pred\u00e1va\u0165 lopaty?). Hlavn\u00fdm nedostatkom t\u00fdchto kurzov je, \u017ee nemaj\u00fa ni\u010d spolo\u010dn\u00e9 so skuto\u010dnou pr\u00e1cou: nikto v\u00e1m ned\u00e1 \u010dist\u00e9, spracovan\u00e9 d\u00e1ta v potrebnom form\u00e1te. A ke\u010f vyjdete z kurzov a za\u010dnete rie\u0161i\u0165 skuto\u010dn\u00fd probl\u00e9m, objav\u00ed sa ve\u013ea nu\u00e1ns.<\/p>\n<p>Preto za\u010d\u00edname s\u00e9riu \u010dl\u00e1nkov \u201e\u010co sa m\u00f4\u017ee pokazi\u0165 pri Data Science\u201c, zalo\u017een\u00fdch na skuto\u010dn\u00fdch udalostiach, ktor\u00e9 sa stali mne, mojim priate\u013eom a kolegom. Budeme rozobera\u0165 na re\u00e1lnych pr\u00edkladoch typick\u00e9 \u00falohy v Data Science: ako to naozaj prebieha. Dnes za\u010dneme \u00falohou zberu d\u00e1t.<\/p>\n<p>A prv\u00e9, o \u010do sa \u013eudia pri pr\u00e1ci so skuto\u010dn\u00fdmi d\u00e1tami zakopn\u00fa, je samotn\u00fd zber t\u00fdchto relevantn\u00fdch d\u00e1t. K\u013e\u00fa\u010dov\u00fd posolstvo tohto \u010dl\u00e1nku:<\/p>\n<blockquote><p><b>Syst\u00e9movo podce\u0148ujeme \u010das, zdroje a \u00fasilie potrebn\u00e9 na zber, \u010distenie a pr\u00edpravu d\u00e1t.<\/b><\/p><\/blockquote>\n<p>\nA najd\u00f4le\u017eitej\u0161ie, prediskutujeme, \u010do robi\u0165, aby sme tomu predi\u0161li.<\/p>\n<p>Pod\u013ea r\u00f4znych odhadov, \u010distenie, transform\u00e1cia, spracovanie d\u00e1t, in\u017einiering funkci\u00ed at\u010f. zaberaj\u00fa 80-90 % \u010dasu, k\u00fdm anal\u00fdza 10-20 %, pri\u010dom takmer cel\u00fd v\u00fdukov\u00fd materi\u00e1l sa s\u00fastre\u010fuje v\u00fdhradne na anal\u00fdzu.<\/p>\n<p>Po\u010fme si rozobra\u0165 jednoduch\u00fa analytick\u00fa \u00falohu v troch variantoch a uvid\u00edme, ak\u00e9 m\u00f4\u017eu by\u0165 \u201eznepokojuj\u00face okolnosti\u201c.<\/p>\n<p>A op\u00e4\u0165 ako pr\u00edklad, zv\u00e1\u017eime podobn\u00e9 vari\u00e1cie \u00falohy zberu d\u00e1t a porovnania komun\u00edt pre:<\/p>\n<ol>\n<li>Dve subredditov\u00e9 komunity Reddit<\/li>\n<li>Dve sekcie Habr<\/li>\n<li>Dve skupiny na Odnoklassniki<\/li>\n<\/ol>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Podmiene\u010dn\u00fd pr\u00edstup v te\u00f3rii<\/h2>\n<p>\nOtvor\u00edte str\u00e1nku a pre\u010d\u00edtate si pr\u00edklady, ak je to zrozumite\u013en\u00e9, vyhra\u010fte si nieko\u013eko hod\u00edn na \u010d\u00edtanie, nieko\u013eko hod\u00edn na k\u00f3d pod\u013ea pr\u00edkladov a ladenie. Prida\u0165 nieko\u013eko hod\u00edn na zber. Prida\u0165 nieko\u013eko hod\u00edn pre rezervu (n\u00e1sobi\u0165 dvoma a prida\u0165 N hod\u00edn).<\/p>\n<p><b>K\u013e\u00fa\u010dov\u00fd moment: \u010dasov\u00fd odhad je zalo\u017een\u00fd na predpokladoch a domnienkach o tom, ko\u013eko to bude trva\u0165.<\/b><\/p>\n<p>Za\u010da\u0165 anal\u00fdzu \u010dasu je potrebn\u00e9 s hodnoten\u00edm nasleduj\u00facich parametrov pre podmiene\u010dn\u00fa \u00falohu, op\u00edsan\u00fa vy\u0161\u0161ie:<\/p>\n<ul>\n<li>Ak\u00e1 je ve\u013ekos\u0165 d\u00e1t a ko\u013eko sa ich mus\u00ed fyzicky zbiera\u0165 (*pozri ni\u017e\u0161ie*).<\/li>\n<li>Ak\u00fd \u010das trv\u00e1 zber jednej polo\u017eky a ko\u013eko treba \u010daka\u0165, ne\u017e sa d\u00e1 z\u00edska\u0165 druh\u00e1.<\/li>\n<li>Zavies\u0165 p\u00edsanie k\u00f3du, ktor\u00fd uchov\u00e1va stav a za\u010d\u00edna re\u0161tart, ke\u010f (a nie ak) v\u0161etko zlyh\u00e1.<\/li>\n<li>Zisti\u0165, \u010di potrebujeme autoriz\u00e1ciu a zoh\u013eadni\u0165 \u010das potrebn\u00fd na z\u00edskanie pr\u00edstupu cez API.<\/li>\n<li>Zoh\u013eadni\u0165 po\u010det ch\u00fdb ako funkciu zlo\u017eitosti d\u00e1t \u2013 pos\u00fadi\u0165 na z\u00e1klade konkr\u00e9tnej \u00falohy: \u0161trukt\u00fara, ko\u013eko transform\u00e1ci\u00ed, \u010do a ako extrahujeme.<\/li>\n<li>Zoh\u013eadni\u0165 chyby siete a probl\u00e9my s netradi\u010dn\u00fdm spr\u00e1van\u00edm projektu.<\/li>\n<li>Pos\u00fadi\u0165, \u010di s\u00fa potrebn\u00e9 funkcie v dokument\u00e1cii, a ak nie, tak ako a ko\u013eko potrebujeme na workaround.<\/li>\n<\/ul>\n<p>\nNajd\u00f4le\u017eitej\u0161ie je, \u017ee na pos\u00fadenie \u010dasu \u2013 skuto\u010dne mus\u00edte str\u00e1vi\u0165 \u010das a \u00fasilie na \u201eprieskum ter\u00e9nu\u201c \u2013 len tak bude va\u0161e pl\u00e1novanie adekv\u00e1tne. Preto, nech v\u00e1s akoko\u013evek tla\u010dia poveda\u0165 \u201eko\u013eko \u010dasu treba na zber d\u00e1t\u201c \u2013 vy\u017eiadajte si \u010das na predbe\u017en\u00fa anal\u00fdzu a od\u00f4vodnite to t\u00fdm, ak\u00e9 sa bud\u00fa \u010dasov\u00e9 odhady l\u00ed\u0161i\u0165 v z\u00e1vislosti od skuto\u010dn\u00fdch parametrov \u00falohy.<\/p>\n<p>A teraz predvedieme konkr\u00e9tne pr\u00edklady, kde sa tak\u00e9to parametre bud\u00fa meni\u0165.<\/p>\n<p><b>K\u013e\u00fa\u010dov\u00fd bod: pos\u00fadenie je zalo\u017een\u00e9 na anal\u00fdze k\u013e\u00fa\u010dov\u00fdch faktorov, ktor\u00e9 ovplyv\u0148uj\u00fa objem a zlo\u017eitost pr\u00e1ce.<\/b><\/p>\n<p>Pos\u00fadenie zalo\u017een\u00e9 na domnienkach \u2013 to je dobr\u00fd pr\u00edstup, ke\u010f s\u00fa funk\u010dn\u00e9 prvky dostato\u010dne mal\u00e9 a nie je ve\u013ea faktorov, ktor\u00e9 m\u00f4\u017eu v\u00fdznamne ovplyvni\u0165 \u0161trukt\u00faru \u00falohy. Ale v pr\u00edpade niektor\u00fdch \u00faloh Data Science je t\u00fdchto faktorov ve\u013emi ve\u013ea a tak\u00fdto pr\u00edstup sa st\u00e1va neadekv\u00e1tnym.<\/p>\n<h2>Porovnanie komun\u00edt Reddit<\/h2>\n<p>\nZa\u010dneme s najjednoduch\u0161\u00edm pr\u00edpadom (ako sa uk\u00e1\u017ee nesk\u00f4r). V skuto\u010dnosti, ak m\u00e1me by\u0165 \u00faprimn\u00ed, ide o takmer ide\u00e1lny pr\u00edpad; skontrolujeme n\u00e1\u0161 checklist zlo\u017eitosti:<\/p>\n<ul>\n<li>Existuje upraven\u00e9, jasne ozna\u010den\u00e9 a zdokumentovan\u00e9 API.<\/li>\n<li>Ve\u013emi jednoducho a hlavne automaticky sa z\u00edska token.<\/li>\n<li>Existuje <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/praw-dev\/praw\">python wrapper<\/a><\/noindex> \u2013 s mno\u017estvom pr\u00edkladov.<\/li>\n<li>Komunita, ktor\u00e1 sa zaober\u00e1 anal\u00fdzou a zbieran\u00edm d\u00e1t na Reddite (a\u017e po youtube vide\u00e1 vysvet\u013euj\u00face, ako pou\u017e\u00edva\u0165 python wrapper) <noindex><a rel=\"nofollow\" href=\"https:\/\/www.programcreek.com\/python\/example\/86599\/praw.Reddit\">tu je napr\u00edklad<\/a><\/noindex>.<\/li>\n<li>Potrebn\u00e9 met\u00f3dy pravdepodobne existuj\u00fa v API. Okrem toho, k\u00f3d vyzer\u00e1 kompaktne a \u010disto, ni\u017e\u0161ie je pr\u00edklad funkcie, ktor\u00e1 zbiera koment\u00e1re k pr\u00edspevku.<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"python\">def get_comments(submission_id):\n    reddit = Reddit(check_for_updates=False, user_agent=AGENT)\n    submission = reddit.submission(id=submission_id)\n    more_comments = submission.comments.replace_more()\n    if more_comments:\n        skipped_comments = sum(x.count for x in more_comments)\n        logger.debug('Skipped %d MoreComments (%d comments)',\n                     len(more_comments), skipped_comments)\n    return submission.comments.list()\n<\/code><\/pre>\n<p><i>Prevzat\u00e9 z <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/timendum\/reddit-utilities\/blob\/master\/thread-export.py\">tejto<\/a><\/noindex> v\u00fdberu u\u017eito\u010dn\u00fdch n\u00e1strojov na zabalenie.<\/i><\/p>\n<p>Aj ke\u010f tu m\u00e1me najlep\u0161\u00ed mo\u017en\u00fd scen\u00e1r, je potrebn\u00e9 zv\u00e1\u017ei\u0165 nieko\u013eko d\u00f4le\u017eit\u00fdch faktorov z re\u00e1lneho \u017eivota:<\/p>\n<ul>\n<li>Limity API \u2014 sme n\u00faten\u00ed z\u00edska\u0165 \u00fadaje v d\u00e1vkach (pauzy medzi po\u017eiadavkami at\u010f.).<\/li>\n<li>\u010cas zberu \u2014 pre \u00fapln\u00fa anal\u00fdzu a porovnanie treba uvies\u0165 dostato\u010dn\u00fd \u010das, len aby spider pre\u0161iel subreddit.<\/li>\n<li>Bot mus\u00ed be\u017ea\u0165 na serveri \u2014 nem\u00f4\u017eete ho jednoducho spusti\u0165 na laptope, da\u0165 do batohu a \u00eds\u0165 na z\u00e1le\u017eitosti. Z tohto d\u00f4vodu som v\u0161etko spustil na VPS. S promo k\u00f3dom habrahabr10 m\u00f4\u017eete u\u0161etri\u0165 \u010fal\u0161\u00edch 10 % z ceny.<\/li>\n<li>Fyzick\u00e1 nedostupnos\u0165 niektor\u00fdch \u00fadajov (s\u00fa vidite\u013en\u00e9 pre administr\u00e1torov alebo s\u00fa pr\u00edli\u0161 \u0165a\u017eko z\u00edskate\u013en\u00e9) \u2014 to treba zoh\u013eadni\u0165, nie v\u0161etky \u00fadaje je mo\u017en\u00e9 z\u00edska\u0165 v rozumnom \u010dase.<\/li>\n<li>Chyby pr\u00e1ce sie\u0165ou: pr\u00e1ca so sie\u0165ou je boles\u0165.<\/li>\n<li>Toto s\u00fa \u017eiv\u00e9 re\u00e1lne \u00fadaje \u2014 nie s\u00fa \u00faplne \u010dist\u00e9.<\/li>\n<\/ul>\n<p>\nSamozrejme, je potrebn\u00e9 zoh\u013eadni\u0165 uveden\u00e9 nuansy vo v\u00fdvoji. Konkr\u00e9tne hodiny\/de\u0144 s\u00fa z\u00e1visl\u00e9 na sk\u00fasenostiach s v\u00fdvojom alebo s podobn\u00fdmi \u00falohami, napriek tomu vid\u00edme, \u017ee \u00faloha je \u010disto in\u017einierska a nevy\u017eaduje \u010fal\u0161ie pohyby na jej rie\u0161enie \u2014 v\u0161etko sa d\u00e1 ve\u013emi dobre oceni\u0165, napl\u00e1nova\u0165 a vykona\u0165.<\/p>\n<h2>Porovnanie sekci\u00ed Habr<\/h2>\n<p>\nPrejdime k zauj\u00edmavej\u0161iemu a netradi\u010dn\u00e9mu pr\u00edpadu porovnania pr\u00fadov a\/alebo sekci\u00ed Habr.<\/p>\n<p>Skontrolujeme n\u00e1\u0161 checklist zlo\u017eitosti \u2014 tu, aby sme pochopili ka\u017ed\u00fd bod, bude potrebn\u00e9 trochu sa v \u00falohe pohra\u0165 a experimentova\u0165.<\/p>\n<ul>\n<li>Najprv si mysl\u00edte, \u017ee existuje API, ale nie je. \u00c1no, \u00e1no, Habr m\u00e1 API, ale len nie je pr\u00edstupn\u00e9 pre pou\u017e\u00edvate\u013eov (alebo mo\u017eno v\u00f4bec nefunguje).<\/li>\n<li>Potom jednoducho za\u010dnete spracov\u00e1va\u0165 html \u2014 \u201eimport requests\u201c, \u010do by sa mohlo pokazi\u0165?<\/li>\n<li>A ako vlastne spracov\u00e1va\u0165? Najjednoduch\u0161\u00ed a \u010dasto pou\u017e\u00edvan\u00fd pr\u00edstup je iterova\u0165 pod\u013ea ID, \u010do potrebujeme poznamena\u0165, \u017ee nie je najefekt\u00edvnej\u0161\u00ed a budete musie\u0165 spracova\u0165 r\u00f4zne pr\u00edpady \u2014 napr\u00edklad hustota re\u00e1lnych ID medzi v\u0161etk\u00fdmi existuj\u00facimi.\n<p><img decoding=\"async\" alt=\"\u010co m\u00f4\u017ee pokazi\u0165 Data Science? Zber d\u00e1t\" src=\"\/wp-content\/uploads\/2020\/07\/2c67fc14f672e92979edd617314210da.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Prevzat\u00e9 z <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/218607\/\">tejto<\/a><\/noindex> \u010dl\u00e1nok.<\/i><\/li>\n<li>Nepripraven\u00e9 d\u00e1ta zabalen\u00e9 v HTML na sieti s\u00fa probl\u00e9mom. Napr\u00edklad, chcete zozbiera\u0165 a ulo\u017ei\u0165 hodnotenie \u010dl\u00e1nku: vytiahli ste score z HTML a rozhodli ste sa ho ulo\u017ei\u0165 ako \u010d\u00edslo na \u010fal\u0161ie spracovanie:\u00a0\n<p>1) int(score) vyvol\u00e1va chybu: preto\u017ee na Habr\u00e9 je m\u00ednus, ako napr\u00edklad v riadku \"\u20135\" \u2014 to je kr\u00e1tka poml\u010dka, nie znamienko m\u00ednus (prekvapuj\u00face, \u017ee?), a tak v nejakom momente bolo potrebn\u00e9 aktivova\u0165 parser s tak\u00fdmto hrozn\u00fdm fixom.<\/p>\n<pre><code class=\"python\">try:\n      score_txt = post.find(class_=\"score\").text.replace(u\"\u2013\",\"-\").replace(u\"+\",\"+\")\n      score = int(score_txt)\n      if check_date(date):\n        post_score += score\n<\/code><\/pre>\n<p>\nD\u00e1tumy, plusy a m\u00ednusy nemusia v\u00f4bec existova\u0165 (ako vid\u00edme vy\u0161\u0161ie vo funkcii check_date a to sa stalo).<\/p>\n<p>2) Neescapovan\u00e9 \u0161peci\u00e1lne znaky \u2014 pr\u00eddu, mus\u00edte by\u0165 pripraven\u00ed.<\/p>\n<p>3) \u0160trukt\u00fara sa men\u00ed v z\u00e1vislosti od typu pr\u00edspevku.<\/p>\n<p>4) Star\u00e9 pr\u00edspevky m\u00f4\u017eu ma\u0165 **podivn\u00fa \u0161trukt\u00faru**.<\/li>\n<li>V podstate spracovanie ch\u00fdb a toho, \u010do m\u00f4\u017ee alebo nem\u00f4\u017ee nasta\u0165, mus\u00edte spracova\u0165 a nie je mo\u017en\u00e9 presne predpoveda\u0165, \u010do sa stane zle a ak\u00e1 m\u00f4\u017ee by\u0165 \u0161trukt\u00fara a \u010do kde m\u00f4\u017ee zlyha\u0165 \u2014 budete musie\u0165 len sk\u00fa\u0161a\u0165 a zoh\u013eadni\u0165 chyby, ktor\u00e9 parser vyhod\u00ed.<\/li>\n<li>Potom si uvedom\u00edte, \u017ee potrebujete parsova\u0165 vo viacer\u00fdch vl\u00e1knach, inak parsovanie v jednom vl\u00e1kne zaberie 30+ hod\u00edn (to je \u010dist\u00fd \u010das vykonania u\u017e funk\u010dn\u00e9ho jednovl\u00e1knov\u00e9ho parsera, ktor\u00fd sp\u00ed a nie je pod \u017eiadnymi banmi). V <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/276383\/\">tejto<\/a><\/noindex> \u010dl\u00e1nku to v ur\u010ditom okamihu viedlo k podobnej sch\u00e9me:<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"\u010co m\u00f4\u017ee pokazi\u0165 Data Science? Zber d\u00e1t\" src=\"\/wp-content\/uploads\/2020\/07\/9ad1853f3ee2c9321ee9f1a5e3efea7d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak\u017ee checklist zlo\u017eitosti:<\/p>\n<ul>\n<li>Pr\u00e1ca so sie\u0165ou a parsovanie HTML s iter\u00e1ciou a preh\u013ead\u00e1van\u00edm pod\u013ea ID.<\/li>\n<li>Dokumenty s\u00fa heterog\u00e9nne.<\/li>\n<li>Mnoho miest, kde m\u00f4\u017ee k\u00f3d \u013eahko zlyha\u0165.<\/li>\n<li>Je potrebn\u00e9 p\u00edsa\u0165 || k\u00f3d.<\/li>\n<li>Ch\u00fdba potrebn\u00e1 dokument\u00e1cia, pr\u00edklady k\u00f3du a\\\/alebo komunita.<\/li>\n<\/ul>\n<p>\nPodmiene\u010dn\u00e9 ohodnotenie \u010dasu na t\u00fato \u00falohu bude 3-5 kr\u00e1t vy\u0161\u0161ie, ne\u017e na zber d\u00e1t z Redditu.<\/p>\n<h2>Porovnanie skup\u00edn Odnoklassniki<\/h2>\n<p>\nPrejdeme k technicky najzauj\u00edmavej\u0161iemu pr\u00edpadu z t\u00fdch, ktor\u00e9 s\u00fa pop\u00edsan\u00e9. Pre m\u0148a bol zauj\u00edmav\u00fd pr\u00e1ve t\u00fdm, \u017ee na prv\u00fd poh\u013ead vyzer\u00e1 pomerne trivi\u00e1lne, ale v\u00f4bec tak\u00e9 nie je \u2014 akon\u00e1hle do neho \u0161\u0165uknete palicou.<\/p>\n<p>Za\u010dneme na\u0161\u00edm checklistom zlo\u017eitosti a uvedom\u00edme si, \u017ee mnoh\u00e9 z nich sa uk\u00e1\u017eu by\u0165 ove\u013ea zlo\u017eitej\u0161ie, ne\u017e sa na prv\u00fd poh\u013ead zdaj\u00fa:<\/p>\n<ul>\n<li>API je, ale takmer \u00faplne ch\u00fdbaj\u00fa potrebn\u00e9 funkcie.<\/li>\n<li>Na ur\u010dit\u00e9 funkcie je potrebn\u00e9 \u017eiada\u0165 o pr\u00edstup cez e-mail, teda vydanie pr\u00edstupu nie je okam\u017eit\u00e9.<\/li>\n<li>Je to hrozne zdokumentovan\u00e9 (za\u010dneme t\u00fdm, \u017ee sa v\u0161ade mie\u0161aj\u00fa rusk\u00e9 a anglick\u00e9 pojmy, pri\u010dom \u00faplne nekonzistentne \u2014 niekedy si mus\u00edte len h\u00e1da\u0165, \u010do od v\u00e1s niekde chc\u00fa) a, navy\u0161e, nie je vhodn\u00e9 designovo na z\u00edskanie \u00fadajov, napr\u00edklad, <noindex><a rel=\"nofollow\" href=\"https:\/\/apiok.ru\/dev\/methods\/rest\/discussions\/discussions.getComments\">potrebnej funkcie<\/a><\/noindex>.<\/li>\n<li>Vy\u017eaduje dokument\u00e1ciu o seden\u00ed, ale v skuto\u010dnosti ju nepou\u017e\u00edva \u2014 a niet \u017eiadneho sp\u00f4sobu, ako pochopi\u0165 v\u0161etky nuansy re\u017eimov API, okrem toho, \u017ee sa len na nie\u010do klik\u00e1 a d\u00fafa, \u017ee nie\u010do bude fungova\u0165.<\/li>\n<li>Ch\u00fdbaj\u00fa pr\u00edklady a komunita, jedin\u00fd bod podpory pri zbere inform\u00e1ci\u00ed \u2014 mal\u00fd <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/alternativshik\/python-odnoklassniki\">wrapper<\/a><\/noindex> v Pythone (bez ve\u013ek\u00e9ho mno\u017estva pr\u00edkladov pou\u017eitia).<\/li>\n<li>Naj\u00faspe\u0161nej\u0161ia mo\u017enos\u0165 sa zd\u00e1 by\u0165 Selenium, preto\u017ee mnoh\u00e9 potrebn\u00e9 \u00fadaje s\u00fa zamknut\u00e9.<br \/>\n1) To znamen\u00e1, \u017ee sa vykon\u00e1va autoriz\u00e1cia cez fikt\u00edvneho pou\u017e\u00edvate\u013ea (a manu\u00e1lnu registr\u00e1ciu).<\/p>\n<p>2) Av\u0161ak so Selenom niet \u017eiadnych z\u00e1ruk spr\u00e1vnej a opakovate\u013enej funkcie (aspo\u0148 v pr\u00edpade ok.ru ur\u010dite).<\/p>\n<p>3) Str\u00e1nka Ok.ru obsahuje chyby JavaScriptu a niekedy sa spr\u00e1va zvl\u00e1\u0161tne a nekonzistentne.<\/p>\n<p>4) Je potrebn\u00e9 venova\u0165 sa pagin\u00e1cii, na\u010d\u00edtavaniu prvkov at\u010f\u2026<\/p>\n<p>5) Chyby API, ktor\u00e9 wrapper vracia, bude potrebn\u00e9 o\u0161etrova\u0165 nepriamo, napr\u00edklad takto (kus experiment\u00e1lneho k\u00f3du):<\/p>\n<pre><code class=\"python\">def get_comments(args, context, discussions):\n    pause = 1\n    if args.extract_comments:\n        all_comments = set()\n#m\u00e1 zmysel sledova\u0165 u\u017e spracovan\u00e9 diskusie\n        for discussion in tqdm(discussions): \n            try:\n                comments = get_comments_from_discussion_via_api(context, discussion)\n            except odnoklassniki.api.OdnoklassnikiError as e:\n                if \"NOT_FOUND\" in str(e):\n                    comments = set()\n                else:\n                    print(e)\n                    bp()\n                    pass\n            all_comments |= comments\n            time.sleep(pause)\n        return all_comments\n<\/code><\/pre>\n<p>\nMoja ob\u013e\u00faben\u00e1 chyba bola:<\/p>\n<p><code>OdnoklassnikiError(\"Error(code: 'None', description: 'HTTP error', method: 'discussions.getComments', params: \u2026)\")<\/code><\/p>\n<p>6) Nakoniec sa kombin\u00e1cia Selenium + API jav\u00ed ako najrozumnej\u0161ia alternat\u00edva.<\/li>\n<li>Je potrebn\u00e9 uchova\u0165 stav a re\u0161tartova\u0165 syst\u00e9m, spracova\u0165 mno\u017estvo ch\u00fdb, vr\u00e1tane nekonzistentn\u00e9ho spr\u00e1vania str\u00e1nky \u2014 pri\u010dom tieto chyby je dos\u0165 \u0165a\u017ek\u00e9 si predstavi\u0165 (ak profesion\u00e1lne nep\u00ed\u0161ete parsery, samozrejme).<\/li>\n<\/ul>\n<p>\nPodmienen\u00e9 hodnotenie \u010dasu pre t\u00fato \u00falohu bude 3-5 kr\u00e1t vy\u0161\u0161ie ako pri zbere d\u00e1t z Habra. Napriek tomu, \u017ee v pr\u00edpade Habra pou\u017e\u00edvame priamy pr\u00edstup s parsovan\u00edm HTML, a v pr\u00edpade OK m\u00f4\u017eeme na kritick\u00fdch miestach pracova\u0165 s API.<\/p>\n<h2>Zhrnutie<\/h2>\n<p>\nAko by ste od v\u00e1s ne\u017eiadaj\u00fa o odhad term\u00ednov \u201ena mieste\u201c (predsa dnes pl\u00e1nujeme!) objemn\u00e9ho modulu pipepl\u00e1nu na spracovanie d\u00e1t, \u010das vykonania sa takmer nikdy ned\u00e1 ani kvalitne odhadn\u00fa\u0165 bez anal\u00fdzy parametrov \u00falohy. <\/p>\n<p>Ak sa pozrieme na to trochu filozoficky, strat\u00e9gie hodnotenia v agile s\u00fa vhodn\u00e9 pre in\u017einierske \u00falohy, ale s experiment\u00e1lnej\u0161\u00edmi a v niektor\u00fdch oh\u013eadoch \u201etvoriv\u00fdmi\u201c a v\u00fdskumn\u00fdmi \u00falohami, t.j. menej predv\u00eddate\u013en\u00fdmi, vznikaj\u00fa \u0165a\u017ekosti, ako v pr\u00edkladoch, ktor\u00e9 sme tu prebrali. <\/p>\n<p>Samozrejme, zber \u00fadajov je len jasn\u00fdm ilustrat\u00edvnym pr\u00edkladom \u2014 zvy\u010dajne sa t\u00e1to \u00faloha zd\u00e1 by\u0165 neuverite\u013ene jednoduch\u00e1 a technicky nen\u00e1ro\u010dn\u00e1, a pr\u00e1ve v detailoch sa tu naj\u010dastej\u0161ie skr\u00fdva diabol. A pr\u00e1ve na tejto \u00falohe sa d\u00e1 uk\u00e1za\u0165 cel\u00fd rozsah mo\u017en\u00fdch variantov toho, \u010do m\u00f4\u017ee \u00eds\u0165 zle a ako ve\u013emi m\u00f4\u017ee pr\u00e1ca me\u0161ka\u0165. <\/p>\n<p>Ak sa na charakteristiky \u00falohy pozrieme z r\u00fdchleho poh\u013eadu bez \u010fal\u0161\u00edch experimentov, Reddit a OK vyzeraj\u00fa podobne: existuje API, python wrapper, ale na podstate je rozdiel obrovsk\u00fd. Ak hodnot\u00edme pod\u013ea t\u00fdchto parametrov, parser Habra vyzer\u00e1 zlo\u017eitej\u0161ie ako OK \u2014 a v praxi je to \u00faplne naopak a presne to sa d\u00e1 zisti\u0165 prostredn\u00edctvom jednoduch\u00fdch experimentov na anal\u00fdzu parametrov \u00falohy.<\/p>\n<p>Z m\u00f4jho poh\u013eadu je najefekt\u00edvnej\u0161\u00edm pr\u00edstupom pribli\u017en\u00fd odhad \u010dasu, ktor\u00fd potrebujete na samotn\u00fa predbe\u017en\u00fa anal\u00fdzu a jednoduch\u00e9 prv\u00e9 experimenty, \u010d\u00edtanie dokument\u00e1cie \u2014 to v\u00e1m umo\u017en\u00ed poskytn\u00fa\u0165 presn\u00fd odhad pre cel\u00fa pr\u00e1cu. V term\u00ednoch popul\u00e1rnej metodol\u00f3gie agile \u2014 \u017eiadam, aby mi bol zalo\u017een\u00fd tiket na \"odhady parametrov \u00falohy\", na z\u00e1klade ktor\u00e9ho m\u00f4\u017eem poskytn\u00fa\u0165 odhad toho, \u010do je mo\u017en\u00e9 vykona\u0165 v r\u00e1mci \"sprintu\" a poskytn\u00fa\u0165 presnej\u0161\u00ed odhad pre ka\u017ed\u00fa \u00falohu.<\/p>\n<p>Preto sa najefekt\u00edvnej\u0161\u00ed argument zd\u00e1 by\u0165 ten, ktor\u00fd by uk\u00e1zal \u201enetechnick\u00e9mu\u201c \u0161pecialistovi, ako ve\u013emi sa bude variabilita \u010dasu a zdrojov l\u00ed\u0161i\u0165 v z\u00e1vislosti od parametrov, ktor\u00e9 e\u0161te treba ohodnoti\u0165.<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"http:\/\/ruvds.com\/ru-rub?utm_source=habr&amp;utm_medium=article&amp;utm_campaign=param&amp;utm_content=datascience1#order\"><img decoding=\"async\" alt=\"\u010co m\u00f4\u017ee pokazi\u0165 Data Science? Zber d\u00e1t\" src=\"\/wp-content\/uploads\/2020\/07\/2ca567a078c8500d37dfcf982e76252c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u0435\u0442 100500 \u043a\u0443\u0440\u0441\u043e\u0432 \u043f\u043e Data Science \u0438 \u0434\u0430\u0432\u043d\u043e \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e, \u0447\u0442\u043e \u0431\u043e\u043b\u044c\u0448\u0435 \u0432\u0441\u0435\u0433\u043e \u0434\u0435\u043d\u0435\u0433 \u0432 Data Science \u043c\u043e\u0436\u043d\u043e \u0437\u0430\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u043d\u043e \u043a\u0443\u0440\u0441\u0430\u043c\u0438 \u043f\u043e Data Science (\u0437\u0430\u0447\u0435\u043c \u043a\u043e\u043f\u0430\u0442\u044c, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0434\u0430\u0432\u0430\u0442\u044c \u043b\u043e\u043f\u0430\u0442\u044b?). \u041e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043c\u0438\u043d\u0443\u0441 \u044d\u0442\u0438\u0445 \u043a\u0443\u0440\u0441\u043e\u0432 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u043e\u043d\u0438 \u043d\u0435 \u0438\u043c\u0435\u044e\u0442 \u043d\u0438\u0447\u0435\u0433\u043e \u043e\u0431\u0449\u0435\u0433\u043e \u0441 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u043e\u0439: \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0434\u0430\u0441\u0442 \u0432\u0430\u043c \u0447\u0438\u0441\u0442\u044b\u0435, \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u0430\u043d\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0432 \u043d\u0443\u0436\u043d\u043e\u043c \u0444\u043e\u0440\u043c\u0430\u0442\u0435. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":89207,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-89206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47\u010co m\u00f4\u017ee \u00eds\u0165 zle s Data Science? Zber d\u00e1t | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-20T05:41:57+00:00","article:modified_time":"2020-07-20T05:41:57+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"89206","title":null,"description":null,"keywords":null,"keyphrases":{"focus":[],"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:51:09","updated":"2026-08-11 12:50:12","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/89206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=89206"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/89206\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/89207"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=89206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=89206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=89206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}