{"id":89206,"date":"2020-07-20T07:41:57","date_gmt":"2020-07-20T05:41:57","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh"},"modified":"2020-07-20T07:41:57","modified_gmt":"2020-07-20T05:41:57","slug":"chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","title":{"rendered":"Co se m\u016f\u017ee pokazit s Data Science? Shrom\u00e1\u017ed\u011bn\u00ed dat","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\"><img decoding=\"async\" alt=\"Co se m\u016f\u017ee pokazit s Data Science? Shrom\u00e1\u017ed\u011bn\u00ed dat\" src=\"\/wp-content\/uploads\/2020\/07\/49fb91fc3e45754e437b2c1fc4fec12f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\nDnes existuje 100500 kurz\u016f Data Science a u\u017e je d\u00e1vno zn\u00e1mo, \u017ee nejv\u00edce pen\u011bz v Data Science se d\u00e1 vyd\u011blat pr\u00e1v\u011b na kurzech Data Science (pro\u010d kopat, kdy\u017e lze prod\u00e1vat lopaty?). Hlavn\u00ed nev\u00fdhodou t\u011bchto kurz\u016f je, \u017ee nemaj\u00ed nic spole\u010dn\u00e9ho se skute\u010dnou prac\u00ed: nikdo v\u00e1m ned\u00e1 \u010dist\u00e1, zpracovan\u00e1 data ve spr\u00e1vn\u00e9m form\u00e1tu. A kdy\u017e skon\u010d\u00edte s kurzy a za\u010dnete \u0159e\u0161it skute\u010dn\u00fd \u00fakol \u2014 vyvst\u00e1v\u00e1 \u0159ada nuanc\u00ed.<\/p>\n<p>Proto zahajujeme s\u00e9rii pozn\u00e1mek \u201eCo se m\u016f\u017ee pokazit s Data Science\u201c, zalo\u017eenou na skute\u010dn\u00fdch ud\u00e1lostech, kter\u00e9 se staly mn\u011b, m\u00fdm p\u0159\u00e1tel\u016fm a koleg\u016fm. Budeme rozeb\u00edrat na re\u00e1ln\u00fdch p\u0159\u00edkladech typick\u00e9 \u00fakoly v oblasti Data Science: jak to ve skute\u010dnosti prob\u00edh\u00e1. Dnes za\u010dneme s \u00fakolem sb\u011bru dat.<\/p>\n<p>A prvn\u00ed v\u011bc, o kterou se lid\u00e9 p\u0159i pr\u00e1ci se skute\u010dn\u00fdmi daty zakop\u00e1vaj\u00ed \u2014 je vlastn\u011b sb\u011br t\u011bchto relevantn\u00edch dat. Kl\u00ed\u010dov\u00e1 my\u0161lenka tohoto \u010dl\u00e1nku:<\/p>\n<blockquote><p><b>Syst\u00e9mov\u011b podce\u0148ujeme \u010das, zdroje a \u00fasil\u00ed na sb\u011br, \u010di\u0161t\u011bn\u00ed a p\u0159\u00edpravu dat.<\/b><\/p><\/blockquote>\n<p>\nA p\u0159edev\u0161\u00edm si probereme, co d\u011blat, aby se tomu zabr\u00e1nilo.<\/p>\n<p>Podle r\u016fzn\u00fdch odhad\u016f \u010di\u0161t\u011bn\u00ed, transformace, zpracov\u00e1n\u00ed dat, feature engineering atd. zab\u00edr\u00e1 80\u201390 % \u010dasu, zat\u00edmco anal\u00fdza 10\u201320 %, p\u0159i\u010dem\u017e v\u011bt\u0161ina vzd\u011bl\u00e1vac\u00edho materi\u00e1lu se zam\u011b\u0159uje v\u00fdhradn\u011b na anal\u00fdzu.<\/p>\n<p>Poj\u010fme si probrat jako typick\u00fd p\u0159\u00edklad jednoduch\u00fd analytick\u00fd \u00fakol ve t\u0159ech variant\u00e1ch a uvid\u00edme, jak\u00e9 mohou b\u00fdt \u201ezhor\u0161uj\u00edc\u00ed okolnosti\u201c.<\/p>\n<p>A pro p\u0159\u00edklad op\u011bt zv\u00e1\u017e\u00edme podobn\u00e9 varianty \u00fakolu sb\u011bru dat a srovn\u00e1n\u00ed komunit pro:<\/p>\n<ol>\n<li>Dva subreddity Reddit<\/li>\n<li>Dv\u011b sekce Habra<\/li>\n<li>Dv\u011b skupiny na Odnoklassniki<\/li>\n<\/ol>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Podm\u00ednkov\u00fd p\u0159\u00edstup v teorii<\/h2>\n<p>\nOtev\u0159ete web a p\u0159e\u010dt\u011bte si p\u0159\u00edklady, pokud je to jasn\u00e9, vy\u010dle\u0148te n\u011bkolik hodin na \u010dten\u00ed, n\u011bkolik hodin na k\u00f3d podle p\u0159\u00edklad\u016f a lad\u011bn\u00ed. P\u0159idejte n\u011bkolik hodin na sb\u011br. P\u0159idejte n\u011bkolik hodin nav\u00edc (vyn\u00e1sobte dv\u011bma a p\u0159idejte N hodin).<\/p>\n<p><b>Kl\u00ed\u010dov\u00fd bod: \u010dasov\u00fd odhad je zalo\u017een na p\u0159edpokladech a domn\u011bnk\u00e1ch o tom, kolik to zabere \u010dasu.<\/b><\/p>\n<p>Za\u010d\u00edt anal\u00fdzu \u010dasu je nutn\u00e9 s odhadem n\u00e1sleduj\u00edc\u00edch parametr\u016f pro podm\u00ednkov\u00fd \u00fakol, popsan\u00fd v\u00fd\u0161e:<\/p>\n<ul>\n<li>Jak\u00e1 je velikost dat a kolik je pot\u0159eba fyzicky shrom\u00e1\u017edit (*viz n\u00ed\u017ee*).<\/li>\n<li>Jak\u00fd \u010das sb\u011bru jedn\u00e9 polo\u017eky a kolik je pot\u0159eba \u010dekat, ne\u017e lze shrom\u00e1\u017edit druhou.<\/li>\n<li>Zahrnout k\u00f3d, kter\u00fd uchov\u00e1v\u00e1 stav a spou\u0161t\u00ed restart, kdy\u017e (a ne pokud) v\u0161e sel\u017ee.<\/li>\n<li>Zjistit, zda pot\u0159ebujeme autorizaci, a zahrnout \u010das pot\u0159ebn\u00fd pro p\u0159\u00edstup p\u0159es API.<\/li>\n<li>Zahrnout po\u010det chyb jako funkci slo\u017eitosti dat \u2014 vyhodnotit na z\u00e1klad\u011b konkr\u00e9tn\u00edho \u00fakolu: struktura, kolik transformac\u00ed, co a jak extrahujeme.<\/li>\n<li>Zahrnout chyby s\u00edt\u011b a probl\u00e9my s nestandardn\u00edm chov\u00e1n\u00edm projektu.<\/li>\n<li>Zhodnotit, zda pot\u0159ebn\u00e9 funkce jsou v dokumentaci, a pokud ne, tak jak a kolik \u010dasu bude pot\u0159eba pro workaround.<\/li>\n<\/ul>\n<p>\nNejd\u016fle\u017eit\u011bj\u0161\u00ed je, \u017ee pro hodnocen\u00ed \u010dasu \u2014 mus\u00edte fakticky investovat \u010das a \u00fasil\u00ed do 'p\u0159\u00edpravy na boj' \u2014 jen tak bude va\u0161e pl\u00e1nov\u00e1n\u00ed adekv\u00e1tn\u00ed. Tak\u017ee a\u0165 v\u00e1s kdokoli tla\u010d\u00ed \u0159\u00edct 'kolik \u010dasu je t\u0159eba na sb\u011br dat' \u2014 vy\u017e\u00e1dejte si \u010das na p\u0159edb\u011b\u017enou anal\u00fdzu a obhajte t\u00edm, jak moc se \u010das bude li\u0161it v z\u00e1vislosti na re\u00e1ln\u00fdch parametrech \u00fakolu.<\/p>\n<p>A nyn\u00ed uk\u00e1\u017eeme konkr\u00e9tn\u00ed p\u0159\u00edklady, kde se tyto parametry budou m\u011bnit.<\/p>\n<p><b>Kl\u00ed\u010dov\u00fd bod: hodnocen\u00ed je zalo\u017eeno na anal\u00fdze kl\u00ed\u010dov\u00fdch faktor\u016f, kter\u00e9 ovliv\u0148uj\u00ed objem a slo\u017eitost pr\u00e1ce.<\/b><\/p>\n<p>Hodnocen\u00ed zalo\u017een\u00e9 na domn\u011bnk\u00e1ch \u2014 je to dobr\u00fd p\u0159\u00edstup, kdy\u017e jsou funk\u010dn\u00ed prvky dost mal\u00e9 a nen\u00ed mnoho faktor\u016f, kter\u00e9 by mohly z\u00e1sadn\u011b ovlivnit strukturu \u00fakolu. Ale v p\u0159\u00edpad\u011b \u0159ady \u00fakol\u016f Data Science se takov\u00fdch faktor\u016f st\u00e1v\u00e1 nesm\u00edrn\u011b mnoho a tento p\u0159\u00edstup se st\u00e1v\u00e1 neadekv\u00e1tn\u00edm.<\/p>\n<h2>Srovn\u00e1n\u00ed komunit Reddit<\/h2>\n<p>\nZa\u010dn\u011bme s t\u00edm nejjednodu\u0161\u0161\u00edm p\u0159\u00edpadem (jak se pozd\u011bji uk\u00e1\u017ee). Opravu, pokud budeme up\u0159\u00edmn\u00ed, \u010del\u00edme prakticky ide\u00e1ln\u00edmu p\u0159\u00edpadu, zkontrolujeme n\u00e1\u0161 checklist slo\u017eitosti:<\/p>\n<ul>\n<li>Existuje \u010dist\u00e9, srozumiteln\u00e9 a dokumentovan\u00e9 API.<\/li>\n<li>Je extr\u00e9mn\u011b jednoduch\u00e9 a hlavn\u011b automaticky se generuje token.<\/li>\n<li>Existuje <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/praw-dev\/praw\">python wrapper<\/a><\/noindex> \u2014 s mno\u017estv\u00edm p\u0159\u00edklad\u016f.<\/li>\n<li>Komunita, kter\u00e1 se zab\u00fdv\u00e1 anal\u00fdzou a sb\u011brem dat na Redditu (a\u017e po YouTube videa vysv\u011btluj\u00edc\u00ed, jak pou\u017e\u00edvat python wrapper) <noindex><a rel=\"nofollow\" href=\"https:\/\/www.programcreek.com\/python\/example\/86599\/praw.Reddit\">tady je nap\u0159\u00edklad<\/a><\/noindex>.<\/li>\n<li>Pot\u0159ebn\u00e9 metody pravd\u011bpodobn\u011b existuj\u00ed v API. Nav\u00edc k\u00f3d vypad\u00e1 kompaktn\u011b a \u010dist\u011b, n\u00ed\u017ee je p\u0159\u00edklad funkce, kter\u00e1 sb\u00edr\u00e1 koment\u00e1\u0159e k p\u0159\u00edsp\u011bvku.<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"python\">def get_comments(submission_id):\n    reddit = Reddit(check_for_updates=False, user_agent=AGENT)\n    submission = reddit.submission(id=submission_id)\n    more_comments = submission.comments.replace_more()\n    if more_comments:\n        skipped_comments = sum(x.count for x in more_comments)\n        logger.debug('Skipped %d MoreComments (%d comments)',\n                     len(more_comments), skipped_comments)\n    return submission.comments.list()\n<\/code><\/pre>\n<p><i>Vzato z <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/timendum\/reddit-utilities\/blob\/master\/thread-export.py\">tato<\/a><\/noindex> v\u00fdb\u011br u\u017eite\u010dn\u00fdch n\u00e1stroj\u016f pro obalen\u00ed.<\/i><\/p>\n<p>I p\u0159es to, \u017ee se zde nach\u00e1z\u00edme v nejlep\u0161\u00edm sc\u00e9n\u00e1\u0159i, je st\u00e1le t\u0159eba zv\u00e1\u017eit n\u011bkolik d\u016fle\u017eit\u00fdch faktor\u016f z re\u00e1ln\u00e9ho \u017eivota:<\/p>\n<ul>\n<li>Limity API \u2013 mus\u00edme z\u00edsk\u00e1vat data po d\u00e1vk\u00e1ch (\u010dekat mezi po\u017eadavky atd.).<\/li>\n<li>\u010cas sb\u011bru \u2013 pro \u00faplnou anal\u00fdzu a srovn\u00e1n\u00ed budeme muset investovat zna\u010dn\u00e9 mno\u017estv\u00ed \u010dasu jen na to, aby pavouk pro\u0161el subreddit.<\/li>\n<li>Bot mus\u00ed b\u011b\u017eet na serveru \u2013 nem\u016f\u017eete ho jen tak spustit na notebooku, sbalit do batohu a j\u00edt na jedn\u00e1n\u00ed. Proto jsem spustil v\u0161e na VPS. S promo k\u00f3dem habrahabr10 m\u016f\u017eete u\u0161et\u0159it dal\u0161\u00edch 10 % n\u00e1klad\u016f.<\/li>\n<li>Fyzick\u00e1 nedostupnost n\u011bkter\u00fdch dat (jsou viditeln\u00e1 pouze admin\u016fm nebo se shroma\u017e\u010fuj\u00ed p\u0159\u00edli\u0161 obt\u00ed\u017en\u011b) \u2013 to je t\u0159eba zohlednit, ne v\u0161echna data je mo\u017en\u00e9 shrom\u00e1\u017edit v p\u0159im\u011b\u0159en\u00e9m \u010dase.<\/li>\n<li>Chyby v pr\u00e1ci s\u00edt\u011b: pr\u00e1ce se s\u00edt\u00ed je bolest.<\/li>\n<li>To jsou \u017eiv\u00e1 skute\u010dn\u00e1 data \u2013 nikdy nejsou \u010dist\u00e1.<\/li>\n<\/ul>\n<p>\nSamoz\u0159ejm\u011b je d\u016fle\u017eit\u00e9 zohlednit uveden\u00e9 nuance ve v\u00fdvoji. Konkr\u00e9tn\u00ed hodiny\/dny z\u00e1vis\u00ed na v\u00fdvoj\u00e1\u0159sk\u00e9 zku\u0161enosti nebo zku\u0161enosti s podobn\u00fdmi \u00fakoly, nicm\u00e9n\u011b vid\u00edme, \u017ee zde je \u00fakol v\u00fdhradn\u011b in\u017een\u00fdrsk\u00fd a nevy\u017eaduje dal\u0161\u00ed \u00fasil\u00ed pro jeho \u0159e\u0161en\u00ed \u2013 v\u0161e lze velmi dob\u0159e ocenit, napl\u00e1novat a realizovat.<\/p>\n<h2>Srovn\u00e1n\u00ed sekc\u00ed Habr<\/h2>\n<p>\nP\u0159ech\u00e1z\u00edme k zaj\u00edmav\u011bj\u0161\u00edmu a netradi\u010dn\u00edmu p\u0159\u00edpadu srovn\u00e1n\u00ed proud\u016f a\/nebo sekc\u00ed Habr.<\/p>\n<p>Zkontrolujeme n\u00e1\u0161 checklist slo\u017eitosti \u2013 zde, abychom pochopili ka\u017ed\u00fd bod, u\u017e bude t\u0159eba se trochu poprat s \u00fakolem a experimentovat.<\/p>\n<ul>\n<li>Nejprve si mysl\u00edte, \u017ee existuje API, ale nen\u00ed. Ano, Habr m\u00e1 API, ale je dostupn\u00e9 pouze admin\u016fm (a mo\u017en\u00e1 ani nefunguje).<\/li>\n<li>Pak za\u010dnete jednodu\u0161e parsovat HTML \u2013 \"import requests\", co se m\u016f\u017ee pokazit?<\/li>\n<li>A jak vlastn\u011b parsovat? Nejjednodu\u0161\u0161\u00ed a nejb\u011b\u017en\u011bj\u0161\u00ed p\u0159\u00edstup je iterovat podle ID, ale nezapome\u0148me, \u017ee to nen\u00ed nejefektivn\u011bj\u0161\u00ed a budeme muset zpracov\u00e1vat r\u016fzn\u00e9 p\u0159\u00edpady \u2013 zde je nap\u0159\u00edklad hustota skute\u010dn\u00fdch ID mezi v\u0161emi existuj\u00edc\u00edmi.\n<p><img decoding=\"async\" alt=\"Co se m\u016f\u017ee pokazit s Data Science? Shrom\u00e1\u017ed\u011bn\u00ed dat\" src=\"\/wp-content\/uploads\/2020\/07\/2c67fc14f672e92979edd617314210da.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Vzato z <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/218607\/\">tato<\/a><\/noindex> \u010dl\u00e1nku.<\/i><\/li>\n<li>Surov\u00e9 \u00fadaje obalen\u00e9 v HTML p\u0159es s\u00ed\u0165 jsou bolest\u00ed. Nap\u0159\u00edklad, chcete shrom\u00e1\u017edit a ulo\u017eit hodnocen\u00ed \u010dl\u00e1nku: vyt\u00e1hli jste score z html a rozhodli se ho ulo\u017eit jako \u010d\u00edslo pro dal\u0161\u00ed zpracov\u00e1n\u00ed.\u00a0\n<p>1) int(score) vyvol\u00e1v\u00e1 chybu: proto\u017ee na Habru je m\u00ednus, nap\u0159\u00edklad v \u0159\u00e1dku \"\u20135\" \u2014 to je kr\u00e1tk\u00e1 spojka, nikoli znak m\u00ednusu (ne\u010dekan\u00e9, \u017ee?), tak\u017ee v ur\u010dit\u00e9m okam\u017eiku bylo nutn\u00e9 obnovit parser s t\u00edmto hrozn\u00fdm opravou.<\/p>\n<pre><code class=\"python\">try:\n      score_txt = post.find(class_=\"score\").text.replace(u\"\u2013\", \"-\").replace(u\"+\", \"+\")\n      score = int(score_txt)\n      if check_date(date):\n        post_score += score\n<\/code><\/pre>\n<p>\nData, plusy a minusy mohou v\u016fbec chyb\u011bt (jak vid\u00edme v\u00fd\u0161e v funkci check_date, to se stalo).<\/p>\n<p>2) Neescapovan\u00e9 speci\u00e1ln\u00ed znaky \u2014 ty p\u0159ijdou, je t\u0159eba se na to p\u0159ipravit.<\/p>\n<p>3) Struktura se m\u011bn\u00ed v z\u00e1vislosti na typu p\u0159\u00edsp\u011bvku.<\/p>\n<p>4) Star\u00e9 p\u0159\u00edsp\u011bvky mohou m\u00edt **podivnou strukturu**.<\/li>\n<li>V podstat\u011b bude t\u0159eba zpracov\u00e1v\u00e1n\u00ed chyb a co se m\u016f\u017ee nebo nemus\u00ed st\u00e1t zpracovat, a nelze p\u0159esn\u011b p\u0159edpov\u011bd\u011bt, co se pokaz\u00ed a jak\u00e1 je\u0161t\u011b m\u016f\u017ee b\u00fdt struktura a co kde vypadne \u2014 prost\u011b to budete muset zkusit a zohlednit chyby, kter\u00e9 vyhazuje parser.<\/li>\n<li>Pak si uv\u011bdom\u00edte, \u017ee je t\u0159eba parsovat ve v\u00edce vl\u00e1knech, jinak parsel v jednom pak zabere 30+ hodin (to je \u010dist\u011b doba prov\u00e1d\u011bn\u00ed ji\u017e funk\u010dn\u00edho jednovl\u00e1knov\u00e9ho parseru, kter\u00fd sp\u00ed a nepodl\u00e9h\u00e1 \u017e\u00e1dn\u00fdm z\u00e1kaz\u016fm). V <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/276383\/\">tato<\/a><\/noindex> \u010dl\u00e1nku, to vedlo v ur\u010dit\u00e9m okam\u017eiku k podobn\u00e9mu sch\u00e9matu:<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Co se m\u016f\u017ee pokazit s Data Science? Shrom\u00e1\u017ed\u011bn\u00ed dat\" src=\"\/wp-content\/uploads\/2020\/07\/9ad1853f3ee2c9321ee9f1a5e3efea7d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCelkov\u011b checklist podle slo\u017eitosti:<\/p>\n<ul>\n<li>Pr\u00e1ce se s\u00edt\u00ed a parsov\u00e1n\u00edm html s iterac\u00ed a proch\u00e1zen\u00edm podle ID.<\/li>\n<li>Dokumenty maj\u00ed heterogenn\u00ed strukturu.<\/li>\n<li>Mnoho m\u00edst, kde k\u00f3d m\u016f\u017ee snadno spadnout.<\/li>\n<li>Je nutn\u00e9 ps\u00e1t || k\u00f3d.<\/li>\n<li>Chyb\u00ed pot\u0159ebn\u00e1 dokumentace, p\u0159\u00edklady k\u00f3du a\/nebo komunita.<\/li>\n<\/ul>\n<p>\nPodm\u00edn\u011bn\u00e9 hodnocen\u00ed \u010dasu pro tento \u00fakol bude 3-5kr\u00e1t vy\u0161\u0161\u00ed, ne\u017e pro sb\u011br dat z Redditu.<\/p>\n<h2>Srovn\u00e1n\u00ed skupin Odnoklassniky<\/h2>\n<p>\nP\u0159ejdeme k technicky nejzaj\u00edmav\u011bj\u0161\u00edmu p\u0159\u00edpadu z popsan\u00fdch. Pro m\u011b byl zaj\u00edmav\u00fd pr\u00e1v\u011b t\u00edm, \u017ee na prvn\u00ed pohled vypad\u00e1 dost trivi\u00e1ln\u011b, ale takov\u00fdm nen\u00ed \u2014 jakmile do n\u011bj zap\u00edchnete h\u016flkou.<\/p>\n<p>Za\u010dneme s na\u0161\u00edm checklistem slo\u017eitosti a poznamen\u00e1me, \u017ee mnoh\u00e9 z nich se uk\u00e1\u017eou b\u00fdt mnohem slo\u017eit\u011bj\u0161\u00ed, ne\u017e vypadaj\u00ed na za\u010d\u00e1tku:<\/p>\n<ul>\n<li>API existuje, ale t\u00e9m\u011b\u0159 \u00fapln\u011b postr\u00e1d\u00e1 pot\u0159ebn\u00e9 funkce.<\/li>\n<li>K n\u011bkter\u00fdm funkc\u00edm je t\u0159eba po\u017e\u00e1dat o p\u0159\u00edstup emailem, co\u017e znamen\u00e1, \u017ee ud\u011blen\u00ed p\u0159\u00edstupu nen\u00ed okam\u017eit\u00e9.<\/li>\n<li>Je to hrozn\u011b zdokumentov\u00e1no (za\u010dn\u011bme t\u00edm, \u017ee se tu m\u00eds\u00ed rusk\u00e9 a anglick\u00e9 term\u00edny, a to naprosto nekonzistentn\u011b - ob\u010das mus\u00edte prost\u011b h\u00e1dat, co od v\u00e1s kde cht\u011bj\u00ed) a nav\u00edc nen\u00ed vhodn\u00e9 pro z\u00edsk\u00e1v\u00e1n\u00ed dat, nap\u0159\u00edklad, <noindex><a rel=\"nofollow\" href=\"https:\/\/apiok.ru\/dev\/methods\/rest\/discussions\/discussions.getComments\">po\u017eadovan\u00e9 funkce<\/a><\/noindex>.<\/li>\n<li>Vy\u017eaduje dokumentaci se sezen\u00edm, ale ve skute\u010dnosti jej nepou\u017e\u00edv\u00e1 - a neexistuje \u017e\u00e1dn\u00fd zp\u016fsob, jak pochopit v\u0161echny nuance re\u017eim\u016f API, krom\u011b toho, \u017ee se prost\u011b proklik\u00e1v\u00e1te a douf\u00e1te, \u017ee n\u011bco bude fungovat.<\/li>\n<li>Chyb\u00ed p\u0159\u00edklady a komunita, jedin\u00fd op\u011brn\u00fd bod p\u0159i shroma\u017e\u010fov\u00e1n\u00ed informac\u00ed - mal\u00fd <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/alternativshik\/python-odnoklassniki\">wrapper<\/a><\/noindex> v pythonu (bez velk\u00e9ho mno\u017estv\u00ed p\u0159\u00edklad\u016f pou\u017eit\u00ed).<\/li>\n<li>Nejv\u00edce funk\u010dn\u00ed varianta vypad\u00e1 na Selenium, proto\u017ee mnoho pot\u0159ebn\u00fdch dat je uzam\u010deno.<br \/>\n1) To znamen\u00e1, \u017ee se prov\u00e1d\u00ed autorizace p\u0159es fiktivn\u00edho u\u017eivatele (a registraci ru\u010dn\u011b).<\/p>\n<p>2) Nicm\u00e9n\u011b se Selenium ned\u00e1 zaru\u010dit spr\u00e1vn\u00e9ho a opakovateln\u00e9ho fungov\u00e1n\u00ed (minim\u00e1ln\u011b v p\u0159\u00edpad\u011b s ok.ru ur\u010dit\u011b).<\/p>\n<p>3) Str\u00e1nka ok.ru obsahuje chyby JavaScriptu a ob\u010das se chov\u00e1 podivn\u011b a nekonzistentn\u011b.<\/p>\n<p>4) Je t\u0159eba se zab\u00fdvat str\u00e1nkov\u00e1n\u00edm, na\u010d\u00edt\u00e1n\u00edm prvk\u016f atd...<\/p>\n<p>5) Chyby API, kter\u00e9 wrapper vrac\u00ed, je nutn\u00e9 komplikovan\u011b zpracov\u00e1vat, nap\u0159\u00edklad takto (kousek experiment\u00e1ln\u00edho k\u00f3du):<\/p>\n<pre><code class=\"python\">def get_comments(args, context, discussions):\n    pause = 1\n    if args.extract_comments:\n        all_comments = set()\n#m\u00e1 smysl sledovat u\u017e zpracovan\u00e9 diskuze\n        for discussion in tqdm(discussions): \n            try:\n                comments = get_comments_from_discussion_via_api(context, discussion)\n            except odnoklassniki.api.OdnoklassnikiError as e:\n                if \"NOT_FOUND\" in str(e):\n                    comments = set()\n                else:\n                    print(e)\n                    bp()\n                    pass\n            all_comments |= comments\n            time.sleep(pause)\n        return all_comments\n<\/code><\/pre>\n<p>\nMoje obl\u00edben\u00e1 chyba byla:<\/p>\n<p><code>OdnoklassnikiError(\"Chyba(k\u00f3d: 'None', popis: 'HTTP chyba', metoda: 'discussions.getComments', parametry: \u2026)\")<\/code><\/p>\n<p>6) Nakonec vypad\u00e1 kombinace Selenium + API jako nejrozumn\u011bj\u0161\u00ed varianta.<\/li>\n<li>Je t\u0159eba zachov\u00e1n\u00ed stavu a restart syst\u00e9mu, zpracov\u00e1n\u00ed mnoha chyb, v\u010detn\u011b nekonzistentn\u00edho chov\u00e1n\u00ed str\u00e1nky - a to jsou chyby, kter\u00e9 jsou docela obt\u00ed\u017en\u00e9 si p\u0159edstavit (pokud profesion\u00e1ln\u011b nep\u00ed\u0161ete parsery, samoz\u0159ejm\u011b).<\/li>\n<\/ul>\n<p>\nPodm\u00ednkov\u00e1 odhadovan\u00e1 doba pro tento \u00fakol bude 3-5kr\u00e1t vy\u0161\u0161\u00ed ne\u017e pro shroma\u017e\u010fov\u00e1n\u00ed dat z Habr. I kdy\u017e v p\u0159\u00edpad\u011b Habr pou\u017e\u00edv\u00e1me p\u0159\u00edm\u00fd p\u0159\u00edstup s parsov\u00e1n\u00edm HTML, zat\u00edmco v p\u0159\u00edpad\u011b OK m\u016f\u017eeme v kritick\u00fdch m\u00edstech pracovat s API.<\/p>\n<h2>Z\u00e1v\u011bry<\/h2>\n<p>\nA\u0165 se na v\u00e1s po\u017eaduje poskytnout odhad \u010dasov\u00e9ho r\u00e1mce \u201ena m\u00edst\u011b\u201c (v\u017edy\u0165 dnes m\u00e1me pl\u00e1nov\u00e1n\u00ed!) pro objemn\u00fd modul pipeline na zpracov\u00e1n\u00ed dat, doba proveden\u00ed se prakticky nikdy ned\u00e1 adekv\u00e1tn\u011b posoudit bez anal\u00fdzy parametr\u016f \u00falohy. <\/p>\n<p>Pokud mluv\u00edme o n\u011bco filozofi\u010dt\u011bji, strategie hodnocen\u00ed v agile se pom\u011brn\u011b dob\u0159e hod\u00ed pro in\u017een\u00fdrsk\u00e9 \u00fakoly, ale s \u00fakoly experiment\u00e1ln\u00edmi a do ur\u010dit\u00e9 m\u00edry \u201etv\u016fr\u010d\u00edmi\u201c a v\u00fdzkumn\u00fdmi, tedy m\u00e9n\u011b p\u0159edv\u00eddateln\u00fdmi, se objevuj\u00ed pot\u00ed\u017ee, jak ukazuj\u00ed p\u0159\u00edklady, kter\u00e9 jsme zde probrali. <\/p>\n<p>Samoz\u0159ejm\u011b, sb\u011br dat je pouze jasn\u00fdm ilustrativn\u00edm p\u0159\u00edkladem \u2014 obvykle se tato \u00faloha zd\u00e1 neuv\u011b\u0159iteln\u011b jednoduch\u00e1 a technicky nen\u00e1ro\u010dn\u00e1, a pr\u00e1v\u011b v detailech se \u010dasto skr\u00fdv\u00e1 \u010f\u00e1bel. A na t\u00e9to \u00faloze se ukazuje cel\u00e1 \u0161k\u00e1la mo\u017en\u00fdch variant toho, co m\u016f\u017ee selhat, a jak moc se m\u016f\u017ee pr\u00e1ce prot\u00e1hnout. <\/p>\n<p>Pokud se letmo pod\u00edv\u00e1te na charakteristiky \u00fakolu bez dal\u0161\u00edch experiment\u016f, Reddit a O.K. vypadaj\u00ed podobn\u011b: existuje API, python wrapper, ale ve skute\u010dnosti je rozd\u00edl obrovsk\u00fd. Pokud to posoud\u00edme podle t\u011bchto parametr\u016f, parsing Habra vypad\u00e1 slo\u017eit\u011bji ne\u017e O.K. \u2014 a ve skute\u010dnosti je to p\u0159esn\u011b naopak, co\u017e lze zjistit proveden\u00edm jednoduch\u00fdch experiment\u016f zam\u011b\u0159en\u00fdch na anal\u00fdzu parametr\u016f \u00falohy.<\/p>\n<p>Podle m\u00fdch zku\u0161enost\u00ed je nej\u00fa\u010dinn\u011bj\u0161\u00ed p\u0159\u00edstup p\u0159ibli\u017en\u00fd odhad \u010dasu, kter\u00fd budete pot\u0159ebovat na p\u0159edb\u011b\u017enou anal\u00fdzu a jednoduch\u00e9 prvn\u00ed experimenty, \u010dten\u00ed dokumentace \u2014 to v\u00e1m umo\u017en\u00ed poskytnout p\u0159esn\u00fd odhad pro celou pr\u00e1ci. V term\u00ednech popul\u00e1rn\u00ed metodologie agile \u2014 pros\u00edm, vytvo\u0159te mi ticket pro \u201eodhady parametr\u016f \u00falohy\u201c, na jeho\u017e z\u00e1klad\u011b mohu poskytnout odhad toho, co je mo\u017en\u00e9 splnit v r\u00e1mci \u201esprintu\u201c a ud\u011blat p\u0159esn\u011bj\u0161\u00ed odhady pro jednotliv\u00e9 \u00fakoly.<\/p>\n<p>Proto se zd\u00e1 b\u00fdt nej\u00fa\u010dinn\u011bj\u0161\u00edm argument, uk\u00e1zat \u201enetechnick\u00e9mu\u201c specialistovi, jak moc se bude li\u0161it \u010das a zdroje v z\u00e1vislosti na parametrech, kter\u00e9 je je\u0161t\u011b t\u0159eba posoudit.<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"http:\/\/ruvds.com\/ru-rub?utm_source=habr&amp;utm_medium=article&amp;utm_campaign=param&amp;utm_content=datascience1#order\"><img decoding=\"async\" alt=\"Co se m\u016f\u017ee pokazit s Data Science? Shrom\u00e1\u017ed\u011bn\u00ed dat\" src=\"\/wp-content\/uploads\/2020\/07\/2ca567a078c8500d37dfcf982e76252c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u0435\u0442 100500 \u043a\u0443\u0440\u0441\u043e\u0432 \u043f\u043e Data Science \u0438 \u0434\u0430\u0432\u043d\u043e \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e, \u0447\u0442\u043e \u0431\u043e\u043b\u044c\u0448\u0435 \u0432\u0441\u0435\u0433\u043e \u0434\u0435\u043d\u0435\u0433 \u0432 Data Science \u043c\u043e\u0436\u043d\u043e \u0437\u0430\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u043d\u043e \u043a\u0443\u0440\u0441\u0430\u043c\u0438 \u043f\u043e Data Science (\u0437\u0430\u0447\u0435\u043c \u043a\u043e\u043f\u0430\u0442\u044c, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0434\u0430\u0432\u0430\u0442\u044c \u043b\u043e\u043f\u0430\u0442\u044b?). \u041e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043c\u0438\u043d\u0443\u0441 \u044d\u0442\u0438\u0445 \u043a\u0443\u0440\u0441\u043e\u0432 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u043e\u043d\u0438 \u043d\u0435 \u0438\u043c\u0435\u044e\u0442 \u043d\u0438\u0447\u0435\u0433\u043e \u043e\u0431\u0449\u0435\u0433\u043e \u0441 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u043e\u0439: \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0434\u0430\u0441\u0442 \u0432\u0430\u043c \u0447\u0438\u0441\u0442\u044b\u0435, \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u0430\u043d\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0432 \u043d\u0443\u0436\u043d\u043e\u043c \u0444\u043e\u0440\u043c\u0430\u0442\u0435. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":89207,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-89206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Co se m\u016f\u017ee pokazit s Data Science? Sb\u011br dat | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-20T05:41:57+00:00","article:modified_time":"2020-07-20T05:41:57+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"89206","title":null,"description":null,"keywords":null,"keyphrases":{"focus":[],"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:51:09","updated":"2026-08-11 12:50:12","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/89206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=89206"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/89206\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/89207"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=89206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=89206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=89206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}