{"id":89206,"date":"2020-07-20T07:41:57","date_gmt":"2020-07-20T05:41:57","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh"},"modified":"2020-07-20T07:41:57","modified_gmt":"2020-07-20T05:41:57","slug":"chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","title":{"rendered":"Ce ar putea merge prost cu Data Science? Colectarea datelor","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\"><img decoding=\"async\" alt=\"Ce ar putea merge prost cu Data Science? Colectarea datelor\" src=\"\/wp-content\/uploads\/2020\/07\/49fb91fc3e45754e437b2c1fc4fec12f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\nAst\u0103zi exist\u0103 100500 de cursuri de Data Science \u0219i este bine cunoscut c\u0103 cei mai mul\u021bi bani din Data Science pot fi c\u00e2\u0219tiga\u021bi tocmai din cursurile de Data Science (de ce s\u0103 te apuci de munc\u0103, c\u00e2nd po\u021bi vinde lope\u021bi?). Principala problem\u0103 a acestor cursuri este c\u0103 nu au nimic \u00een comun cu munca real\u0103: nimeni nu \u00ee\u021bi va oferi date curate \u0219i prelucrate \u00een formatul dorit. \u0218i c\u00e2nd ie\u0219i de la cursuri \u0219i \u00eencepi s\u0103 rezolvi o problem\u0103 real\u0103 \u2013 apar multe nuan\u021be.<\/p>\n<p>De aceea, \u00eencepem o serie de articole intitulate \u201eCe poate merge prost cu Data Science\u201d, bazate pe evenimente reale \u00eent\u00e2mplate cu mine, colegii \u0219i prietenii mei. Vom analiza exemple reale de probleme tipice de Data Science: cum se petrece realmente. Ast\u0103zi vom \u00eencepe cu problema colect\u0103rii datelor.<\/p>\n<p>\u0218i prima dificultate cu care se confrunt\u0103 oamenii atunci c\u00e2nd \u00eencep s\u0103 lucreze cu date reale este, de fapt, colectarea acelor date relevante pentru noi. Mesajul cheie al acestui articol:<\/p>\n<blockquote><p><b>Subestim\u0103m \u00een mod sistematic timpul, resursele \u0219i eforturile necesare pentru colectarea, cur\u0103\u021barea \u0219i preg\u0103tirea datelor.<\/b><\/p><\/blockquote>\n<p>\n\u0218i, mai presus de toate, vom discuta despre ce s\u0103 facem pentru a preveni acest lucru.<\/p>\n<p>Potrivit unor estim\u0103ri, cur\u0103\u021barea, transformarea, procesarea datelor, ingineria caracteristicilor etc. ocup\u0103 80-90% din timp, \u00een timp ce analiza dureaz\u0103 10-20%, de\u0219i practic tot materialul de studiu se concentreaz\u0103 exclusiv pe analiz\u0103.<\/p>\n<p>S\u0103 analiz\u0103m un exemplu tipic de problem\u0103 analitic\u0103 \u00een trei variante \u0219i s\u0103 vedem cum apar \u201ecircumstan\u021bele agravante\u201d.<\/p>\n<p>\u0218i, pentru exemplu, vom analiza asemenea varia\u021bii ale problemei colect\u0103rii datelor \u0219i compar\u0103rii comunit\u0103\u021bilor pentru:<\/p>\n<ol>\n<li>Dou\u0103 subreddit-uri Reddit<\/li>\n<li>Dou\u0103 sec\u021biuni de pe Habr<\/li>\n<li>Dou\u0103 grupuri de pe Odnoklassniki<\/li>\n<\/ol>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Abordarea teoretic\u0103 condi\u021bionat\u0103<\/h2>\n<p>\nDeschide site-ul \u0219i cite\u0219te exemplele; dac\u0103 \u00een\u021belegi, aloc\u0103 c\u00e2teva ore pentru lectur\u0103, c\u00e2teva ore pentru codul din exemple \u0219i depanare. Adaug\u0103 c\u00e2teva ore pentru colectare. \u00centre timp, adaug\u0103 c\u00e2teva ore ca rezerv\u0103 (multiplic\u0103 cu dou\u0103 \u0219i adaug\u0103 N ore).<\/p>\n<p><b>Punctul cheie: estimarea timpului se bazeaz\u0103 pe presupuneri \u0219i conjeturi cu privire la c\u00e2t timp va dura.<\/b><\/p>\n<p>Analiza timpului trebuie s\u0103 \u00eenceap\u0103 cu evaluarea urm\u0103torilor parametri pentru problema condi\u021bionat\u0103 descris\u0103 mai sus:<\/p>\n<ul>\n<li>Care este dimensiunea datelor \u0219i c\u00e2t trebuie s\u0103 adun\u0103m fizic (*vezi mai jos*).<\/li>\n<li>Care este timpul de colectare a unei \u00eenregistr\u0103ri \u0219i c\u00e2t trebuie s\u0103 a\u0219tep\u021bi \u00eenainte de a putea colecta a doua.<\/li>\n<li>Implementa\u021bi un cod care s\u0103 p\u0103streze starea \u0219i s\u0103 \u00eenceap\u0103 un restart, c\u00e2nd (nu dac\u0103) totul va c\u0103dea.<\/li>\n<li>Determina\u021bi dac\u0103 avem nevoie de autentificare \u0219i aloca\u021bi timp pentru a ob\u021bine accesul prin API.<\/li>\n<li>Include\u021bi num\u0103rul de erori ca o func\u021bie a complexit\u0103\u021bii datelor \u2014 evalua\u021bi \u00een func\u021bie de sarcina specific\u0103: structura, c\u00e2te transform\u0103ri, ce \u0219i cum extragem.<\/li>\n<li>Include\u021bi erorile de re\u021bea \u0219i problemele cu comportamentele neconven\u021bionale ale proiectului.<\/li>\n<li>Evalua\u021bi dac\u0103 func\u021biile necesare se afl\u0103 \u00een documenta\u021bie \u0219i, dac\u0103 nu, atunci c\u00e2t \u0219i cum este necesar pentru o solu\u021bie alternativ\u0103.<\/li>\n<\/ul>\n<p>\nCel mai important, pentru a evalua timpul \u2014 de fapt, trebuie s\u0103 investi\u021bi timp \u0219i efort \u00een \u00abexplorarea situa\u021biei\u00bb \u2014 doar atunci planificarea dvs. va fi adecvat\u0103. A\u0219adar, oric\u00e2t de mult a\u021bi fi \u00eempin\u0219i s\u0103 spune\u021bi \u00abc\u00e2t timp este necesar pentru a colecta datele\u00bb \u2014 acorda\u021bi-v\u0103 timp pentru o analiz\u0103 preliminar\u0103 \u0219i argumenta\u021bi c\u0103 timpul va varia \u00een func\u021bie de parametrii reali ai sarcinii.<\/p>\n<p>\u0218i acum v\u0103 vom demonstra exemple specificate, unde astfel de parametri vor varia.<\/p>\n<p><b>Punctul cheie: evaluarea este bazat\u0103 pe analiza factorilor cheie care influen\u021beaz\u0103 volumul \u0219i complexitatea muncii.<\/b><\/p>\n<p>O evaluare bazat\u0103 pe presupuneri este o abordare bun\u0103 atunci c\u00e2nd elementele func\u021bionale sunt destul de mici \u0219i nu sunt at\u00e2t de multe factori care pot influen\u021ba semnificativ structura sarcinii. Dar \u00een cazul mai multor sarcini de Data Science, ace\u0219ti factori devin extrem de numero\u0219i \u0219i abordarea devine inadecvat\u0103.<\/p>\n<h2>Compararea comunit\u0103\u021bilor Reddit<\/h2>\n<p>\nS\u0103 \u00eencepem cu cea mai simpl\u0103 situa\u021bie (a\u0219a cum se va dovedi ulterior). De fapt, dac\u0103 suntem complet one\u0219ti, avem practic un caz ideal, s\u0103 verific\u0103m lista noastr\u0103 de verificare a complexit\u0103\u021bii:<\/p>\n<ul>\n<li>Exist\u0103 un API clar, bine documentat \u0219i \u00een\u021beleg\u0103tor.<\/li>\n<li>Extrem de simplu \u0219i, cel mai important, ob\u021binem token-ul automat.<\/li>\n<li>Exist\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/praw-dev\/praw\">python wrapper<\/a><\/noindex> \u2014 cu o mul\u021bime de exemple.<\/li>\n<li>Comunitatea care se ocup\u0103 cu analiza \u0219i colectarea datelor pe Reddit (inclusiv videoclipuri YouTube explic\u00e2nd cum s\u0103 utiliza\u021bi python wrapper) <noindex><a rel=\"nofollow\" href=\"https:\/\/www.programcreek.com\/python\/example\/86599\/praw.Reddit\">iat\u0103, de exemplu,<\/a><\/noindex>.<\/li>\n<li>Metodele de care avem nevoie exist\u0103 probabil \u00een API. Mai mult, codul arat\u0103 compact \u0219i curat, mai jos este un exemplu de func\u021bie care colecteaz\u0103 comentarii la postare.<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"python\">def get_comments(submission_id):\n    reddit = Reddit(check_for_updates=False, user_agent=AGENT)\n    submission = reddit.submission(id=submission_id)\n    more_comments = submission.comments.replace_more()\n    if more_comments:\n        skipped_comments = sum(x.count for x in more_comments)\n        logger.debug('Skipped %d MoreComments (%d comments)',\n                     len(more_comments), skipped_comments)\n    return submission.comments.list()\n<\/code><\/pre>\n<p><i>Preluat din <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/timendum\/reddit-utilities\/blob\/master\/thread-export.py\">aceast\u0103<\/a><\/noindex> colec\u021bii de utilitare convenabile pentru ambalare.<\/i><\/p>\n<p>De\u0219i avem \u00een fa\u021b\u0103 cea mai bun\u0103 situa\u021bie, trebuie s\u0103 lu\u0103m \u00een considerare o serie de factori importan\u021bi din via\u021ba real\u0103:<\/p>\n<ul>\n<li>Limitele API-ului \u2014 suntem nevoi\u021bi s\u0103 lu\u0103m datele \u00een batch-uri (s\u0103 a\u0219tept\u0103m \u00eentre cereri etc.).<\/li>\n<li>Timpul de colectare \u2014 pentru o analiz\u0103 \u0219i comparare complet\u0103, va trebui s\u0103 aloc\u0103m un timp considerabil doar pentru ca spider-ul s\u0103 traverseze subreddit-ul.<\/li>\n<li>Botul trebuie s\u0103 ruleze pe server \u2014 nu po\u021bi pur \u0219i simplu s\u0103-l porne\u0219ti pe laptop, s\u0103-l pui \u00een rucsac \u0219i s\u0103 pleci \u00een afaceri. A\u0219a c\u0103 am rulat totul pe VPS. Cu codul promo\u021bional habrahabr10 po\u021bi economisi \u00eenc\u0103 10% din cost.<\/li>\n<li>Accesibilitatea fizic\u0103 a unor date (care sunt vizibile doar pentru administratori sau care sunt prea greu de colectat) \u2014 trebuie s\u0103 \u021bii cont de asta, nu toate datele pot fi colectate \u00eentr-un timp rezonabil.<\/li>\n<li>Erorile de func\u021bionare a re\u021belei: lucrul cu re\u021beaua poate fi o provocare.<\/li>\n<li>Acestea sunt date reale \u2014 nu sunt niciodat\u0103 curate.<\/li>\n<\/ul>\n<p>\nDesigur, este necesar s\u0103 planifici aceste nuan\u021be \u00een dezvoltare. Orele\/zilele concrete depind de experien\u021ba de dezvoltare sau de experien\u021ba \u00een lucrul cu sarcini similare, totu\u0219i vedem c\u0103 aceasta este o sarcin\u0103 pur tehnic\u0103 \u0219i nu necesit\u0103 mi\u0219c\u0103ri suplimentare pentru a fi rezolvat\u0103 \u2014 totul poate fi evaluat foarte bine, detaliat \u0219i realizat.<\/p>\n<h2>Compararea sec\u021biunilor Habr<\/h2>\n<p>\nTrecem la o situa\u021bie mai interesant\u0103 \u0219i mai non-trivial\u0103 compar\u00e2nd fluxurile \u0219i\/sau sec\u021biunile Habr.<\/p>\n<p>S\u0103 verific\u0103m lista noastr\u0103 de control pentru dificult\u0103\u021bi \u2014 aici, pentru a \u00een\u021belege fiecare punct, va trebui s\u0103 interac\u021bion\u0103m pu\u021bin cu sarcina \u00een sine \u0219i s\u0103 experiment\u0103m.<\/p>\n<ul>\n<li>La \u00eenceput crezi c\u0103 exist\u0103 API, dar nu este. Da, Habr are API, dar acesta nu este disponibil pentru utilizatori (sau poate nu func\u021bioneaz\u0103 deloc).<\/li>\n<li>Apoi pur \u0219i simplu \u00eencepi s\u0103 parsezi HTML \u2014 \u201eimport requests\u201d, ce ar putea s\u0103 mearg\u0103 r\u0103u?<\/li>\n<li>\u0218i cum se face de fapt parsarea? Cea mai simpl\u0103 \u0219i frecvent utilizat\u0103 abordare este s\u0103 iterezi prin ID-uri, observ\u0103m c\u0103 nu este cea mai eficient\u0103 \u0219i va trebui s\u0103 gestion\u0103m diverse cazuri \u2014 ca exemplu, densitatea real\u0103 a ID-urilor printre toate cele existente.\n<p><img decoding=\"async\" alt=\"Ce ar putea merge prost cu Data Science? Colectarea datelor\" src=\"\/wp-content\/uploads\/2020\/07\/2c67fc14f672e92979edd617314210da.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Preluat din <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/218607\/\">aceast\u0103<\/a><\/noindex> articolului.<\/i><\/li>\n<li>Datele brute, \u00eenf\u0103\u0219urate \u00een HTML pe tot internetul \u2013 reprezint\u0103 o adev\u0103rat\u0103 durere. De exemplu, vrei s\u0103 colectezi \u0219i s\u0103 salvezi ratingul unui articol: ai extras score din html \u0219i ai decis s\u0103-l salvezi ca num\u0103r pentru proces\u0103ri ulterioare:\u00a0\n<p>1) int(score) genereaz\u0103 o eroare: pe Habr minusul, cum ar fi \u00een \u0219irul &quot;\u20135&quot; \u2014 acesta este un dash scurt, nu un semn de minus (ciudat, nu-i a\u0219a?), a\u0219a c\u0103 la un moment dat a fost nevoie s\u0103 revizuiesc parserul cu o astfel de solu\u021bie groaznic\u0103.<\/p>\n<pre><code class=\"python\">try:\n      score_txt = post.find(class_=&quot;score&quot;).text.replace(u&quot;\u2013&quot;,&quot;-&quot;).replace(u&quot;+&quot;,&quot;+&quot;)\n      score = int(score_txt)\n      if check_date(date):\n        post_score += score\n<\/code><\/pre>\n<p>\nDatele, plusurile \u0219i minusurile pot s\u0103 nu existe deloc (a\u0219a cum vedem mai sus \u00een func\u021bia check_date \u0219i s-a \u00eent\u00e2mplat).<\/p>\n<p>2) Caracteristicile speciale neescapate \u2013 ele vor veni, trebuie s\u0103 fim preg\u0103ti\u021bi.<\/p>\n<p>3) Structura se schimb\u0103 \u00een func\u021bie de tipul post\u0103rii.<\/p>\n<p>4) Post\u0103rile mai vechi pot avea **o structur\u0103 ciudat\u0103**.<\/li>\n<li>Practica gestion\u0103rii erorilor \u0219i a ceea ce poate sau nu poate s\u0103 se \u00eent\u00e2mple va trebui tratat\u0103, iar nu po\u021bi anticipa sigur ce va merge prost \u0219i cum ar putea s\u0103 arate structura sau ce anume va c\u0103dea \u2013 va trebui pur \u0219i simplu s\u0103 \u00eencerci \u0219i s\u0103 iei \u00een considerare erorile pe care le arunc\u0103 parserul.<\/li>\n<li>Apoi, \u00ee\u021bi dai seama c\u0103 trebuie s\u0103 parsezi \u00een mai multe fire, altfel parcarea \u00eentr-un singur fir va dura 30+ ore (asta este pur \u0219i simplu timpul de execu\u021bie al parserului func\u021bional pe un singur fir, care doarme \u0219i nu este supus la niciun ban). \u00cen <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/276383\/\">aceast\u0103<\/a><\/noindex> articol, aceasta a condus \u00een vreun moment la o schem\u0103 similar\u0103:<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"Ce ar putea merge prost cu Data Science? Colectarea datelor\" src=\"\/wp-content\/uploads\/2020\/07\/9ad1853f3ee2c9321ee9f1a5e3efea7d.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPrin urmare, lista de verficare conform dificult\u0103\u021bii:<\/p>\n<ul>\n<li>Lucrul cu re\u021beaua \u0219i parcarea html cu itera\u021bie \u0219i parcurgerea pe ID.<\/li>\n<li>Documentele au o structur\u0103 heterogen\u0103.<\/li>\n<li>Multe locuri unde codul poate c\u0103dea cu u\u0219urin\u021b\u0103.<\/li>\n<li>Este necesar s\u0103 scrii || cod.<\/li>\n<li>Lipsesc documenta\u021bia, exemplele de cod \u0219i\/sau comunitatea necesar\u0103.<\/li>\n<\/ul>\n<p>\nEvaluarea condi\u021bionat\u0103 a timpului pentru aceast\u0103 sarcin\u0103 va fi de 3-5 ori mai mare dec\u00e2t pentru colectarea datelor de pe Reddit.<\/p>\n<h2>Compararea grupurilor de pe Odnoklassniki<\/h2>\n<p>\nS\u0103 trecem la cel mai tehnic caz interesant dintre cele descrise. Pentru mine a fost captivant tocmai pentru c\u0103, la prima vedere, pare destul de trivial, dar de fapt nu este deloc \u2013 de \u00eendat\u0103 ce dai cu stick-ul \u00een el.<\/p>\n<p>Va \u00eencepe cu lista noastr\u0103 de verificare a dificult\u0103\u021bii \u0219i s\u0103 not\u0103m c\u0103 multe dintre ele se vor dovedi a fi mult mai complicate dec\u00e2t par la \u00eenceput:<\/p>\n<ul>\n<li>Exist\u0103 API, dar majoritatea func\u021biilor necesare lipsesc aproape complet.<\/li>\n<li>Pentru anumite func\u021bii trebuie s\u0103 ceri acces prin email, adic\u0103 acordarea accesului nu este instantanee.<\/li>\n<li>Documenta\u021bia este extrem de slab\u0103 (\u00eencep\u00e2nd cu amestecarea constant\u0103 a termenilor ru\u0219i \u0219i engleze\u0219ti, \u00eentr-o manier\u0103 complet inconsistent\u0103 \u2014 uneori trebuie doar s\u0103 ghice\u0219ti ce vor de la tine) \u0219i, mai mult dec\u00e2t at\u00e2t, nu este adaptat\u0103 designului pentru extragerea datelor, de exemplu, <noindex><a rel=\"nofollow\" href=\"https:\/\/apiok.ru\/dev\/methods\/rest\/discussions\/discussions.getComments\">func\u021bia de care avem nevoie<\/a><\/noindex>.<\/li>\n<li>Cere sesiune \u00een documenta\u021bie, iar \u00een realitate nu o folose\u0219te \u2014 \u0219i nu exist\u0103 nicio modalitate de a \u00een\u021belege toate nuan\u021bele modurilor API, \u00een afar\u0103 de a \u00eencerca \u0219i a spera c\u0103 ceva va func\u021biona.<\/li>\n<li>Lipsesc exemple \u0219i comunitatea, singura surs\u0103 de informare fiind un mic <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/alternativshik\/python-odnoklassniki\">wrapper<\/a><\/noindex> \u00een Python (f\u0103r\u0103 multe exemple de utilizare).<\/li>\n<li>Cel mai practic pare a fi Selenium, deoarece multe date necesare sunt blocate.<br \/>\n1) Adic\u0103, autentificarea se face printr-un utilizator fictiv (\u0219i \u00eenregistrarea manual\u0103).<\/p>\n<p>2) Totu\u0219i, cu Selenium nu exist\u0103 garan\u021bii pentru o func\u021bionare corect\u0103 \u0219i repetabil\u0103 (cel pu\u021bin \u00een cazul ok.ru cu siguran\u021b\u0103).<\/p>\n<p>3) Site-ul Ok.ru con\u021bine erori JavaScript \u0219i uneori se comport\u0103 ciudat \u0219i inconsistent.<\/p>\n<p>4) Trebuie s\u0103 te ocupi de paginare, \u00eenc\u0103rcarea elementelor etc\u2026<\/p>\n<p>5) Erorile API returnate de wrapper trebuie procesate pe cale improvizat\u0103, de exemplu, a\u0219a (un fragment de cod experimental):<\/p>\n<pre><code class=\"python\">def get_comments(args, context, discussions):\n    pause = 1\n    if args.extract_comments:\n        all_comments = set()\n#are sens s\u0103 \u021binem eviden\u021ba discu\u021biilor deja procesate\n        for discussion in tqdm(discussions): \n            try:\n                comments = get_comments_from_discussion_via_api(context, discussion)\n            except odnoklassniki.api.OdnoklassnikiError as e:\n                if &quot;NOT_FOUND&quot; in str(e):\n                    comments = set()\n                else:\n                    print(e)\n                    bp()\n                    pass\n            all_comments |= comments\n            time.sleep(pause)\n        return all_comments\n<\/code><\/pre>\n<p>\nCea mai drag\u0103 eroare a mea a fost:<\/p>\n<p><code>OdnoklassnikiError(&quot;Eroare(cod: 'None', descriere: 'eroare HTTP', metod\u0103: 'discussions.getComments', params: \u2026)&quot;)<\/code><\/p>\n<p>6) \u00cen cele din urm\u0103, combina\u021bia Selenium + API pare a fi cea mai ra\u021bional\u0103 op\u021biune.<\/li>\n<li>Este necesar\u0103 men\u021binerea st\u0103rii \u0219i repornirea sistemului, gestionarea multor erori, inclusiv comportamente inconsistente ale site-ului \u2014 iar aceste erori sunt destul de greu de imaginat (dac\u0103 nu scrii profesional parsere, evident).<\/li>\n<\/ul>\n<p>\nEstimarea condi\u021bionat\u0103 a timpului pentru aceast\u0103 sarcin\u0103 va fi de 3-5 ori mai mare dec\u00e2t pentru extragerea datelor de pe Habr. De\u0219i \u00een cazul Habr utiliz\u0103m o abordare frontal\u0103 prin parsarea HTML-ului, \u00een cazul OK putem lucra \u00een locuri critice cu API.<\/p>\n<h2>Conclusions<\/h2>\n<p>\nIndiferent de c\u00e2t de mult ar solicita evaluarea termenelor \"la fa\u021ba locului\" (avem planificare ast\u0103zi!), volumul modulului pipeline-ului de procesare a datelor este practic imposibil de evaluat calitativ f\u0103r\u0103 o analiz\u0103 a parametrilor sarcinii. <\/p>\n<p>Dac\u0103 vorbim pu\u021bin mai filozofic, strategiile de evaluare \u00een agile se potrivesc bine pentru sarcini ingineresti, dar pentru sarcini mai experimentale \u0219i, \u00eentr-un anumit sens, \"crea\u021bionale\" \u0219i de cercetare, adic\u0103 mai pu\u021bin previzibile, apar dificult\u0103\u021bi, a\u0219a cum sunt exemplele pe care le-am discutat aici. <\/p>\n<p>Desigur, colectarea datelor este un exemplu ilustrativ clar - de obicei, aceast\u0103 sarcin\u0103 pare incredibil de simpl\u0103 \u0219i tehnic necomplicat\u0103, iar tocmai \u00een detalii se ascunde adesea diavolul. Aceast\u0103 sarcin\u0103 ilustreaz\u0103 \u00eentregul spectru de variante posibile a ceea ce poate merge prost \u0219i c\u00e2t de mult poate dura efectiv munca. <\/p>\n<p>Dac\u0103 arunc\u0103m o privire pe specifica\u021biile sarcinii f\u0103r\u0103 experimente suplimentare, Reddit \u0219i OK par asem\u0103n\u0103toare: exist\u0103 API, wrapper python, dar esen\u021bial, diferen\u021ba este enorm\u0103. Dac\u0103 judec\u0103m dup\u0103 ace\u0219ti parametri, parsingul Habr pare mai complicat dec\u00e2t OK - dar \u00een practic\u0103 este exact invers \u0219i acest lucru poate fi descoperit efectu\u00e2nd experimente simple pentru analiza parametrilor sarcinii.<\/p>\n<p>Din experien\u021ba mea, cea mai eficient\u0103 abordare este o estimare aproximativ\u0103 a timpului necesar pentru analiza preliminar\u0103 \u0219i primele experimente simple, citirea documenta\u021biei - acestea v\u0103 vor permite s\u0103 oferi\u021bi o evaluare exact\u0103 pentru \u00eentreaga munc\u0103. \u00cen termeni de metodologie popular\u0103 agile - v\u0103 rog s\u0103 deschide\u021bi un tichet pentru \u201eevaluarea parametrilor sarcinii\u201d, pe baza c\u0103ruia pot oferi o evaluare a ceea ce este posibil s\u0103 se finalizeze \u00een cadrul \u201esprintului\u201d \u0219i s\u0103 ofer o estimare mai precis\u0103 pentru fiecare sarcin\u0103.<\/p>\n<p>Prin urmare, cel mai eficient, pare a fi argumentul care ar putea ar\u0103ta unui specialist \"non-tehnic\" c\u00e2t de mult va varia timpul \u0219i resursele \u00een func\u021bie de parametrii care urmeaz\u0103 s\u0103 fie evalua\u021bi.<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"http:\/\/ruvds.com\/ru-rub?utm_source=habr&amp;utm_medium=article&amp;utm_campaign=param&amp;utm_content=datascience1#order\"><img decoding=\"async\" alt=\"Ce ar putea merge prost cu Data Science? Colectarea datelor\" src=\"\/wp-content\/uploads\/2020\/07\/2ca567a078c8500d37dfcf982e76252c.jpeg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><br \/>\n<br \/>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ruvds\/blog\/510944\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u0435\u0442 100500 \u043a\u0443\u0440\u0441\u043e\u0432 \u043f\u043e Data Science \u0438 \u0434\u0430\u0432\u043d\u043e \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e, \u0447\u0442\u043e \u0431\u043e\u043b\u044c\u0448\u0435 \u0432\u0441\u0435\u0433\u043e \u0434\u0435\u043d\u0435\u0433 \u0432 Data Science \u043c\u043e\u0436\u043d\u043e \u0437\u0430\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u043d\u043e \u043a\u0443\u0440\u0441\u0430\u043c\u0438 \u043f\u043e Data Science (\u0437\u0430\u0447\u0435\u043c \u043a\u043e\u043f\u0430\u0442\u044c, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0434\u0430\u0432\u0430\u0442\u044c \u043b\u043e\u043f\u0430\u0442\u044b?). \u041e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043c\u0438\u043d\u0443\u0441 \u044d\u0442\u0438\u0445 \u043a\u0443\u0440\u0441\u043e\u0432 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u043e\u043d\u0438 \u043d\u0435 \u0438\u043c\u0435\u044e\u0442 \u043d\u0438\u0447\u0435\u0433\u043e \u043e\u0431\u0449\u0435\u0433\u043e \u0441 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u043e\u0439: \u043d\u0438\u043a\u0442\u043e \u043d\u0435 \u0434\u0430\u0441\u0442 \u0432\u0430\u043c \u0447\u0438\u0441\u0442\u044b\u0435, \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u0430\u043d\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0432 \u043d\u0443\u0436\u043d\u043e\u043c \u0444\u043e\u0440\u043c\u0430\u0442\u0435. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":89207,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-89206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-20T05:41:57+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Ce ar putea merge prost cu Data Science? Colectarea datelor | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u043f\u043e\u0439\u0442\u0438 \u043d\u0435 \u0442\u0430\u043a \u0441 Data Science? \u0421\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 | ProHoster","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-20T05:41:57+00:00","article:modified_time":"2020-07-20T05:41:57+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"89206","title":null,"description":null,"keywords":null,"keyphrases":{"focus":[],"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:51:09","updated":"2026-08-11 12:50:12","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/89206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=89206"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/89206\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/89207"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=89206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=89206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=89206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}