{"id":89206,"date":"2020-07-20T07:41:57","date_gmt":"2020-07-20T05:41:57","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh"},"modified":"2020-07-20T07:41:57","modified_gmt":"2020-07-20T05:41:57","slug":"chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/chto-mozhet-pojti-ne-tak-s-data-science-sbor-dannyh","title":{"rendered":"What can go wrong with Data Science? Data Collection","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"What<\/a><\/noindex>
\nToday, there are countless courses on Data Science, and it\u2019s well-known that the most money in Data Science can be earned through Data Science courses (why dig when you can sell shovels?). The main downside of these courses is that they are not related to real work: no one will provide you with clean, processed data in the required format. When you finish the courses and start tackling real problems, many nuances come to light.<\/p>\n

That\u2019s why we\u2019re starting a series of notes titled 'What Can Go Wrong with Data Science,' based on real events that have happened to me, my friends, and colleagues. We'll analyze typical Data Science tasks with real examples: how it actually happens. We\u2019ll begin today with the task of data collection.<\/p>\n

The first hurdle people encounter when they start working with real data is actually collecting the relevant data we need. The key message of this article is:<\/p>\n

We systematically underestimate the time, resources, and effort needed for data collection, cleaning, and preparation.<\/b><\/p><\/blockquote>\n

\nAnd importantly, we\u2019ll discuss what to do to avoid this.<\/p>\n

According to various estimates, cleaning, transformation, data processing, feature engineering, etc. take up 80-90% of the time, while analysis only takes 10-20%, yet almost all educational materials focus solely on analysis.<\/p>\n

Let's break down a typical simple analytical task in three variations and see what 'aggravating circumstances' arise.<\/p>\n

For the example, we will again consider similar variations of the data collection task and community comparison for:<\/p>\n

    \n
  1. Two Reddit subreddits<\/li>\n
  2. Two sections of Habr<\/li>\n
  3. Two Odnoklassniki groups<\/li>\n<\/ol>\n

    <\/a><\/noindex><\/p>\n

    A conditional theoretical approach<\/h2>\n

    \nOpen the website and read examples, allocate several hours for reading, several hours for coding based on examples, and debugging. Add a few hours for collection. Throw in some extra hours for good measure (multiply by two and add N hours).<\/p>\n

    The key point: the time estimation is based on assumptions and guesses about how long this will take.<\/b><\/p>\n

    The time analysis should start with assessing the following parameters for the conditional task described above:<\/p>\n