🥇What can go wrong with Data Science? Data Collection

Today, there are countless courses on Data Science, and it’s well-known that the most money in Data Science can be earned through Data Science courses (why dig when you can sell shovels?). The main downside of these courses is that they are not related to real work: no one will provide you with clean, processed data in the required format. When you finish the courses and start tackling real problems, many nuances come to light.

That’s why we’re starting a series of notes titled 'What Can Go Wrong with Data Science,' based on real events that have happened to me, my friends, and colleagues. We'll analyze typical Data Science tasks with real examples: how it actually happens. We’ll begin today with the task of data collection.

The first hurdle people encounter when they start working with real data is actually collecting the relevant data we need. The key message of this article is:

We systematically underestimate the time, resources, and effort needed for data collection, cleaning, and preparation.

And importantly, we’ll discuss what to do to avoid this.

According to various estimates, cleaning, transformation, data processing, feature engineering, etc. take up 80-90% of the time, while analysis only takes 10-20%, yet almost all educational materials focus solely on analysis.

Let's break down a typical simple analytical task in three variations and see what 'aggravating circumstances' arise.

For the example, we will again consider similar variations of the data collection task and community comparison for:

Two Reddit subreddits
Two sections of Habr
Two Odnoklassniki groups

A conditional theoretical approach

Open the website and read examples, allocate several hours for reading, several hours for coding based on examples, and debugging. Add a few hours for collection. Throw in some extra hours for good measure (multiply by two and add N hours).

The key point: the time estimation is based on assumptions and guesses about how long this will take.

The time analysis should start with assessing the following parameters for the conditional task described above:

What is the data size and how much needs to be physically collected (*see below*).
What is the time to collect one record and how long do you need to wait before collecting the second.
Implement code that preserves state and starts a restart when (not if) everything fails.
Determine if we need authentication and set aside time to gain access via API.
Include the number of errors as a function of data complexity — evaluate based on a specific task: structure, how many transformations, what and how we extract.
Include network errors and issues with non-standard project behavior.
Assess whether the required functions are in the documentation, and if not, how much time is needed for a workaround.

The most important factor for estimating time — you actually need to spend time and effort on 'reconnaissance in action' — only then will your planning be adequate. So no matter how much you are pushed to say 'how long will it take to collect data' — make sure to allocate time for preliminary analysis and justify it by how much time will vary based on real task parameters.

Now we will demonstrate specific examples where such parameters will change.

Key point: estimation is based on the analysis of key factors affecting the volume and complexity of work.

Estimation based on guesswork is a good approach when functional elements are relatively small and there are not many factors that can significantly affect the structure of the task. However, in the case of a number of Data Science tasks, such factors become extremely numerous, and this approach becomes inadequate.

Comparison of Reddit communities

Let's start with the simplest case (as it will turn out later). Honestly, we have an almost ideal case; let's check our complexity checklist:

There is a neat, clear, and documented API.
It is extremely simple, and most importantly, tokens are generated automatically.
There is python wrapper — with plenty of examples.
A community that focuses on analysis and data collection on Reddit (including YouTube videos explaining how to use the python wrapper) for example,.
The methods we need most likely exist in the API. Moreover, the code looks compact and clean; below is an example of a function that collects comments on a post.

def get_comments(submission_id):
    reddit = Reddit(check_for_updates=False, user_agent=AGENT)
    submission = reddit.submission(id=submission_id)
    more_comments = submission.comments.replace_more()
    if more_comments:
        skipped_comments = sum(x.count for x in more_comments)
        logger.debug('Skipped %d MoreComments (%d comments)',
                     len(more_comments), skipped_comments)
    return submission.comments.list()

Taken from this collection of convenient wrapper utilities.

Despite the fact that we have the best scenario here, we still need to consider a number of important factors from real life:

API limits — we are forced to retrieve data in batches (waiting between requests, etc.).
Collection time — for a complete analysis and comparison, you need to allocate significant time just for the spider to crawl through the subreddit.
The bot must run on a server — you can't just launch it on your laptop, pack it into your backpack, and go about your business. That’s why I ran everything on a VPS. You can save an additional 10% using the promo code habrahabr10.
The physical inaccessibility of some data (it’s visible only to admins or is too complex to gather) — this needs to be taken into account; not all data can be collected in a reasonable timeframe.
Network errors: working with the network is a pain.
These are real live data — they are never clean.

Of course, the indicated nuances need to be accounted for in the development. The specific hours/days depend on the development experience or previous work on similar tasks; however, we see that this task is purely engineering and does not require any extra movements to solve — everything can be evaluated, planned, and executed very well.

Comparison of Habr sections

Moving on to a more interesting and non-trivial case of comparing flows and/or sections of Habr.

Let's check our complexity checklist — here, to understand each point, you’ll need to fiddle a bit with the task itself and experiment.

At first, you think there’s an API, but there isn’t. Yes, Habr has an API, but it’s only available to admins (or maybe it doesn’t work at all).
Then you just start parsing HTML — "import requests", what can go wrong?
And how do you even parse? The simplest and most commonly used approach is to iterate by ID; we note that it's not the most efficient and you'll have to handle different cases — here's an example of the density of real IDs among all existing ones.

Taken from this articles.
Raw data wrapped in HTML across the network can be a pain. For example, if you want to collect and store the rating of an article: you extracted the score from HTML and decided to save it as a number for further processing.
1) int(score) throws an error: on Habr, a minus, like in the string "–5" — is an en dash, not a minus sign (unexpected, right?), so at some point, you had to revive the parser with such a terrible fix.
```
try:
      score_txt = post.find(class_="score").text.replace(u"–","-").replace(u"+","+")
      score = int(score_txt)
      if check_date(date):
        post_score += score
```
There may not be dates, pluses, or minuses at all (as we see above with the check_date function, this has happened).
2) Unescaped special characters — they will come, you need to be prepared.
3) The structure changes depending on the type of post.
4) Old posts may have **strange structures**.
Essentially, error handling and what might or might not happen must be managed, and you cannot predict exactly what will go wrong and what the structure could be like, and where something might break — you will just have to try and account for the errors thrown by the parser.
Then you realize that you need to parse in multiple threads, otherwise parsing in one thread will take 30+ hours (this is purely the execution time of a working single-threaded parser that is sleeping and not subject to any bans). In this the article, this led to a pattern like this at some point:

Thus, the checklist of complexity:

Working with the network and parsing HTML with iteration and ID enumeration.
Documents have an irregular structure.
There are many places where the code can easily fail.
You need to write || code.
There is a lack of necessary documentation, code examples, and/or community.

The estimated time for this task will be 3-5 times higher than for gathering data from Reddit.

Comparison of Vkontakte groups

Let's move on to the most technically interesting case described. For me, it was interesting because at first glance, it seems quite trivial, but it turns out not to be — as soon as you poke it with a stick.

It will start with our checklist of complexity, and we note that many of them will turn out to be much more complicated than they appear at first:

The API exists, but it almost completely lacks the necessary functions.
For certain functions, you need to request access via email, meaning that access issuance is not instantaneous.
It is terribly documented (to begin with, Russian and English terms are mixed everywhere, and completely inconsistently — sometimes you just have to guess what is expected of you) and, moreover, it is not designed for data retrieval, for example, the function we need.
Documentation requires sessions, yet in practice it doesn’t use them — and there’s no way to figure out all the intricacies of the API modes except by poking around and hoping something will work.
There are no examples and community support, the only reference point for gathering information is a small wrapper in Python (without much usage examples).
The most viable option seems to be Selenium, as many necessary data are locked.
1) That is, authorization is done through a fictitious user (and registration by hand).
2) However, with Selenium, there are no guarantees of correct and repeatable operation (at least in the case of ok.ru for sure).
3) The Ok.ru site contains JavaScript errors and occasionally behaves oddly and inconsistently.
4) Pagination, element loading, etc. have to be managed…
5) API errors returned by the wrapper will need to be awkwardly handled, for example, like this (a snippet of experimental code):
```
def get_comments(args, context, discussions):
    pause = 1
    if args.extract_comments:
        all_comments = set()
#makes sense to keep track of already processed discussions
        for discussion in tqdm(discussions): 
            try:
                comments = get_comments_from_discussion_via_api(context, discussion)
            except odnoklassniki.api.OdnoklassnikiError as e:
                if "NOT_FOUND" in str(e):
                    comments = set()
                else:
                    print(e)
                    bp()
                    pass
            all_comments |= comments
            time.sleep(pause)
        return all_comments
```
My favorite error was:
OdnoklassnikiError("Error(code: 'None', description: 'HTTP error', method: 'discussions.getComments', params: …)")
6) Ultimately, the Selenium + API option looks like the most rational choice.
State preservation and system restart are required, handling numerous errors, including the website's inconsistent behavior — and these errors are quite difficult to envision (unless you are a professional parser developer, of course).

The estimated time for this task will be 3 to 5 times higher than that for data collection from Habr. Despite the fact that in the case of Habr we use a blunt approach with HTML parsing, while in the case of OK we can work with the API in critical areas.

Conclusions

No matter how much you're asked to estimate the timelines "on-site" (after all, it's planning time today!), evaluating the duration of a substantial data processing pipeline module is almost never feasible without analysis of the task parameters.

If we speak a bit more philosophically, agile estimation strategies work well for engineering tasks, but challenges arise with more experimental and, in some sense, creative and research-oriented tasks, i.e., those that are less predictable. These challenges are similar to the examples discussed here.

Certainly, data collection is just a vivid illustrative example — this task usually seems incredibly simple and technically straightforward, yet the devil is often in the details. This task vividly demonstrates the full spectrum of possible variations of what can go wrong and how much the work may drag on.

If we glance at the task specifications without additional experiments, Reddit and OK seem similar: there’s an API, a Python wrapper, but in essence, the difference is enormous. Judging by these parameters, parsing Habr seems more complex than OK — whereas, in practice, it is quite the opposite, and this can be determined through simple experimental analyses of the task parameters.

In my experience, the most effective approach is a rough time estimate for the preliminary analysis and the initial simple experiments, along with reading documentation — these will allow you to provide an accurate estimate for the entire job. In terms of the popular agile methodology — I ask for a ticket to be opened for "task parameter estimation," based on which I can assess what might be accomplished within the sprint and give a more precise estimate for each task.

Therefore, the most effective argument seems to be one that demonstrates to a "non-technical" specialist how significantly time and resources can vary based on parameters that still need to be evaluated.

Source: habr.com