🥇How to Distinguish a Fraud from Data Science?

You may have heard about analysts, machine learning specialists, and artificial intelligence experts, but have you heard about those who are unjustly overpaid? Meet data charlatans! These tricksters, lured by lucrative jobs, create a bad reputation for real data professionals. In this article, we will explore how to expose such individuals.

Data charlatans are everywhere

Data charlatans are so adept at hiding in plain sight that you might be one of them, without even realizing it. Chances are, your organization has been sheltering these tricksters for years, but here’s the good news: they are easy to identify if you know what to look for.
The first warning sign is a misunderstanding that analytics and statistics are very different disciplines. I will explain this further.

Different disciplines

Statisticians are trained to make inferences about what lies beyond their data, while analysts are trained to study the contents of a data set. In other words, analysts draw conclusions about what is in their data, while statisticians make conclusions about what is not in the data. Analysts help pose good questions (formulate hypotheses), while statisticians help obtain good answers (test hypotheses).

There are also fanciful hybrid roles where a person tries to straddle both chairs… Why not? The fundamental principle of data science is that when dealing with uncertainty, you cannot use the same data point for both hypothesis generation and testing. When data is limited, uncertainty forces you to choose between statistics or analytics. Explanation here.

Without statistics, you may remain stuck and unable to understand whether the newly formulated statement withstands scrutiny, and without analysis, you move blindly with little chance of taming the unknown. It’s a hard choice.

The charlatan's way out of this predicament is to ignore it and then pretend to be surprised when something unexpected is discovered. The logic of hypothesis testing boils down to the question: do the data surprise us enough to change our opinion? How can we be surprised by data if we have already seen them?

Whenever charlatans find a pattern, they get inspired, then test the same data for of the same pattern, to publish results with a legitimate p-value or two alongside their theory. In doing so, they are lying to you (and possibly to themselves as well). Such a p-value doesn't matter if you don't stick to your hypothesis. up to how they viewed their data. Charlatans mimic the actions of analysts and statisticians without understanding the reasons behind them. As a result, the entire field of data science suffers from a bad reputation.

True statisticians always draw their conclusions carefully.

Due to the almost mystical reputation of statisticians engaged in rigorous reasoning, the amount of fake information in Data Science is at an all-time high. It's easy to deceive and not get caught, especially if an unsuspecting victim thinks it’s all about the equations and data. A dataset is just a dataset, right? No. How you use it matters.

Fortunately, you need just one clue to catch charlatans: they 'rediscover America in hindsight.' They rehash phenomena that they already know are present in the data.

In contrast to charlatans, good analysts are unbiased and understand that inspiring ideas may have many different explanations. At the same time, good statisticians carefully define their conclusions before they make them.

Analysts are free from accountability... as long as they stay within the limits of their data. If they are tempted to claim things they haven't seen, that’s a different job altogether. They should 'take off their analyst shoes’ and 'put on their statistician shoes.' After all, regardless of the official job title, there’s no rule preventing you from studying both professions if you want. Just don’t confuse them.

Being good at statistics does not mean you're good at analytics, and vice versa. If someone tries to tell you otherwise, that's a red flag. If this person informs you that making statistical conclusions from data you've already analyzed is allowed, that's a double caution sign.

Quirky explanations

When observing data charlatans in the wild, you'll notice they love to concoct fantastic stories to "explain" the observed data. The more academic, the better. It doesn't matter that these stories are backfilled.

When charlatans do this — let me not hold back — they are lying. No amount of equations or beautiful concepts compensates for the fact that they provide zero proof for their versions. Don't be surprised at how unusual their explanations can be.

It's the same as demonstrating your "extrasensory" abilities by first looking at the cards in your hands and then predicting what you're holding… exactly what you're holding. This is retrospective bias, and the profession of data scientist is packed with it.

Analysts say: "You just played the queen of diamonds." Statisticians say: "I wrote down my hypotheses on this piece of paper before we started. Let's play, look at some data, and see if I'm right." Charlatans say: "I knew you were going to play that queen of diamonds because..."

Data splitting is a quick fix to the problem that every organization faces.

When there's not enough data, you have to choose between statistics and analytics, but when there's plenty of data, there's a wonderful opportunity to leverage analytics without deception. and statistics. You have the perfect defense against charlatans — it's data splitting, and in my opinion, it's the most powerful idea in Data Science.

To protect yourself from charlatans, all you need to do is ensure you keep some test data out of their curious eyes, and then treat everything else as analytics. When you encounter a theory you're at risk of accepting, use it to assess the situation, and then reveal your secret test data to verify that the theory isn't nonsense. It’s that simple!

Make sure no one is allowed to view test data during the research phase. To achieve this, stick to research data. Test data should not be used for analysis.

This is a significant step forward compared to what people were accustomed to in the era of 'small data', where you had to explain how you knew what you knew to finally convince people that you actually knew something.

We apply the same rules to ML/AI.

Some charlatans posing as experts in ML/AI are also easy to spot. You can catch them just like any other bad engineer: the 'solutions' they try to build consistently fail. An early warning sign is a lack of experience with standard industry languages and programming libraries.

But what about people creating systems that seem to work? How do you know something suspicious is happening? The same rule applies! A charlatan is a shady character who shows you how well the model performed… on the same data they used to create the model.

If you’ve created a wildly complex machine learning system, how do you know how good it is? You won't know until you demonstrate that it works with new data it hasn’t seen before.

When you have seen the data before forecasting — it's unlikely to be aprediction.

When you have enough data to split, you don’t need to rely on the elegance of your formulas to justify the project (an old-fashioned habit I see everywhere, not just in science). You can say: "I know it works because I can take a dataset that it hasn't seen before and accurately predict what will happen there… and I will be right. Again and again."

Testing your model/theory on new data is the best basis for trust.

I do not tolerate data charlatans. I don’t care if your opinion relies on different gimmicks. I’m not impressed by the elegance of explanations. Show me that your theory/model works (and continues to work) on a whole range of new data that you've never seen before. That is the real test of the robustness of your opinion.

Consulting with Data Science specialists.

If you want to be taken seriously by everyone who understands this humor, stop hiding behind quirky equations to maintain personal biases. Show what you have. If you want those who 'got it' to see your theory/model as something more than just inspiring poetry, have the courage to put on a grand display of how well it works on a completely new data set… in front of witnesses!

Addressing the executives

Refuse to take any 'ideas' about data seriously until they have been tested on new data. Don’t feel like putting in the effort? Stick to analytics, but don’t rely on these ideas—they are unreliable and have not been tested for reliability. Moreover, when an organization has data in abundance, there is no harm in making separation a foundation in science and maintaining it at the infrastructure level by controlling access to test data for statistics. This is a great way to thwart attempts to deceive you!

If you want to see more examples of charlatans plotting something nefarious— here's a wonderful thread on Twitter.

Summary

When data is too scarce for separation, only a charlatan tries to strictly follow inspiration, retroactively discovering America, mathematically 'rediscovering' phenomena already known to exist in the data, and calling astonishment statistically significant. This distinguishes them from an unbiased analyst dealing with inspiration and a meticulous statistician providing evidence in forecasting.

When there’s plenty of data, make it a habit to separate the data, so you can have the best of both worlds! Be sure to do analytics and statistics separately on distinct subsets of the original data pile.

Analysts offer you inspiration and breadth of perspective.
Statisticians offer you rigorous testing.
Charlatans offer you a twisted retrospective view that pretends to be analytics plus statistics.

It's possible that after reading the article, you might think to yourself, "Am I a fraud?" That's normal. You can dispel this thought in two ways: first, look back and see what you have accomplished; did your work with data provide practical benefits? Secondly, you can further enhance your qualifications (which certainly won't hurt), especially since we equip our students with practical skills and knowledge that enable them to become true data scientists.