{"id":97051,"date":"2020-10-15T14:43:34","date_gmt":"2020-10-15T12:43:34","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-raspoznat-sharlatana-ot-data-science"},"modified":"2020-10-15T14:43:34","modified_gmt":"2020-10-15T12:43:34","slug":"kak-raspoznat-sharlatana-ot-data-science","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-raspoznat-sharlatana-ot-data-science","title":{"rendered":"How to recognize a charlatan in Data Science?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"How<\/a><\/noindex>
\nYou may have heard about analysts, machine learning specialists, and artificial intelligence experts, but have you heard about those who are unjustly overpaid? Meet data charlatans<\/strong>! These tricksters, lured by lucrative jobs, create a bad reputation for real data professionals. In this article, we will explore how to expose such individuals.
\n<\/a><\/noindex><\/p>\n

Data charlatans are everywhere<\/h2>\n

\nData charlatans are so adept at hiding in plain sight that you might be one of them<\/i>, without even realizing it. Chances are, your organization has been sheltering these tricksters for years, but here\u2019s the good news: they are easy to identify if you know what to look for.
\nThe first warning sign is a misunderstanding that analytics and statistics are very different disciplines<\/strong>. I will explain this further.<\/p>\n

Different disciplines<\/h2>\n

\nStatisticians are trained to make inferences about what lies beyond their data, while analysts are trained to study the contents of a data set. In other words, analysts draw conclusions about what is in their data, while statisticians make conclusions about what is not in the data. Analysts help pose good questions (formulate hypotheses), while statisticians help obtain good answers (test hypotheses).<\/p>\n

There are also fanciful hybrid roles where a person tries to straddle both chairs\u2026 Why not? The fundamental principle of data science is that when dealing with uncertainty, you cannot use the same<\/i> data point for both hypothesis generation and testing. When data is limited, uncertainty forces you to choose between statistics or analytics. Explanation<\/i> here<\/i><\/a><\/noindex>.<\/p>\n

Without statistics, you may remain stuck and unable to understand whether the newly formulated statement withstands scrutiny, and without analysis, you move blindly with little chance of taming the unknown. It\u2019s a hard choice.<\/p>\n

The charlatan's way out of this predicament is to ignore it and then pretend to be surprised when something unexpected is discovered. The logic of hypothesis testing boils down to the question: do the data surprise us enough to change our opinion? How can we be surprised by data if we have already seen them?<\/p>\n

Whenever charlatans find a pattern, they get inspired, then test the same data<\/i> for of the same pattern<\/i>, to publish results with a legitimate p-value or two alongside their theory. In doing so, they are lying to you (and possibly to themselves as well). Such a p-value doesn't matter if you don't stick to your hypothesis. up to<\/b> how they viewed their data. Charlatans mimic the actions of analysts and statisticians without understanding the reasons behind them. As a result, the entire field of data science suffers from a bad reputation.<\/p>\n

True statisticians always draw their conclusions carefully.<\/h2>\n

\nDue to the almost mystical reputation of statisticians engaged in rigorous reasoning, the amount of fake information in Data Science is at an all-time high. It's easy to deceive and not get caught, especially if an unsuspecting victim thinks it\u2019s all about the equations and data. A dataset is just a dataset, right? No. How you use it matters.<\/p>\n

Fortunately, you need just one clue to catch charlatans: they 'rediscover America in hindsight.' They rehash phenomena that they already know are present in the data. <\/p>\n

In contrast to charlatans, good analysts are unbiased and understand that inspiring ideas may have many different explanations. At the same time, good statisticians carefully define their conclusions before they make them.<\/p>\n

Analysts are free from accountability... as long as they stay within the limits of their data. If they are tempted to claim things they haven't seen, that\u2019s a different job altogether. They should 'take off their analyst shoes\u2019 and 'put on their statistician shoes.' After all, regardless of the official job title, there\u2019s no rule preventing you from studying both professions if you want. Just don\u2019t confuse them.<\/p>\n

Being good at statistics does not mean you're good at analytics, and vice versa. If someone tries to tell you otherwise, that's a red flag. If this person informs you that making statistical conclusions from data you've already analyzed is allowed, that's a double caution sign.<\/p>\n

Quirky explanations<\/h2>\n

\nWhen observing data charlatans in the wild, you'll notice they love to concoct fantastic stories to \"explain\" the observed data. The more academic, the better. It doesn't matter that these stories are backfilled.<\/p>\n

When charlatans do this \u2014 let me not hold back \u2014 they are lying. No amount of equations or beautiful concepts compensates for the fact that they provide zero proof for their versions. Don't be surprised at how unusual their explanations can be.<\/p>\n

It's the same as demonstrating your \"extrasensory\" abilities by first looking at the cards in your hands and then predicting what you're holding\u2026 exactly what you're holding. This is retrospective bias, and the profession of data scientist is packed with it.<\/p>\n

\"How<\/p>\n

Analysts say: \"You just played the queen of diamonds.\" Statisticians say: \"I wrote down my hypotheses on this piece of paper before we started. Let's play, look at some data, and see if I'm right.\" Charlatans say: \"I knew you were going to play that queen of diamonds because...\" <\/p>\n

Data splitting is a quick fix to the problem that every organization faces.<\/p><\/blockquote>\n

\nWhen there's not enough data, you have to choose between statistics and analytics, but when there's plenty of data, there's a wonderful opportunity to leverage analytics without deception. and<\/i> statistics. You have the perfect defense against charlatans \u2014 it's data splitting, and in my opinion, it's the most powerful idea in Data Science. <\/p>\n

To protect yourself from charlatans, all you need to do is ensure you keep some test data out of their curious eyes, and then treat everything else as analytics. When you encounter a theory you're at risk of accepting, use it to assess the situation, and then reveal your secret test data to verify that the theory isn't nonsense. It\u2019s that simple!<\/p>\n

\"How
\nMake sure no one is allowed to view test data during the research phase. To achieve this, stick to research data. Test data should not be used for analysis.<\/p>\n

This is a significant step forward compared to what people were accustomed to in the era of 'small data', where you had to explain how you knew what you knew to finally convince people that you actually knew something.<\/p>\n

We apply the same rules to ML\/AI.<\/h2>\n

\nSome charlatans posing as experts in ML\/AI are also easy to spot. You can catch them just like any other bad engineer: the 'solutions' they try to build consistently fail. An early warning sign is a lack of experience with standard industry languages and programming libraries.<\/p>\n

But what about people creating systems that seem to work? How do you know something suspicious is happening? The same rule applies! A charlatan is a shady character who shows you how well the model performed\u2026 on the same data they used to create the model.<\/p>\n

If you\u2019ve created a wildly complex machine learning system, how do you know how good it is? You won't know until you demonstrate that it works with new data it hasn\u2019t seen before.<\/p>\n

When you have seen the data before forecasting \u2014 it's unlikely to be a<\/i>prediction.<\/p><\/blockquote>\n

\nWhen you have enough data to split, you don\u2019t need to rely on the elegance of your formulas to justify the project (an old-fashioned habit I see everywhere, not just in science). You can say: \"I know it works because I can take a dataset that it hasn't seen before and accurately predict what will happen there\u2026 and I will be right. Again and again.\"<\/i><\/p>\n

Testing your model\/theory on new data is the best basis for trust.<\/p><\/blockquote>\n

\nI do not tolerate data charlatans. I don\u2019t care if your opinion relies on different gimmicks. I\u2019m not impressed by the elegance of explanations. Show me that your theory\/model works (and continues to work) on a whole range of new data that you've never seen before. That is the real test of the robustness of your opinion.<\/p>\n

Consulting with Data Science specialists.<\/h2>\n

\nIf you want to be taken seriously by everyone who understands this humor, stop hiding behind quirky equations to maintain personal biases. Show what you have. If you want those who 'got it' to see your theory\/model as something more than just inspiring poetry, have the courage to put on a grand display of how well it works on a completely new data set\u2026 in front of witnesses!<\/p>\n

Addressing the executives<\/h2>\n

\nRefuse to take any 'ideas' about data seriously until they have been tested on new<\/i> data. Don\u2019t feel like putting in the effort? Stick to analytics, but don\u2019t rely on these ideas\u2014they are unreliable and have not been tested for reliability. Moreover, when an organization has data in abundance, there is no harm in making separation a foundation in science and maintaining it at the infrastructure level by controlling access to test data for statistics. This is a great way to thwart attempts to deceive you!<\/p>\n

If you want to see more examples of charlatans plotting something nefarious\u2014 here's a wonderful thread on Twitter<\/a><\/noindex>.<\/p>\n

Summary<\/h2>\n

\nWhen data is too scarce for separation, only a charlatan tries to strictly follow inspiration, retroactively discovering America, mathematically 'rediscovering' phenomena already known to exist in the data, and calling astonishment statistically significant. This distinguishes them from an unbiased analyst dealing with inspiration and a meticulous statistician providing evidence in forecasting.<\/p>\n

When there\u2019s plenty of data, make it a habit to separate the data, so you can have the best of both worlds! Be sure to do analytics and statistics separately on distinct subsets of the original data pile.<\/p>\n