tl;dr:
- Machine learning seeks patterns in data. However, artificial intelligence can be 'biased' — that is, it can find incorrect patterns. For example, a skin cancer detection system based on photographs may pay particular attention to images taken in a medical office. Machine learning does not understand: its algorithms only identify patterns in numbers, and if the data is not representative, the result will be similarly flawed. Catching such bugs can be challenging due to the very mechanics of machine learning.
- The most obvious and concerning problematic area is human diversity. There are many reasons why data about people may lose objectivity even at the collection stage. But it’s essential not to think that this issue only pertains to people: similar difficulties arise when trying to detect flooding in a warehouse or a malfunctioning gas turbine. Some systems may have biases regarding skin color, while others may be biased against Siemens sensors.
- Such issues are not new to machine learning and are not exclusive to it. Incorrect assumptions are made in any complex structures, and understanding why a certain decision was made is always challenging. This must be addressed comprehensively: by creating tools and processes for verification — and educating users so they do not blindly follow AI recommendations. Machine learning does indeed perform some tasks much better than we do — but dogs, for instance, are far more effective than humans at detecting drugs, which is certainly not a reason to involve them as witnesses and render judgments based on their testimony. Interestingly, dogs are much smarter than any machine learning system.
Today, machine learning is one of the most critical foundational technological trends. It is one of the primary ways technology will change the world around us in the next decade. Some aspects of these changes raise concerns. For example, the potential impact of machine learning on the labor market, or its use for unethical purposes (say, by authoritarian regimes). There is another problem that this post is dedicated to: bias in artificial intelligence.
It's a complicated story.
Google's AI can find cats. This news from 2012 was something special back then.
What is 'AI bias'?
'Raw data' is both an oxymoron and a bad idea; data needs to be well and carefully prepared. -Jeffrey Boker
Until around 2013, to create a system that, say, recognizes cats in photographs, you had to describe the logical steps. How to find corners in an image, recognize eyes, analyze textures for fur, count paws, and so on. Then assemble all the components—only to discover that it doesn’t really work. It’s somewhat like a mechanical horse—while theoretically possible, in practice, it’s too complex to describe. In the end, you end up with hundreds (or even thousands) of handwritten rules. And not a single working model.
With the advent of machine learning, we stopped using 'manual' rules for object recognition. Instead, we take a thousand samples of 'this', X, a thousand samples of 'that', Y, and make the computer build a model based on their statistical analysis. Then we provide this model with some example data, and it determines with some accuracy whether it fits one of the sets. Machine learning generates a model based on data, not by a person writing it. The results are impressive, especially in the field of image and pattern recognition, which is why the entire tech industry is now moving towards machine learning (ML).
But it’s not as simple as it seems. In the real world, your thousands of examples X or Y also contain A, B, J, L, O, R, and even L. They may be unevenly distributed, and some of them might occur so frequently that the system pays more attention to them than to the objects you are interested in.
What does this mean in practice? My favorite example is when image recognition systems . It’s clear why: most of the example photos of 'sheep' were taken in pastures where they live, and in those images, the grass takes up much more space than the little white fluffy ones, which is why the systems consider grass to be the most important.
There are more serious examples. Recently, one on the detection of skin cancer in photographs. It turns out that dermatologists often photograph a ruler alongside manifestations of skin cancer to record the size of the formations. In examples of healthy skin, there are no rulers. For the AI system, these rulers (or rather, the pixels identified as "ruler") became one of the distinguishing features between example sets, sometimes even more important than a small rash on the skin. Thus, the system designed to recognize skin cancer sometimes identified the rulers instead.
The key point here is that the system has no semantic understanding of what it is observing. We look at a set of pixels and see a sheep, skin, or rulers, while the system sees only a string of numbers. It does not perceive three-dimensional space, nor does it see objects, textures, or sheep. It simply identifies patterns in data.
The complexity of diagnosing such issues lies in the fact that a neural network (the model generated by your machine learning system) consists of thousands or even hundreds of thousands of nodes. There is no simple way to peer into the model and see how it makes decisions. Having such a method would imply that the process is straightforward enough to describe all rules manually, without the use of machine learning. People are concerned that machine learning has become a kind of "black box". (I will explain later why this comparison is somewhat exaggerated.)
This, in broad terms, is the issue of bias in artificial intelligence or machine learning: a system designed to find patterns in data may identify incorrect patterns, and you may not notice it. This is a fundamental characteristic of the technology, and it is clear to anyone working with it in academic circles and large tech companies. However, the implications are complex, and our potential solutions to these implications are too.
Let's first talk about the consequences.
AI may implicitly make choices in favor of certain categories of people, based on a multitude of subtle signals.
AI Bias Scenarios
The most obvious and alarming manifestation of this issue can occur when it comes to human diversity. Recently, , that Amazon attempted to build a machine learning system for initial candidate screening. Since there are more men among Amazon employees, the examples of 'successful hires' were also predominantly male, and the résumé selection provided by the system included more men. Amazon noticed this and decided not to launch the system into production.
The most important aspect of this example is that the system reportedly favored male candidates, despite gender not being specified in the résumés. The system identified different patterns in the examples of 'successful hires': for instance, women might use specific words to describe achievements, or have unique hobbies. Of course, the system was unaware of what 'hockey' was, who 'people' are, or what 'success' means — it simply conducted a statistical analysis of the text. However, the patterns it identified would likely remain unnoticed by a human, and some of them (for example, how people of different genders describe success) would probably be difficult for us to see, even when presented with them.
Then things get worse. A machine learning system that works very well at detecting cancer in light skin may perform poorly on darker skin, or vice versa. This isn't necessarily due to bias, but rather because you likely need to build a separate model for different skin tones, selecting other characteristics. Machine learning systems are not interchangeable, even in such a narrow field as image recognition. You need to configure the system, sometimes just through trial and error, to accurately capture features in the data of interest until you reach the desired level of accuracy. But you might not notice that the system is 98% accurate with one group, while only 91% (which is still more accurate than human analysis) with another.
I have primarily used examples relating to people and their characteristics so far. This is indeed the main focus of the discussion surrounding this issue. However, it is important to understand that bias towards people is just part of the problem. We will use machine learning for many things, and sampling error will be relevant in all of them. On the other hand, if you are working with people, data bias may not be related to them.
To understand this, let's return to the example of skin cancer and consider three hypothetical ways the system might fail.
- Uneven distribution of people: an imbalanced number of skin tone photographs leads to false positive or false negative results associated with pigmentation.
- The data on which the system is trained contains frequently occurring and unevenly distributed characteristics that are unrelated to people and have no diagnostic value: a ruler in photographs of skin cancer manifestations or grass in photos of sheep. In this case, the outcome will differ if the system finds pixels of something that the human eye identifies as a 'ruler' in the image.
- The data contains an external characteristic that a person cannot see, even if they are looking for it.
What does this mean? We inherently know that data can represent different groups of people differently, and at the very least, we can plan to search for such exceptions. In other words, there are many social reasons to assume that data about groups of people already contains some bias. If we look at a photo with a ruler, we will see that ruler—we just ignored it before, knowing that it wasn't relevant, and forgetting that the system knows nothing.
But what if all your unhealthy skin photographs were taken in an office with incandescent lamps, while the healthy ones were under fluorescent light? What if, after finishing the healthy skin shoot, you updated your phone's operating system before shooting the unhealthy skin, and Apple or Google slightly altered the noise suppression algorithm? A person wouldn't notice this, no matter how much they looked for such features. However, a machine-based system would immediately see and utilize this. It knows nothing.
While we talked about false correlations, it may also happen that the data is accurate and the results are correct, but you do not want to use them for ethical, legal, or managerial reasons. In some jurisdictions, for instance, it is not permissible to offer women a discount on insurance, even though women may drive more safely. We can easily imagine a system that assigns a lower risk coefficient to female names when analyzing historical data. Okay, let's remove names from the sample. But remember the example with Amazon: the system can determine gender by other factors (even if it does not know what gender is, or what a car is), and you won't notice this until a regulator retrospectively analyzes the rates you propose and fines you.
Finally, it is often implied that we will use such systems only for projects related to people and social interactions. This is not true. If you are making gas turbines, you will surely want to apply machine learning to the telemetry transmitted by dozens or hundreds of sensors on your product (audio, video, temperature, and any other sensors generate data that can be very easily adapted to create a machine learning model). Hypothetically, you might say: "Here’s data on a thousand failed turbines obtained before their breakdowns, and here’s data from a thousand turbines that did not fail. Build a model to identify the differences between them." Now imagine that Siemens sensors are present on 75% of the bad turbines and only 12% of the good ones (with no connection to failures). The system will build a model to find turbines with Siemens sensors. Oops!
Image — Moritz Hardt, UC Berkeley
Managing AI Bias
What can we do about this? We can approach the issue from three angles:
- Methodological rigor in data collection and management for training the system.
- Technical tools for analyzing and diagnosing model behavior.
- Training, education, and caution when implementing machine learning into products.
In Molière's book "The Bourgeois Gentleman," there is a joke about a man who learns that literature is divided into prose and poetry, and he is delighted to discover that he has been speaking in prose all his life without knowing it. Perhaps this is how statisticians feel today: unknowingly, they have dedicated their careers to artificial intelligence and sampling error. Searching for sampling error and worrying about it is not a new issue; we just need to approach its solution systematically. As mentioned earlier, in some cases, it is indeed easier to study issues related to data about people. We assume a priori that we may have biases regarding different groups of people, but the bias against Siemens sensors is hard for us to even imagine.
The new element in all this, of course, is that people are no longer engaged in statistical analysis directly. It is conducted by machines that create large, complex models that are difficult to understand. The issue of transparency is one of the main aspects of the bias problem. We are scared that the system is not only biased but that there is no way to detect its bias, and that machine learning differs from other forms of automation, which are supposed to consist of clear logical steps that can be verified.
There are two problems here. We may still be able to conduct some kind of audit of machine learning systems. And auditing any other system is not any easier.
Firstly, one of the directions of modern research in the field of machine learning is the search for methods to identify the important functionality of machine learning systems. At the same time, machine learning (in its current state) is a completely new field of science that is rapidly changing, so one should not think that things that are impossible today may not soon become quite real. The project is an interesting example of this.
Secondly, the idea that one can assess and understand the decision-making processes in existing systems or organizations is good in theory, but not so much in practice. Understanding how decisions are made in a large organization is quite difficult. Even if there is a formal decision-making process, it does not reflect how people actually interact, and they often lack a logical systemic approach to their decisions. As my colleague said, , people are also black boxes.
Take a thousand people from several intersecting companies and institutions, and the problem becomes even more complex. We know in hindsight that the "Space Shuttle" was destined to disintegrate upon re-entry, and individual people within NASA had information that led them to believe something bad might happen, but the system as a whole did not know this. NASA had just undergone a similar audit after losing the previous shuttle, and nonetheless, it lost another one — for very similar reasons. It’s easy to argue that organizations and people follow clear logical rules that can be verified, understood, and changed — but experience proves otherwise. This is the».
I often compare machine learning to databases, especially relational ones — a new foundational technology that has changed the possibilities of computing and the world around it, becoming part of everything we constantly use without even realizing it. Databases also have issues, and they are of a similar nature: the system may be built on false assumptions or poor data, but this will be hard to notice, and the people using the system will do what it tells them without questioning it. There are plenty of old jokes about tax workers who once incorrectly recorded your name, and convincing them to correct the mistake is much harder than actually changing the name. This can be thought of in different ways, but it is unclear how best to consider it: as a technical issue in SQL, or as a bug in an Oracle release, or as a breakdown in bureaucratic institutions? How hard is it to find the error in a process that led to the system lacking a feature like spell-checking? Could this have been understood before people started complaining?
This problem is even easier to illustrate with stories about drivers who, due to outdated data in their GPS, drive into rivers. Okay, the maps need to be constantly updated. But how much is TomTom to blame for your car being swept into the sea?
What I mean is that, yes — machine learning bias will create problems. But these problems will be similar to those we've encountered in the past, and they can be noticed and solved (or not) about as well as we've managed in the past. Therefore, the scenario where AI bias causes harm is unlikely to occur with leading researchers working in a large organization. More likely, some insignificant tech contractor or software vendor will create something haphazardly using open-source components, libraries, and tools that they don't fully understand. And an unsuspecting client will be lured in by the phrase "artificial intelligence" in the product description and, without asking further questions, will distribute it to their underpaid workers, forcing them to do what the AI says. This is exactly what happened with databases. This is not an artificial intelligence problem, and not even a software problem. It's a human factor.
Conclusion
Machine learning can do everything you can teach a dog — but you can never be sure what exactly you taught that dog.
I often feel that the term 'artificial intelligence' only complicates discussions like this one. This term creates a false impression that we have actually created it — this intelligence. That we are on the path to HAL9000 or Skynet — to something that truly understands. understands. But no. These are just machines, and it's much more accurate to compare them, say, to a washing machine. It handles laundry much better than a human, but if you put dishes in it instead of clothes, it will... wash them. The dishes will even become clean. But this is not what you expected, and it will happen not because the system has any bias against dishes. The washing machine knows neither what dishes are nor what clothes are — it’s merely an example of automation, conceptually no different from how processes were automated before.
Whatever the subject at hand — machines, airplanes, or databases — these systems will be both very powerful and very limited. They will completely depend on how people use these systems, whether their intentions are good or bad, and how well they understand their operation.
Therefore, to say that 'artificial intelligence is math, so it cannot have biases' is completely incorrect. But it is equally incorrect to claim that machine learning is 'subjective by nature.' Machine learning finds patterns in data, and what patterns it will find depends on the data, and the data depends on us. Just like what we do with it. Machine learning does indeed perform some tasks much better than we do — but dogs, for example, are much more effective than humans at detecting drugs, which is not a reason to bring them in as witnesses and pass judgment based on their testimony. And dogs, by the way, are much smarter than any machine learning system.
Translation: .
Editing: .
Community: .
Source: habr.com
