Has the machine learning bubble burst, or is it the dawn of a new era?

Recently released article, which shows the trends in machine learning of the last few years quite well. In short: the number of startups in the field of machine learning has sharply decreased in the last two years.

Well then. Let's discuss whether the bubble has burst, how to move forward, and talk about where this all came from.

To begin with, let's discuss what was the booster for this curve. Where did it come from? Probably everyone remembers the victory of machine learning in 2012 at the ImageNet competition. After all, this was the first global event! But in reality, it's not that simple. The growth of the curve actually began slightly earlier. I would break it down into several key moments.

1. The year 2008 marks the emergence of the term 'big data.' Real products started to appear from 2010 onwards. Big data is directly related to machine learning. Without big data, stable operation of algorithms that existed at that time is impossible. And these were not neural networks. Until 2012, neural networks were the domain of a marginal minority. However, completely different algorithms that had existed for years, if not decades, began to be used: SVM(1963, 1993), Random Forest (1995), AdaBoost (2003),... Startups of those years were primarily focused on the automatic processing of structured data: cash registers, users, advertising, and much more.

   The derivative of this first wave is a set of frameworks, such as XGBoost, CatBoost, LightGBM, etc.

2. In 2011-2012, convolutional neural networks won several image recognition competitions. Their actual use took a while longer. I would say that mass-scale meaningful startups and solutions began to appear around 2014. It took two years to digest the fact that neural networks indeed work, to create convenient frameworks that could be installed and run in a reasonable time, and to develop methods that would stabilize and speed up convergence time.

   Convolutional networks allowed solving machine vision tasks: image classification and object detection, recognizing objects and people, improving images, etc.

3. 2015-2017. A boom of algorithms and projects tied to recurrent networks or their analogs (LSTM, GRU, TransformerNet, etc.). Effective speech-to-text algorithms and machine translation systems emerged. Some are based on convolutional networks for feature extraction, while others leverage the ability to collect large, high-quality datasets.

"Has the bubble burst? Is the hype overheated? Did they die like blockchain?"
That's right! Tomorrow Siri will stop working on your phone, and the day after, Tesla won't distinguish a turn from a kangaroo.

Neural networks are already functioning. They are present in dozens of devices. They genuinely enable monetization, changing markets and the surrounding world. The hype looks somewhat different:

Simply put, neural networks are no longer something new. Yes, many people have inflated expectations. But a significant number of companies have learned to apply neural networks and develop products based on them. Neural networks offer new functionalities, allow for job reductions, and decrease service costs:

• Manufacturing companies integrate algorithms for defect analysis on assembly lines.
• Livestock farms purchase systems for cow monitoring.
• Automatic combines.
• Automated call centers.
• Filters in SnapChat. (At least something useful!)

But the main and less obvious point is: "There are no new ideas, or they won't bring instant capital." Neural networks have solved dozens of problems. And they'll solve even more. All the obvious ideas that existed have spawned numerous startups. But everything that was apparent has already been taken. Over the past two years, I've encountered no new ideas for applying neural networks. Not a single new approach (well, okay, there are some intricacies with GANs).

Each subsequent startup is increasingly complex. It requires not just two guys training a neural network on public data. It demands programmers, servers, a team of annotators, complex support, etc.

As a result, there are fewer startups. However, production is increasing. Need to implement license plate recognition? There are hundreds of specialists in the market with relevant experience. You can hire one, and in a couple of months, your employee will develop the system. Or you can buy an off-the-shelf solution. But creating a new startup? That's madness!

We need to create a visitor tracking system—why pay for a bunch of licenses when you can build your own tailored for your business in 3-4 months?

Right now, neural networks are experiencing the same path as many other technologies have before.

Remember how the term 'web developer' has evolved since 1995? Currently, the market is not saturated with specialists. There are very few professionals. But I can argue that in 5-10 years, there won't be much difference between a Java programmer and a neural network developer. There will be enough specialists in both fields.

There will simply be a class of tasks that will be solved using neural networks. When a task arises, you hire a specialist.

"So what’s next? Where's the promised artificial intelligence?"

Here we have a small, but interesting conundrum :)

The technology stack we have today, it seems, will not lead us to artificial intelligence after all. Ideas and their novelty have largely run their course. Let's discuss what is holding back current development.

Restrictions

Let’s start with autonomous vehicles. It seems obvious that making fully autonomous cars with today's technology is possible. But how many years will it take? That's unclear. Tesla believes this will happen in a couple of years—

There are many other specialists, who estimate it will take 5-10 years.

In my opinion, most likely in about 15 years, city infrastructure will change in such a way that the emergence of autonomous vehicles will become inevitable, becoming an extension of it. But this cannot be considered intelligence. A modern Tesla is a very complex conveyor for filtering data, searching it, and retraining. It’s rules-rules-rules, data collection, and filters applied to them (here's here I wrote a bit more about this, or you can see it from this the marker).

The first problem

And it is here that we see the first fundamental problemBig data. This is precisely what has spawned the current wave of neural networks and machine learning. Nowadays, to accomplish something complex and automated, a lot of data is required. Not just a lot, but an immense amount. Automated algorithms for gathering, annotating, and using this data are needed. If we want the machine to recognize trucks against the sun, we first need to gather a sufficient number of them. If we want the machine not to freak out about a bicycle attached to a trunk, we need more samples.

And one example won't be enough. Hundreds? Thousands?

The second problem

The second problem — is visualizing what our neural network has understood. This is a very non-trivial task. Until now, few people understand how to visualize this. These articles are quite recent; they are just a few examples, albeit distant:
Visualization is fixated on textures. It clearly shows what the neural network tends to get fixated on + what it perceives as initial information.

Visualization attention when translations. In reality, attention can often be used precisely to show what triggered such a network reaction. I've encountered such things for both debugging and product solutions. There are many articles on this topic. But the more complex the data, the harder it becomes to achieve stable visualization.

And yes, the good old set of "look at what's inside the network in filters". These images were popular about 3-4 years ago, but everyone quickly understood that while the pictures are beautiful, they don't carry much meaning.

I haven't mentioned dozens of other gadgets, methods, hacks, and studies on how to display the inner workings of the network. Do these tools work? Do they help quickly understand what the problem is and debug the network? Extract the last percentages? Well, it's pretty much like this:

You can look at any competition on Kaggle. And the descriptions of how people make their final solutions. We stacked 100-500-800 million models and it worked!

Of course, I'm exaggerating. But these approaches do not provide quick and direct answers.

With enough experience, by poking around various options, one can issue a verdict on why your system made such a decision. However, correcting the system's behavior will be difficult. Setting a workaround, shifting the threshold, adding a dataset, or taking another backend network.

The third problem

The third fundamental problem — neural networks teach not logic, but statistics. Statistically, this is a face.:

Logically, it doesn't quite seem like it. Neural networks don’t learn anything complex unless forced to. They always learn the simplest features possible. Are there eyes, a nose, a head? Then it's a face! Unless you provide an example where eyes don’t indicate a face. Again, millions of examples.

There’s Plenty of Room at the Bottom

I would say that these three global problems currently limit the development of neural networks and machine learning. Wherever these problems haven’t limited progress, it's already in active use.

Is this the end? Have neural networks stagnated?

It's unknown. But, of course, everyone hopes not.

There are many approaches and directions aimed at solving the fundamental problems I've outlined above. But so far, none of these approaches have led to anything fundamentally new, nothing that hasn't been solved before. Currently, all fundamental projects are based on stable approaches (like Tesla), or remain test projects by institutes or corporations (like Google Brain, OpenAI).

In rough terms, the main direction is to create some high-level representation of input data. In a sense, a type of 'memory'. The simplest example of memory is various 'embeddings' — representations of images. For instance, all face recognition systems. The network learns to obtain a stable representation of a face that doesn't depend on angle, lighting, or resolution. Essentially, the network minimizes the metric of 'different faces — far' and 'same — close.'

For such learning, tens and hundreds of thousands of examples are needed. However, the result bears some aspects of 'one-shot learning.' Now we don’t need hundreds of faces to remember a person. Just one face, and that’s it — we recognize.!
However, there's a small problem... The network can only learn sufficiently simple objects. When trying to differentiate not faces, but, for example, 'people by their clothing' (the task of re-identification) — the quality drops significantly. And the network can no longer learn sufficiently obvious changes in perspective.

Moreover, learning from millions of examples is also somewhat of a tedious endeavor.

There are works on significantly reducing the selection. For instance, one of the first works on one-shot learning by Google.:

There are many such works, for instance 1 or 2 or 3.

One downside is that training typically works well on simple, ‘MNIST-like examples’. However, when moving to complex tasks, a larger dataset, model objects, or some sort of magic is required.
In general, the work on One-Shot learning is a very interesting topic. You find a lot of ideas. However, the two main issues I mentioned (pre-training on huge datasets / instability on complex data) really hinder the learning process.

On the other hand, GANs—generative adversarial networks—are relevant to the topic of Embedding. You have probably read a lot of articles about this on Habr.1, 2,3)
The feature of GANs is the formation of some internal state space (essentially the same as Embedding), which allows for image creation. This could involve faces, or could involve actions..

The problem with GANs is that the more complex the generated object, the harder it is to describe it in the logic of ‘generator-discriminator’. As a result, the only widely recognized real-world applications of GANs are DeepFake, which manipulates representations of faces (for which there is a large dataset).

I have encountered very few other useful applications. Usually, they are just gimmicks that involve editing pictures.

And again. No one understands how this will allow us to move towards a bright future. The representation of logic / space in a neural network is good. But a vast number of examples are needed; we don’t understand how the neural network represents this internally, and we don’t understand how to make the neural network remember some genuinely complex representation.

Reinforcement learning is an approach from a completely different angle. You surely remember how Google beat everyone in Go. Recent victories in Starcraft and Dota also stand out. However, things are far from rosy and promising in this area. The complexities of RL are best explained in this article..

To briefly summarize what the author wrote:

• Off-the-shelf models are not suitable / work poorly in most cases.
• Practical problems can be solved more easily in other ways. Boston Dynamics does not use RL due to its complexity / unpredictability / complexity of computations.
• For RL to work, a complicated function is needed. Often it’s difficult to create / write it.
• It is difficult to train models. You have to spend a ton of time getting them to converge and escape local optima.
• As a result, it's difficult to repeat the model, and the model is unstable with the slightest changes.
• It often overfits to some random patterns, even down to the random number generator.

The key point is that RL does not currently work in production. Google has some experiments ( 1, 2 ). But I haven't seen a single production system.

Memory. The downside of everything described above is unstructuredness. One approach to try to manage all this is to give the neural network access to separate memory so that it can write and rewrite its results. Then the neural network can be defined by the current state of memory. This is very similar to classic processors and computers.

The most famous and popular article — from DeepMind:

It seems like this is the key to understanding intelligence? But likely not. The system still requires a huge array of data for training. It mainly works with structured tabular data. Meanwhile, when Facebook solved a similar problem, they chose the path of 'screw memory, let's just make a more complex neural net with more examples— and it will learn by itself'.

Disentanglement. Another way to create meaningful memory is to take the same embeddings but introduce additional criteria during training that would allow distinguishing 'meanings' within them. For example, we want to train the neural network to distinguish a person's behavior in a store. If we took the standard approach, we would have to create a dozen networks. One searches for the person, the second determines what they are doing, the third their age, the fourth—gender. A separate logic looks at the part of the store where they are acting/learning on this. The third determines their trajectory, etc.

Or, if there were an infinite amount of data, one could train a single network on all possible outcomes (it is clear that such a dataset cannot be gathered).

The disentanglement approach tells us — let’s train the network so that it can distinguish concepts itself. It should form embeddings from video, where one area identifies an action, one — position on the floor over time, one — a person's height, and another — their gender. In this training, we would like to almost never guide the network towards these key concepts, allowing it to itself highlight and group areas. There are not many such articles (some of them 1, 2, 3) and in general, they are quite theoretical.

But this direction should theoretically address the problems listed at the beginning.

Decomposing an image by parameters such as 'wall color/floor color/object shape/object color/etc.'

Decomposing a face by parameters like 'size, eyebrows, orientation, skin color, etc.'

Other

There are many other, not so global areas that allow for reducing datasets, working with more heterogeneous data, etc.

Attention. It probably doesn't make sense to highlight this as a separate method. It’s just an approach that enhances others. Many articles are dedicated to it (1,2,3). The essence of Attention is to enhance the network's reaction specifically to significant objects during training, often through some external target guidance or a small external network.

3D simulation. If a good 3D engine is created, it can often cover 90% of the training data (I’ve even seen an example where a good engine covered almost 99% of the data). There are many ideas and hacks on how to make a network trained on a 3D engine work with real data (fine-tuning, style transfer, etc.). However, creating a good engine is usually several orders of magnitude more complex than gathering data. Examples of when engines were created:
Robot training (google, braingarden)
Training recognition of products in a store (but in the two projects we worked on — we managed without this).
Training at Tesla (again, the video mentioned above).

Conclusions

The entire article is in some sense a conclusion. Probably, the main message I wanted to convey is — 'the free ride is over, neural networks no longer provide simple solutions.' Now we need to work hard to build complex solutions. Or work hard by conducting complex scientific research.

Overall, this topic is debatable. Perhaps readers have more interesting examples?

Source: habr.com