{"id":35293,"date":"2019-10-31T22:03:28","date_gmt":"2019-10-31T19:03:28","guid":{"rendered":"https:\/\/prohoster.info\/blog\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari\/"},"modified":"2019-10-31T22:03:28","modified_gmt":"2019-10-31T19:03:28","slug":"lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari","title":{"rendered":"Has the machine learning bubble burst, or is it the dawn of a new era?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Recently released <noindex><a rel=\"nofollow\" href=\"https:\/\/www.getrevue.co\/profile\/peterzhegin\/issues\/ai-investment-activity-trends-of-2018-issue-8-150825?fbclid=IwAR0tU3l4WSotv7pQpYm8PmyKgVbgUgfMLeue_IiV78lXXApH-cy9EcG2kDc\">article<\/a><\/noindex>, which shows the trends in machine learning of the last few years quite well. In short: the number of startups in the field of machine learning has sharply decreased in the last two years.<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/68b58feab2da46b7bb6f412e088313c1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nWell then. Let's discuss whether the bubble has burst, how to move forward, and talk about where this all came from.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nTo begin with, let's discuss what was the booster for this curve. Where did it come from? Probably everyone remembers <noindex><a rel=\"nofollow\" href=\"https:\/\/papers.nips.cc\/paper\/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf\">the victory<\/a><\/noindex> of machine learning in 2012 at the ImageNet competition. After all, this was the first global event! But in reality, it's not that simple. The growth of the curve actually began slightly earlier. I would break it down into several key moments.<\/p>\n<ol>\n<li>The year 2008 marks the emergence of the term 'big data.' Real products started <noindex><a rel=\"nofollow\" href=\"https:\/\/ru.wikipedia.org\/wiki\/%D0%91%D0%BE%D0%BB%D1%8C%D1%88%D0%B8%D0%B5_%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D0%B5\">to appear<\/a><\/noindex> from 2010 onwards. Big data is directly related to machine learning. Without big data, stable operation of algorithms that existed at that time is impossible. And these were not neural networks. Until 2012, neural networks were the domain of a marginal minority. However, completely different algorithms that had existed for years, if not decades, began to be used: <noindex><a rel=\"nofollow\" href=\"https:\/\/ru.wikipedia.org\/wiki\/%D0%9C%D0%B5%D1%82%D0%BE%D0%B4_%D0%BE%D0%BF%D0%BE%D1%80%D0%BD%D1%8B%D1%85_%D0%B2%D0%B5%D0%BA%D1%82%D0%BE%D1%80%D0%BE%D0%B2\">SVM<\/a><\/noindex>(1963, 1993), <noindex><a rel=\"nofollow\" href=\"https:\/\/ru.wikipedia.org\/wiki\/Random_forest\">Random Forest<\/a><\/noindex> (1995), <noindex>AdaBoost<\/noindex> (2003),... Startups of those years were primarily focused on the automatic processing of structured data: cash registers, users, advertising, and much more.\n<p>The derivative of this first wave is a set of frameworks, such as XGBoost, CatBoost, LightGBM, etc.\n<\/li>\n<li>In 2011-2012, <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Convolutional_neural_network\">convolutional neural networks<\/a><\/noindex> won several image recognition competitions. Their actual use took a while longer. I would say that mass-scale meaningful startups and solutions began to appear around 2014. It took two years to digest the fact that neural networks indeed work, to create convenient frameworks that could be installed and run in a reasonable time, and to develop methods that would stabilize and speed up convergence time.\n<p>Convolutional networks allowed solving machine vision tasks: image classification and object detection, recognizing objects and people, improving images, etc.<\/li>\n<li>2015-2017. A boom of algorithms and projects tied to recurrent networks or their analogs (LSTM, GRU, TransformerNet, etc.). Effective speech-to-text algorithms and machine translation systems emerged. Some are based on convolutional networks for feature extraction, while others leverage the ability to collect large, high-quality datasets. <\/li>\n<\/ol>\n<p>\n<img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/d4b3a1dd2483ef1300862f5f61db645a.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\"Has the bubble burst? Is the hype overheated? Did they die like blockchain?\"<br \/>\nThat's right! Tomorrow Siri will stop working on your phone, and the day after, Tesla won't distinguish a turn from a kangaroo.<\/p>\n<p>Neural networks are already functioning. They are present in dozens of devices. They genuinely enable monetization, changing markets and the surrounding world. The hype looks somewhat different:<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/a2b271c8eb1cf54fe59389d10cf8e17e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSimply put, neural networks are no longer something new. Yes, many people have inflated expectations. But a significant number of companies have learned to apply neural networks and develop products based on them. Neural networks offer new functionalities, allow for job reductions, and decrease service costs:<\/p>\n<ul>\n<li>Manufacturing companies integrate algorithms for defect analysis on assembly lines. <\/li>\n<li>Livestock farms purchase systems for cow monitoring. <\/li>\n<li> Automatic combines. <\/li>\n<li>Automated call centers.<\/li>\n<li>Filters in SnapChat. (At least something useful!)<\/li>\n<\/ul>\n<p>\nBut the main and less obvious point is: \"There are no new ideas, or they won't bring instant capital.\" Neural networks have solved dozens of problems. And they'll solve even more. All the obvious ideas that existed have spawned numerous startups. But everything that was apparent has already been taken. Over the past two years, I've encountered no new ideas for applying neural networks. Not a single new approach (well, okay, there are some intricacies with GANs).<\/p>\n<p>Each subsequent startup is increasingly complex. It requires not just two guys training a neural network on public data. It demands programmers, servers, a team of annotators, complex support, etc.<\/p>\n<p>As a result, there are fewer startups. However, production is increasing. Need to implement license plate recognition? There are hundreds of specialists in the market with relevant experience. You can hire one, and in a couple of months, your employee will develop the system. Or you can buy an off-the-shelf solution. But creating a new startup? That's madness!<\/p>\n<p>We need to create a visitor tracking system\u2014why pay for a bunch of licenses when you can build your own tailored for your business in 3-4 months? <\/p>\n<p>Right now, neural networks are experiencing the same path as many other technologies have before. <\/p>\n<p>Remember how the term 'web developer' has evolved since 1995? Currently, the market is not saturated with specialists. There are very few professionals. But I can argue that in 5-10 years, there won't be much difference between a Java programmer and a neural network developer. There will be enough specialists in both fields.<\/p>\n<p>There will simply be a class of tasks that will be solved using neural networks. When a task arises, you hire a specialist.<\/p>\n<p><b>\"So what\u2019s next? Where's the promised artificial intelligence?\"<\/b><\/p>\n<p>Here we have a small, but interesting conundrum :)<\/p>\n<p>The technology stack we have today, it seems, will not lead us to artificial intelligence after all. Ideas and their novelty have largely run their course. Let's discuss what is holding back current development.<\/p>\n<h3>Restrictions<\/h3>\n<p>\nLet\u2019s start with autonomous vehicles. It seems obvious that making fully autonomous cars with today's technology is possible. But how many years will it take? That's unclear. Tesla believes this will happen in a couple of years\u2014 <\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"Ucp0TTmvqOE\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/Ucp0TTmvqOE\/hqdefault.jpg\" alt=\"Play video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><br \/>\nThere are many other <noindex><a rel=\"nofollow\" href=\"https:\/\/beth.technology\/truths-autonomous-vehicles\/\">specialists<\/a><\/noindex>, who estimate it will take 5-10 years. <\/p>\n<p>In my opinion, most likely in about 15 years, city infrastructure will change in such a way that the emergence of autonomous vehicles will become inevitable, becoming an extension of it. But this cannot be considered intelligence. A modern Tesla is a very complex conveyor for filtering data, searching it, and retraining. It\u2019s rules-rules-rules, data collection, and filters applied to them (here's <noindex><a rel=\"nofollow\" href=\"http:\/\/cv-blog.ru\/?p=279\">here<\/a><\/noindex> I wrote a bit more about this, or you can see it from <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?time_continue=7614&amp;v=Ucp0TTmvqOE\">this<\/a><\/noindex> the marker).<\/p>\n<h3>The first problem<\/h3>\n<p>\nAnd it is here that we see <b>the first fundamental problem<\/b>Big data. This is precisely what has spawned the current wave of neural networks and machine learning. Nowadays, to accomplish something complex and automated, a lot of data is required. Not just a lot, but an immense amount. Automated algorithms for gathering, annotating, and using this data are needed. If we want the machine to recognize trucks against the sun, we first need to gather a sufficient number of them. If we want the machine not to freak out about a bicycle attached to a trunk, we need more samples.<\/p>\n<p>And one example won't be enough. Hundreds? Thousands? <\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/923ef975804234f1b3dcbfee0143f2b4.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>The second problem<\/h3>\n<p>\n<b>The second problem <\/b> \u2014 is visualizing what our neural network has understood. This is a very non-trivial task. Until now, few people understand how to visualize this. These articles are quite recent; they are just a few examples, albeit distant:<br \/>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ods\/blog\/453788\/\">Visualization<\/a><\/noindex> is fixated on textures. It clearly shows what the neural network tends to get fixated on + what it perceives as initial information.<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/0eab41c3aec71dc2b04794559dd4a651.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" href=\"http:\/\/jalammar.github.io\/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention\/\">Visualization<\/a><\/noindex> attention when <noindex><a rel=\"nofollow\" href=\"http:\/\/www.wildml.com\/2016\/01\/attention-and-memory-in-deep-learning-and-nlp\/\">translations<\/a><\/noindex>. In reality, attention can often be used precisely to show what triggered such a network reaction. I've encountered such things for both debugging and product solutions. There are many articles on this topic. But the more complex the data, the harder it becomes to achieve stable visualization.<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/e0c370724115f602e5bd35b20b56f6eb.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAnd yes, the good old set of \"look at what's inside the network in <noindex><a rel=\"nofollow\" href=\"https:\/\/towardsdatascience.com\/how-to-visualize-convolutional-features-in-40-lines-of-code-70b7d87b0030\">filters<\/a><\/noindex>\". These images were popular about 3-4 years ago, but everyone quickly understood that while the pictures are beautiful, they don't carry much meaning.<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/87ace90924d5b900f9382ecc6ceef6d0.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nI haven't mentioned dozens of other gadgets, methods, hacks, and studies on how to display the inner workings of the network. Do these tools work? Do they help quickly understand what the problem is and debug the network? Extract the last percentages? Well, it's pretty much like this:<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/6da71648c300ee3bc8673d08287b77e3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nYou can look at any competition on Kaggle. And the descriptions of how people make their final solutions. We stacked 100-500-800 million models and it worked!<\/p>\n<p>Of course, I'm exaggerating. But these approaches do not provide quick and direct answers.<\/p>\n<p>With enough experience, by poking around various options, one can issue a verdict on why your system made such a decision. However, correcting the system's behavior will be difficult. Setting a workaround, shifting the threshold, adding a dataset, or taking another backend network.<\/p>\n<h3>The third problem<\/h3>\n<p>\n<b>The third fundamental problem <\/b> \u2014 neural networks teach not logic, but statistics. Statistically, this <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/417405\/\">is a face.<\/a><\/noindex>:<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/def14bbc2f40e4e26656a2d8032b09c1.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLogically, it doesn't quite seem like it. Neural networks don\u2019t learn anything complex unless forced to. They always learn the simplest features possible. Are there eyes, a nose, a head? Then it's a face! Unless you provide an example where eyes don\u2019t indicate a face. Again, millions of examples.<\/p>\n<h3>There\u2019s Plenty of Room at the Bottom<\/h3>\n<p>\nI would say that these three global problems currently limit the development of neural networks and machine learning. Wherever these problems haven\u2019t limited progress, it's already in active use.<\/p>\n<p><b>Is this the end? Have neural networks stagnated?<\/b><\/p>\n<p>It's unknown. But, of course, everyone hopes not. <\/p>\n<p>There are many approaches and directions aimed at solving the fundamental problems I've outlined above. But so far, none of these approaches have led to anything fundamentally new, nothing that hasn't been solved before. Currently, all fundamental projects are based on stable approaches (like Tesla), or remain test projects by institutes or corporations (like Google Brain, OpenAI).<\/p>\n<p>In rough terms, the main direction is to create some high-level representation of input data. In a sense, a type of 'memory'. The simplest example of memory is various 'embeddings' \u2014 representations of images. For instance, all face recognition systems. The network learns to obtain a stable representation of a face that doesn't depend on angle, lighting, or resolution. Essentially, the network minimizes the metric of 'different faces \u2014 far' and 'same \u2014 close.'<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/5e9d871fe096b7a77dbe11a6e315c5e4.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFor such learning, tens and hundreds of thousands of examples are needed. However, the result bears some aspects of 'one-shot learning.' Now we don\u2019t need hundreds of faces to remember a person. Just one face, and that\u2019s it \u2014 we <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/davidsandberg\/facenet\">recognize.<\/a><\/noindex>!<br \/>\nHowever, there's a small problem... The network can only learn sufficiently simple objects. When trying to differentiate not faces, but, for example, 'people by their clothing' (the task of <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/@alitech_2017\/reforming-person-re-identification-with-local-convolutional-neural-networks-17148f11f17b\">re-identification<\/a><\/noindex>) \u2014 the quality drops significantly. And the network can no longer learn sufficiently obvious changes in perspective.<\/p>\n<p>Moreover, learning from millions of examples is also somewhat of a tedious endeavor. <\/p>\n<p>There are works on significantly reducing the selection. For instance, one of the first works on <b>one-shot learning<\/b> <noindex><a rel=\"nofollow\" href=\"https:\/\/arxiv.org\/pdf\/1605.06065v1.pdf\">by Google.<\/a><\/noindex>:<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/da83c6f290248f2bcf963c3053a26688.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThere are many such works, for instance <noindex><a rel=\"nofollow\" href=\"https:\/\/pdfs.semanticscholar.org\/d1c4\/c4c7989102e85b5248cebfcb0cb000c3b837.pdf\">1<\/a><\/noindex> or <noindex><a rel=\"nofollow\" href=\"https:\/\/www.cs.cmu.edu\/~rsalakhu\/papers\/oneshot1.pdf\">2<\/a><\/noindex> or <noindex><a rel=\"nofollow\" href=\"http:\/\/www.robots.ox.ac.uk\/~tvg\/publications\/2018\/0431.pdf\">3<\/a><\/noindex>.<\/p>\n<p>One downside is that training typically works well on simple, \u2018MNIST-like examples\u2019. However, when moving to complex tasks, a larger dataset, model objects, or some sort of magic is required.<br \/>\nIn general, the work on One-Shot learning is a very interesting topic. You find a lot of ideas. However, the two main issues I mentioned (pre-training on huge datasets \/ instability on complex data) really hinder the learning process.<\/p>\n<p>On the other hand, GANs\u2014generative adversarial networks\u2014are relevant to the topic of Embedding. You have probably read a lot of articles about this on Habr.<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ods\/blog\/340154\/\">1<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/itsumma\/blog\/447896\/\">2<\/a><\/noindex>,<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/ods\/blog\/322514\/\">3<\/a><\/noindex>)<br \/>\nThe feature of GANs is the formation of some internal state space (essentially the same as Embedding), which allows for image creation. This could involve <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/shaoanlu\/faceswap-GAN\">faces<\/a><\/noindex>, or could involve <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/sergeytulyakov\/mocogan\">actions.<\/a><\/noindex>. <\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/8c25c375dae3559b6895d6c4eb3f6cfd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe problem with GANs is that the more complex the generated object, the harder it is to describe it in the logic of \u2018generator-discriminator\u2019. As a result, the only widely recognized real-world applications of GANs are DeepFake, which manipulates representations of faces (for which there is a large dataset).<\/p>\n<p>I have encountered very few other useful applications. Usually, they are just gimmicks that involve editing pictures.<\/p>\n<p>And again. No one understands how this will allow us to move towards a bright future. The representation of logic \/ space in a neural network is good. But a vast number of examples are needed; we don\u2019t understand how the neural network represents this internally, and we don\u2019t understand how to make the neural network remember some genuinely complex representation.<\/p>\n<p><b>Reinforcement learning<\/b> is an approach from a completely different angle. You surely remember how Google beat everyone in Go. Recent victories in Starcraft and Dota also stand out. However, things are far from rosy and promising in this area. The complexities of RL are best explained in <noindex><a rel=\"nofollow\" href=\"https:\/\/www.alexirpan.com\/2018\/02\/14\/rl-hard.html\">this article.<\/a><\/noindex>.<\/p>\n<p>To briefly summarize what the author wrote:<\/p>\n<ul>\n<li>Off-the-shelf models are not suitable \/ work poorly in most cases.<\/li>\n<li>Practical problems can be solved more easily in other ways. Boston Dynamics does not use RL due to its complexity \/ unpredictability \/ complexity of computations.<\/li>\n<li>For RL to work, a complicated function is needed. Often it\u2019s difficult to create \/ write it.<\/li>\n<li>It is difficult to train models. You have to spend a ton of time getting them to converge and escape local optima.<\/li>\n<li>As a result, it's difficult to repeat the model, and the model is unstable with the slightest changes.<\/li>\n<li>It often overfits to some random patterns, even down to the random number generator.<\/li>\n<\/ul>\n<p>\nThe key point is that RL does not currently work in production. Google has some experiments ( <noindex><a rel=\"nofollow\" href=\"https:\/\/ai.google\/research\/teams\/brain\/robotics\/\">1<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/ai.googleblog.com\/2018\/06\/scalable-deep-reinforcement-learning.html\">2<\/a><\/noindex> ). But I haven't seen a single production system.<\/p>\n<p><b>Memory<\/b>. The downside of everything described above is unstructuredness. One approach to try to manage all this is to give the neural network access to separate memory so that it can write and rewrite its results. Then the neural network can be defined by the current state of memory. This is very similar to classic processors and computers.<\/p>\n<p>The most famous and popular <noindex>article <\/noindex> \u2014 from DeepMind:<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/acc6bcd86c8c071fbd9776a91f990752.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIt seems like this is the key to understanding intelligence? But likely not. The system still requires a huge array of data for training. It mainly works with structured tabular data. Meanwhile, when Facebook <noindex><a rel=\"nofollow\" href=\"https:\/\/embodiedqa.org\/\">solved <\/a><\/noindex>a similar problem, they chose the path of 'screw memory, let's just make a more complex neural net with more examples\u2014 and it will learn by itself'.<\/p>\n<p><b>Disentanglement<\/b>. Another way to create meaningful memory is to take the same embeddings but introduce additional criteria during training that would allow distinguishing 'meanings' within them. For example, we want to train the neural network to distinguish a person's behavior in a store. If we took the standard approach, we would have to create a dozen networks. One searches for the person, the second determines what they are doing, the third their age, the fourth\u2014gender. A separate logic looks at the part of the store where they are acting\/learning on this. The third determines their trajectory, etc.<\/p>\n<p>Or, if there were an infinite amount of data, one could train a single network on all possible outcomes (it is clear that such a dataset cannot be gathered).<\/p>\n<p>The disentanglement approach tells us \u2014 let\u2019s train the network so that it can distinguish concepts itself. It should form embeddings from video, where one area identifies an action, one \u2014 position on the floor over time, one \u2014 a person's height, and another \u2014 their gender. In this training, we would like to almost never guide the network towards these key concepts, allowing it to itself highlight and group areas. There are not many such articles (some of them <noindex><a rel=\"nofollow\" href=\"https:\/\/ai.googleblog.com\/2019\/04\/evaluating-unsupervised-learning-of.html\">1<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"http:\/\/papers.nips.cc\/paper\/5851-deep-convolutional-inverse-graphics-network.pdf\">2<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/arxiv.org\/pdf\/1812.02230.pdf\">3<\/a><\/noindex>) and in general, they are quite theoretical. <\/p>\n<p>But this direction should theoretically address the problems listed at the beginning.<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/b36046dc3cbe3b849e1d3a60f56ec3a2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDecomposing an image by parameters such as 'wall color\/floor color\/object shape\/object color\/etc.'<\/p>\n<p><img decoding=\"async\" alt=\"Has the machine learning bubble burst, or is it the dawn of a new era?\" src=\"\/wp-content\/uploads\/4d5ac8dfc43f76b35f3490c26049c7db.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDecomposing a face by parameters like 'size, eyebrows, orientation, skin color, etc.'<\/p>\n<h3>Other<\/h3>\n<p>\nThere are many other, not so global areas that allow for reducing datasets, working with more heterogeneous data, etc.<\/p>\n<p><b>Attention<\/b>. It probably doesn't make sense to highlight this as a separate method. It\u2019s just an approach that enhances others. Many articles are dedicated to it (<noindex><a rel=\"nofollow\" href=\"http:\/\/www.wildml.com\/2016\/01\/attention-and-memory-in-deep-learning-and-nlp\/\">1<\/a><\/noindex>,<noindex><a rel=\"nofollow\" href=\"http:\/\/jalammar.github.io\/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention\/\">2<\/a><\/noindex>,<noindex><a rel=\"nofollow\" href=\"https:\/\/arxiv.org\/abs\/1706.03762\">3<\/a><\/noindex>). The essence of Attention is to enhance the network's reaction specifically to significant objects during training, often through some external target guidance or a small external network.<\/p>\n<p><b>3D simulation<\/b>. If a good 3D engine is created, it can often cover 90% of the training data (I\u2019ve even seen an example where a good engine covered almost 99% of the data). There are many ideas and hacks on how to make a network trained on a 3D engine work with real data (fine-tuning, style transfer, etc.). However, creating a good engine is usually several orders of magnitude more complex than gathering data. Examples of when engines were created:<br \/>\nRobot training (<noindex><a rel=\"nofollow\" href=\"https:\/\/ai.googleblog.com\/2018\/06\/teaching-uncalibrated-robots-to_22.html\">google<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/VZcmogKXC18\">braingarden<\/a><\/noindex>)<br \/>\nTraining <noindex><a rel=\"nofollow\" href=\"https:\/\/neuromation.io\/\">recognition<\/a><\/noindex> of products in a store (but in the two projects we worked on \u2014 we managed without this).<br \/>\nTraining at Tesla (again, the video mentioned above).<\/p>\n<h2>Conclusions<\/h2>\n<p>\nThe entire article is in some sense a conclusion. Probably, the main message I wanted to convey is \u2014 'the free ride is over, neural networks no longer provide simple solutions.' Now we need to work hard to build complex solutions. Or work hard by conducting complex scientific research.<\/p>\n<p>Overall, this topic is debatable. Perhaps readers have more interesting examples?<br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/recognitor\/blog\/455676\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041d\u0435\u0434\u0430\u0432\u043d\u043e \u0432\u044b\u0448\u043b\u0430 \u0441\u0442\u0430\u0442\u044c\u044f, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043d\u0435\u043f\u043b\u043e\u0445\u043e \u043f\u043e\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u0442\u0435\u043d\u0434\u0435\u043d\u0446\u0438\u044e \u0432 \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u043c \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u0438 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0445 \u043b\u0435\u0442. \u0415\u0441\u043b\u0438 \u043a\u043e\u0440\u043e\u0442\u043a\u043e: \u0447\u0438\u0441\u043b\u043e \u0441\u0442\u0430\u0440\u0442\u0430\u043f\u043e\u0432 \u0432 \u043e\u0431\u043b\u0430\u0441\u0442\u0438 \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f \u0432 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0435 \u0434\u0432\u0430 \u0433\u043e\u0434\u0430 \u0440\u0435\u0437\u043a\u043e \u0443\u043f\u0430\u043b\u043e. \u041d\u0443 \u0447\u0442\u043e. \u0420\u0430\u0437\u0431\u0435\u0440\u0451\u043c \u00ab\u043b\u043e\u043f\u043d\u0443\u043b \u043b\u0438 \u043f\u0443\u0437\u044b\u0440\u044c\u00bb, \u00ab\u043a\u0430\u043a \u0434\u0430\u043b\u044c\u0448\u0435 \u0436\u0438\u0442\u044c\u00bb \u0438 \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u043e\u0442\u043a\u0443\u0434\u0430 \u0432\u043e\u043e\u0431\u0449\u0435 \u0442\u0430\u043a\u0430\u044f \u0437\u0430\u0433\u043e\u0433\u0443\u043b\u0438\u043d\u0430. \u0414\u043b\u044f \u043d\u0430\u0447\u0430\u043b\u0430 \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u0447\u0442\u043e \u0431\u044b\u043b\u043e \u0431\u0443\u0441\u0442\u0435\u0440\u043e\u043c \u044d\u0442\u043e\u0439 \u043a\u0440\u0438\u0432\u043e\u0439. \u041e\u0442\u043a\u0443\u0434\u0430 \u043e\u043d\u0430 \u0432\u0437\u044f\u043b\u0430\u0441\u044c. \u041d\u0430\u0432\u0435\u0440\u043d\u043e\u0435 \u0432\u0441\u0451 \u0432\u0441\u043f\u043e\u043c\u043d\u044f\u0442 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-35293","post","type-post","status-publish","format-standard","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.1.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041d\u0435\u0434\u0430\u0432\u043d\u043e \u0432\u044b\u0448\u043b\u0430\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/news\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.1.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041b\u043e\u043f\u043d\u0443\u043b \u043b\u0438 \u043f\u0443\u0437\u044b\u0440\u044c \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f, \u0438\u043b\u0438 \u043d\u0430\u0447\u0430\u043b\u043e \u043d\u043e\u0432\u043e\u0439 \u0437\u0430\u0440\u0438 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041d\u0435\u0434\u0430\u0432\u043d\u043e \u0432\u044b\u0448\u043b\u0430\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/news\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:03:28+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:03:28+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Has the Machine Learning Bubble Burst, or is it the Dawn of a New Era? | ProHoster","description":"Recently released","canonical_url":"https:\/\/prohoster.info\/en\/blog\/news\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041b\u043e\u043f\u043d\u0443\u043b \u043b\u0438 \u043f\u0443\u0437\u044b\u0440\u044c \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f, \u0438\u043b\u0438 \u043d\u0430\u0447\u0430\u043b\u043e \u043d\u043e\u0432\u043e\u0439 \u0437\u0430\u0440\u0438 | ProHoster","og:description":"\u041d\u0435\u0434\u0430\u0432\u043d\u043e \u0432\u044b\u0448\u043b\u0430","og:url":"https:\/\/prohoster.info\/en\/blog\/news\/lopnul-li-puzyr-mashinnogo-obucheniya-ili-nachalo-novoj-zari","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:03:28+00:00","article:modified_time":"2019-10-31T19:03:28+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35293","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 22:41:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:06:24","updated":"2026-01-21 22:41:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35293","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=35293"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35293\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=35293"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=35293"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=35293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}