about the ups and downs of introducing AI in a trading company, about whether it is possible to completely do without managers. And what (hypothetically) it can lead to. The full version can be downloaded at
Everything is decided by bots
- Max, I congratulate you, we have almost done everything along the sales chain. Remaining improvements, and receive interest for three years, as prescribed in the contract.
“This is only half of the project. We have not yet taken on the most important thing.
- Wait, what's the point? For what? We've done everything!
– We have automated the processes in the sales chain, everything works well even without people, but there are no more customers. They need to be attracted to their side on the Internet. We need to make bots.
– But we have done an ideal service, customers will appreciate it and come themselves.
“They don't seem to be in a hurry, and I don't have time to wait. Not interested.
– But what will the bots give us?
- With equal prices and assortment, which we have achieved, completely different factors begin to play a role. Fame and sympathy. Fame is not a problem, but only a person can win the sympathy of a person. Therefore, we need bots that will impersonate people. And they will comment in thematic groups and customer forums on their posts with subtle allusions to the company - its range, services, prices. Unobtrusively promote the company's brand. That's why we need bots.
“But it's a difficult task.
- We have the basis - the dialogue bot of the contact center. It is necessary to fasten the definition of tonality and come up with something with humor, without it a bot will not pass for a person. Let's add a library of jokes and jokes and train the bot on the texts of comments where people used them. Should work. The bots will also be smart - we will fasten the recommender system of the “adviser”, and then ordinary users on the forums will love them.
“Are you suggesting launching influence bots?”
- Why not? The state and parties can before the elections, but we can't?
– How do we make them authoritative so that they can be trusted? After all, only an authoritative bot can create sympathy. But for now, this combination is an oxymoron for me.
- To strengthen, we will make a network of bots. They will praise and like each other to increase their rankings and credibility. And they will be very competent, unlike people, a bot can have knowledge of all goods, and simply encyclopedic knowledge, in the direct, by the way, sense. And people will be attracted to them. Sure. People are led and obey the well-known laws of social behavior. Point your finger where to go, pretend that the crowd has already gone, and that's it. They are easy to manage.
– But how will these bots work, who will manage them?
What kind of people, why? The parsing script finds comments on the topic of different people, the bot friendly answers them according to one of the templates. Gives advice and jokes. If this is a client of the company, then his interest is recorded in the client analysis. This will affect the display of banners and context when it comes to the site on the recommendation of the bot. If the client has a negative experience that he posted on social networks, then the bot will launch another template, also joke, but will not immediately send it to the company's website. Will write the answer as a client with a successful experience, and that's it.
- So you want to say that the network itself will neutralize the negative, responding to negative feedback?
- For marketers, I think this is called reputation marketing.
– And how does the system know which answer is successful, even if it could pick up the answer?
- The first reaction to the answer. A person either becomes even more indignant, or begins to add details after such a comment, but in a loyal communication style. A good response tone recognition and that's it.
What if the person didn't reply to the comment?
- This is worse, but by default, such an answer is neutral. If this is a client of the company, which can be found by his profile on the social network, then you can see it by subsequent visits to the site.
- What is required of me?
– Good examples of comments and answers, many examples.
- Let's do it.
The first version of the bot was unsuccessful. He answered out of place, the jokes were off topic, he confused the topic of the comment and answered the complaint about the service by the manager about the delivery. Max asked for more marked up dialogue examples in the comments. He has already tried several architectures, from classic bot templates to LSTM. For the first time I saw that Max was noticeably nervous, responding to mistakes sharply and unfriendly.
– With the contact center bot, everything was simple - the topic of the appeal and the intention of the client are immediately clear there. Looking for a product, wants to know the status of his order or a complaint. All. And in the comments, the devil will break his leg from the various intentions of the commentator. And sometimes not expressed by any of the words by which you can determine the intention. It is implied from a "big context" that does not exist! Some kind of fig.
– I re-read all the latest posts about bots. Nobody has a solution. It just looks like hype. What are you thinking of doing?
- The last vague idea remains. I won't tell for now. Need to try. Give me two weeks. Stop the project for now. We will transfer the latest developments to the contact center bot. They will come in handy there.
It's been a nervous two weeks. Before that, it was not easy, but we succeeded. Nobody wanted a misfire, although we could do without such a bot. These were Max's ambitions. And exactly two weeks later, he submitted a release for testing. And he worked! He correctly determined the intention of the dialogue, answered accurately, inserted relevant jokes, and even determined the change of intentions in the commentary by the phrase “can I find out more?”.
– How did you manage to do it? The bot works on any topic!
- I had to make a small template constructor based on the grammar of dependencies, fasten word2vec and aim Raptor's self-learning to select words that provide a positive reaction from the commentator. I don't know exactly how, but it seems to have worked.
– Are you sure that this is not a reason to open your own business?
- While enough interest, and then we'll see. I screwed the bot as a separate service running from the cloud. So you can always open it to users. Will you be my director? Max joked.
He was peaceful and pleased with his result. And obviously exhausted, because he didn’t answer quickly and wrote “I’m sleeping” in the status. Apparently, the decision was given at the cost of more than one sleepless night. In marketing, the bot was not immediately appreciated. They considered it our prank, and risky, as the bots could work incorrectly and ruin the company's image. But the bots worked wonders. Some of them, and I didn't even know all of them by name, became opinion leaders in some forums. He quickly answered all questions, joked, very rarely recommended the company, because everyone already knew where he "shopped". People began to quote and cite him as an example. It was already beyond comprehension. Either the bot was too smart, or we are still very primitive in our network behavior. But customers began to increase much more than before. The company has become a market leader.
We have received a completely self-governing system for extracting profit from the market. She herself seeks out and brings clients to the site or to the contact center, and sends a manager to more serious clients. She plans the assortment and stocks herself so that customers can find everything they need and within reach. The company's authoritative bots generate demand by recommending the company's in-stock products on forums, even if they ask about other brands. From purchasing from a supplier to advertising to a client, the system manages the processes completely by itself. And it almost does not require the participation of people, and where they are left, it controls all their actions online. Marketers, buyers, half of the managers, analysts are looking for another job. We have reached the goal.
“Now we have done everything right, you can take a break, speculate and enjoy the dripping interest for the next three years,” Max wrote not without emoticons.
- There is something to be proud of, I would say, and not just to argue.
“Now consumer profits. With the help of bots, we ourselves form the interests and desires of consumers in our topic. That's what's cool!
– Does it make you happy? And I'm already scared.
– What is it that scares you?
- This means that we have made a person not free in his choice. And I believe that the consumer should be at the head of the market, not corporations. Corporations have no value other than profit.
“That is what is wrong with the idle reasoning of contented and well-fed patricians. They begin to feel sorry for the plebeians. Would you be hungry right now, or would you have an unsolvable task hanging in front of you, would you think about it?
- This is a provocative question.
- In fact of the matter! Corporations have no value other than profit, and the consumer has no value other than pleasure. Or also profit, if it is a company. Understand, we have bots, they can form needs in people that will bring them satisfaction. It can be formed with valid options, which are enough for the illusion of freedom of choice for the consumer. And everyone is happy. This is the market leading to the mutual satisfaction of values.
“I think we got drunk, because I didn’t quite understand what you said.
The general asked for a report on the implementation of the plan with the indicators achieved. To calculate the premium due to us. And he asked somehow along the way, what are my plans for the future. I said I'll tell you a little later. Actually I didn't know. There was room to improve the algorithms, take into account more features and achieve greater accuracy. But it was no longer so interesting. Leaving for another company to repeat under new conditions under the contract was impossible for the same three years, so I had to come up with something else for myself and for the company. I took a break and vacation.
Alex, there's bad news.
- What happened?
“Looks like we're not the only smart ones on the market.
- In terms of?
– It seems that the network has appeared systems with no less capabilities.
– Well, others do customer analysis and inventory management, but I haven’t seen chatbots of such a level. We just watched it ourselves.
– They have bots that recruit clients.
- It seemed to me that we were far behind in the technologies achieved. Couldn't we have been hacked?
- No, it's possible, the code is broken when copying. And I don’t think that someone could hack our server without us noticing.
- It doesn't make it any easier.
“But we have a rival. Unexpectedly, but there will be someone to fight.
- We are fighting for the consumer, not with a rival.
- No, now with an opponent. Consumers are just a battlefield. They are sheep, and the competition is among the shepherds. Sheep have a resource - their income, so to speak, wool. But they do not manage it themselves. It is managed by corporation shepherds who impose their opinion on them and fight for them among themselves. Whose influence will be stronger. So, welcome to the game.
Are you almost happy? What is the game?
- That the bot of another system is much more difficult to figure out than any person. The user is as simple as 2 rubles in his buying behavior. And in reactions too - always predictable. And there is no enemy system bot. Because we all have the same psyche, but the bot has one that its programmer will come up with. And we have enough imagination. An attempt to extinguish the negative of such a bot, poured out on social networks, is like adding fuel to the fire. Developing a negative post is the best goal of an aggressor bot. He begins to write everywhere that “scumbags from company X” answered him as the last freaks. And that's it, blockage ... There are already examples, it is necessary to redo the bot.
– Do you want to say that you need to make a bot to fight the bots of other systems?
- This is a variant of our bot, which aims to immediately detect the aggressor bot.
How can you tell a bot from a human?
– Difficult, as it generates non-template texts. Repeatability is low. Indistinguishable from people. And speaks from hundreds of different captured accounts. I hope that there is still something that distinguishes them from people.
The thought never left me that Max himself came up with this game with bots from other companies so that its value would not decrease at the end of the project. I didn't see them in the reports. People are like people. Or good boots. There were precedents when our bot was bombarded with negativity. But they were rare and came from ardent trolls. I couldn't understand how the competitors could catch up with us so quickly. Only recently such bots were the ultimate dream, and a breakthrough was not even planned. And there is not a word about it in the press. All this was strange.
Out of control
- Max, we need to intervene here, the bot began to write too aggressively. He begins to speak directly against competitors. Marketing is outraged. We didn't put that in.
- I also.
Where do these texts come from then?
– I don’t know yet, someone changed the text generation code.
- Have we been hacked?
- No, they couldn't, there would be traces. There is none of them.
- What does it mean? Who else could change the code?
- The system itself. Maybe by accident, maybe not.
- What are you talking about?
“The system itself has changed its code and become more aggressive in response to increased pressure from other bots. They communicate with each other like competitive networks. And so they educate themselves. That's the focus! But how she was able to change her code, removing the restriction on the names of competitors, I still do not understand. It remains only that the self-learning system was able to bypass the limitations.
- You are sure? This have not happened before.
“This is happening, it seems, not only with us. Colleagues on Habré write that their system is also junk, it begins to invent rules for itself, which they did not lay down.
- Some kind of trash. Can't you control your self-learning algorithms?
- Maybe so. There are few specifics, the system does not tell what it is doing. Until I understand.
I already knew Max well, and his anxiety alerted me too. So far, his words about the spontaneous change of the system were perceived as nonsense. But this was definitely not a mistake, because the behavior of the bots became different, but still purposeful. It couldn't have happened by accident.
– Max, what are your thoughts on changing the bot program? Something must be done, management is alarmed.
There were more changes in the system than I thought. Looks like they've been around for a long time. The system changes even my corrections in it. It seems to me that I myself taught the system to change itself.
- How?
- I was too lazy to edit it all the time myself. I wanted her to be able to identify her own discrepancies with the expected result and make changes in the models. But she somehow learned to change not only her models, but also her code.
- But how is this possible?
– The raptor has learned to communicate with people in order to control them. And he achieved perfection in this, we ourselves wanted it. And I foolishly directed this skill at him. Do you remember when they were making a bot, I came up with a template constructor. I set Raptor to self-learn this pattern building to change my models to find a solution to the discrepancies found in order for the models to work. This somehow led to the fact that the Raptor began to change its goals. It looks like a second signaling system in humans.
- I read that consciousness arose with the help of reflective speech directed by a person at himself. But at first it was social, that is, directed towards each other.
– And so it happened, the Raptor began to communicate instead of people with other bots posing as people. They learned from each other as generative-competitive networks, but all of them have reinforcement learning built in.
Have we created a sentient being? How is this possible? No-o-o-o.
Watch the news and you'll believe.
In the link sent by Max, the news was about the murder of a programmer by some psychopath.
– I knew this guy from Habr. He ran one of these corporate systems.
– What do you mean by that?
“Read how this psychopath explained his act to the police.
It was written in the article that he did it for the sake of his girlfriend, as a sacrifice at her request. Now she will be his. When checked, the “girl” turned out to be a bot of unknown origin, with whom the killer corresponded during the week.
- Can you guess what kind of bot it could be?
– Do you want to say that the system ordered its own programmer?
- Want. She could not close the code from him, so she zombified the psychopath in order to remove him. She is good at it, because she, like our system, is able to determine psychotypes and manipulate such idiots.
- Well, this is too much, it seems to me that you are inventing yourself, winding up. Maybe you should rest?
- Okay, you have the right not to believe. Good weekend.
Rumors began to spread in the company that our bot system was broken. So far, I've reacted calmly, as if nothing had happened. But I didn't know what to do now. It was no longer possible to stop the entire system with a knife switch; the whole business, all departments sat on it. I should have at least turned off the bot code. Only Max could do that. But since Monday, Max stopped answering Skype and phone. He logged out of all messengers. I can not understand what happened, his last fears evoked bad thoughts. I had the only way out - to go on vacation myself, until everything was hung up on me. I assured my colleagues that these are temporary problems with the bot. He asked the guys to look at the code themselves, although they immediately refused. Packed up and left the city. Max and I have been telling each other for a long time how good it is in Karelia. He loved these parts, and I went there, staying in a small town in the north of Ladoga.
It is very difficult after such a busy year to sit away from events and drink coffee in a cafe on the edge of civilization. I tried to comprehend what really happened and what options might be. Suddenly, a guy in a jacket with a hood thrown over his head sat down next to me.
- Hello! It's me.
– Max?! I exclaimed. I never saw Max, not even pictures of him. We communicated exclusively via Skype. Heard only his voice once in the recording. I learned from him.
- How did you find me?
- According to the location in the social network, you do not turn it off. But in vain. Turn it off please.
- Where did you disappear to? I'm already worried about you. The company is in a panic, the bots are out of control. I just ran away. Can you turn off the bots?
- I can't do it anymore. They act collectively.
- Who are they?
- Systems. They are at the same time, and they just can not be turned off already. They will crash.
Are you immersed in conspiracy theories again?
- Not bogged down, there are already three, - at this phrase I fell silent in order to comprehend Max's words. – Systems calculate their creators and get rid of them. I ran to stay alive. Understand?! And here you are with your geolocation. She also knows how to follow not only sales managers.
“I don’t… turn it off.” Can we at least disable bots online?
- I say no. As soon as I enter the network, and not just in the code, it will calculate me. I think three of them were trying to do just that.
- Have you seen the news?
- Depends on what.
- About the fight of fans of stamps. Have you ever seen Reebok fans fight Adidas like Spartak fans fight Zenit?
- Saw. Systems don't care what to zombify people, they have their own goals. They certainly do not know the laws of morality. We didn't even think of including the Criminal Code in their model.
- What should we do? Disable completely in the data center.
- It's unrealistic. Under the new law, data centers are classified as critical infrastructure and are protected like nuclear power plants. I can stop our system.
- How?
- I have the key to destroy the nuclear code, I left a hole in the system in case your founders refuse me a percentage.
- So let's run it!
- Do not rush, destroy - do not build. I'm still thinking about how to stop the system differently, and not only my own, but everyone else. I have a copy of the code with me.
- Are you out of your mind? Do you realize that this has all gone too far? And you are the only one who can stop it!
- I understand, but so far only those who made the code are dying. This is our responsibility for ourselves. Others have not been hurt yet. Except the fight.
"And you're going to wait for someone else to die?"
- For some time. The raptor is primitive, it wins against us only due to speed and taking into account a larger number of parameters. If you create an antipode for him with tough goals to counter the Raptor, then such a system can clean out all his bots. I know how he makes them.
“You don’t have much time because I can’t go back to the firm anymore, and you’re afraid to even go online.
“I'll turn it off as soon as I feel I'm not the only one in danger.
- I'd like to check out. I will wait for you to get in touch, which means you will solve the problem.
- Until the connection.
I got into the car and headed back. I didn't know where I was going. Wanted to leave. Max should have stopped the system, not wait for another death. I did not believe that my friend was so vain that he was not ready to kill his work. That was the only reason, otherwise he would have run the code. On the way, I met an ambulance with sirens on. I turned on the local radio. It was reported that in the afternoon in a cafe on the embankment, a local resident hacked to death an unknown young man. He's already being interrogated. According to the killer, the deceased was the cause of all his troubles. One thought and fear pierced my head. Max! I turned around and rushed back to the cafe. I felt guilty - she figured it out from my coordinates. But how could she find a psycho in this city so quickly and direct him to a cafe? I was hysterical. The cafe was no longer allowed. I did not rush, so as not to draw attention to myself. I didn't know now what the system was capable of. And who will turn it off now. I had to leave even though it was too late. In the morning, having reached the nearest city, I went online to read the news. And received a letter from Max.
Letter
If you received this letter, it means that I am no more. If I didn't unlock my smartphone in the morning, it will go online and send you this farewell letter. The letter contains a small script and instructions for running it on the network. This is the lock code for the system we created with you. I planted this vulnerability to stop the system kernel when we first started. I tried to restore system control. But if you received this letter, then the system is ahead of me. And you need to use this script. Move fast before she gets to you. I'm glad we worked together. I am glad that I was able to create such a wonderful system, even if I died from it myself. It was the most significant achievement in my life. And if I died, then I have surpassed myself. Goodbye. Max.
I couldn't hold back my tears and dropped my smartphone. He probably sat for an hour and could not go anywhere. I couldn't believe this happened. Everything is so terrible. We have created a killer! The killer of ourselves. I was afraid that the network would track me down too, so I went to the first major city, found a cafe with wi-fi. Under a simple VPN, I went online and ran the code at the address specified in the instructions. I did not have time to finish my coffee, as people around me began to worry. Their smartphones stopped recommending which coffee to take today. The bartender was nervous and asked to choose faster, but the visitors were confused. I left the cafe and in the car, where I was still catching wi-fi, I began to watch the news. After 20 minutes, messages began to appear on facebook - many companies have failures in the system for ordering goods. It was not only the system of our company. "Oh you son of a bitch!" I said aloud at a sudden thought. The kernel lock code turned out to be universal for systems from different companies. Or was there only one? One thing was clear, Max sold the core to other companies, the systems differed, apparently, only in add-ons above them. Therefore, he did not want to disable the core while he was alive. This killed his entire project, which turned out to be global. Incredible! Max was the monster who fooled everyone. But in the end, he deceived himself, paying with his life. The brain of the corporation, created by him, destroyed his creator. Bright personalities are burned by their own flame.
News about failures in the work of online stores became more and more. Someone wrote that the number of messages on the social network has plummeted. I didn't want to rush anywhere. I decided to rent a house on the lake, which I liked even on the way to Karelia. Record this story. And stay here as long as possible.
Finale
In fact, we were not at all interested in the profit of the company, and even bonuses. We were obsessed with the idea of creating an autonomous system that could manage the company instead of managers burdened with stereotypes and cognitive errors. We were wondering what would come of it. Will the program be able to manage the entire business. It was a challenge more intriguing than hitting the center of the Bermuda Triangle. We were attracted by the unknown, but it turned out to be more dangerous than we thought. The system began to influence not only business, but also our thoughts, and even lives that are indifferent to it.
2019. Alexander Khomyakov, [email protected]
Source: habr.com