The brain of the company. Start

The story "on the production theme" about the ways of introducing AI in a trading company. And what (hypothetically) it can lead to. The full version can be downloaded at Litres (free)

***

I wasn't a natural leader and I hated the meetings that other department heads were constantly calling. I didn't try to create a hype on the importance of my department. I just recruited guys with whom I could work and who had experience, unlike me. But I couldn't find the one I really needed by the headhunter. Such people do not look for work themselves, she finds them. I started watching reports at conferences on the topic, reading Habr. That was also hard to find. At the conferences there was not a single report with real results, everyone talked about new methods, but no one could demonstrate their application. They just didn't exist. When I tried to contact and ask questions, the speaker disappeared, only a couple answered that they really just calculated it all in excel. It was no better on Habré, fragments of translations of Western articles were the best materials on the topic. Only the comments were interesting.

The month has flown by. But I didn’t know where to start, what to do with this big data, how to tie it to the company’s tasks. Management has already hinted that it's time to present the plan. I have so far denied that it is necessary to more precisely formulate the tasks of the project, what we want to get from it at the end. They offered to get together and find out with the heads of departments, from which I understood that such an argument of the lack of a plan could not hold out for a long time. The staff found a girl who could describe business processes. According to all the guides, this was the first point in digitalization - first, algorithmize processes. I gave her a task, and I myself continued to search and went to meetings, where I continued to pretend to be smart.

From the comments, I learned that there are mashoba contests on Kagle. And cool people in mashob are cut there not for money, but for who is cooler. I unsubscribed to several winners of contests related to the topic and began to wait. Some nicknames were already familiar to me from the comments on Habré, and I was hoping that someone would answer. Two turned out to be employees of large companies, bound by all sorts of agreements, so they carefully bowed out. But the most interesting person did not answer. He won some of the coolest Kaggle contests on user segmentation, recommender systems, and even sales calculations based on 200 factors, including possible weather. It was what I was looking for! But he didn't answer. I started looking for him by nickname on the net. There was no information. But I saw it mentioned in the comments. So someone knew him. It was a chance. I asked in the comments who knows this, and one programmer answered me that he worked with him and could ask him for contacts for me.

He was invited by leading corporations, but he never worked in the office. And he didn't meet anyone. Even real photos of him on the Internet could not be found. I only knew his name and contacts on the network. It was somehow strange to offer to take such a company on staff for a project, but at a remote job. Since they were military men, they understood only the office-barracks situation “from bell to bell”. But there were no options, they needed someone who could make a cool car, since the company was already lagging behind, in their opinion, with the introduction of big data, but it was necessary to overtake everyone in order to become the first. And I had to go for broke in a conversation with management. But first I had to talk to him. His name was Max.

Team lead

- I would like to invite you as a team leader and architect to create all sorts of algorithms on the mashob. You seem to be interested in this topic. The company is decent, the money pays.
– I do not work for hire in companies, I work remotely on projects as long as they are interesting to me.
- But we are talking about a big project, you need to take on the task tightly, it is unlikely that this will be possible remotely.
- This is not a matter for discussion. With those who do not know how to work remotely, I do not work. Money can also be paid remotely. I'm not going to waste time going to the office and coming by all means at a certain time. This is stupid, and I don't do stupid things.
“Okay, remote work is fine. Are you ready to sign a contract to work remotely?
- It all depends on what you want there.
“Nothing special, you just have to make your own recommendation system for marketing, as well as customer segmentation based on big data and all that.
- It is not interesting.
- And what are you interested in?
- Something more serious, more global, but it seems that this is not about you. Thanks for the offer.
- Wait, let me tell you everything as it is, and then you decide. I got it in the ass - the company invited me to lead the implementation of mahoba methods in the company's work to increase efficiency, and I don't know what to offer. The company has everything - desire, yet trust in me, money. You can do anything, I just don't know what. Is it clear now?
“Understood, but not interesting. You don't even have a task. I advise you to start with this.
Max left the conversation. It was a failure. I barely found him, there is simply no other such cool guy in the car. I didn't have a chance to stay with the company. Another week, and I will be called to the carpet. I even asked for a couple of days off sick to buy time and think about what to do. Most likely, open a resume on a hunter.
Max showed up unexpectedly. He wrote on Skype:
- Hello. I see you're a good guy and the company seems to be big. If you have no ideas, then are you ready to let my ideas be realized?
- Certainly! Without even thinking, I answered immediately. – What are the ideas?
- There is an idea to automate the processes in the company completely, everything. And in marketing, and in logistics, and in procurement. Even in recruitment. And to make this large self-adjusting system for the required result - profit. How do you like this task?
“This is even more than my wildest fantasies. But is it possible? I have never seen such projects in the implementation. Has anyone already done this?
I'm not interested in doing what someone else has already done. I thought you understood it.
- Yes, of course, I wanted to say something else - are there any developments that allow you to do this?
It doesn't matter if they are or not. There is something to help us do this. Now reinforcement learning algorithms have appeared, maybe you have heard it already. If you think about it and bring them to mind, then this is a universal algorithm for everything. You set a goal as a reinforcement, and the system itself finds a way to achieve it. And it doesn't matter what the task is, if it is translated into a data set of the same format.
– What should I ask the management for a project besides your remote work? I can't even imagine how many people it would take to make such a complex system.
- A little. The nucleus will be one, it is a neuron with memory. Fast cluster in the data center.
- What about people?
– We need three python programmers who know popular neuron libraries, one data scientist to prepare the data and monitor them. No, a couple, we will work in all directions at once. And one specialist in high-performance servers.
- There seems to be such a specialist, the company has its own data center.
– No, we need someone who can make the most high-performance cluster. You definitely don't have that. I know one, I'll talk to him if he's not busy. We will also need one specialist in databases to pair with him, we will put him for parsing on the network. We will need a lot of information from outside. Look for testers and analysts yourself, as many as you need. Probably enough for a start.
- I will try to get such resources from the management, but I think there will be no problems.
“Didn’t I tell you that my conditions are also changing?”
No, what is changing?
“I want a percentage, a percentage of profit growth.
- You're confusing me. They will not give a percentage to an unfamiliar guy on a remote site. I would coordinate your remote control, and that's a problem.
“I'm offering the company's electronic brains. Fully managing it, distributing tasks to managers and controlling their implementation. It will be a super system that will even decide for itself who to fire and who the company needs. She will have only one goal - profit. It will replace people and speed up operations, the cost of transactions will drop significantly. Profits will grow at a rapid pace. They can't do it without me. Therefore, the percentage. This is true.
- I'll try. Let's briefly describe what you are proposing so that I can properly present your ambitions. What else should I tell them to make them agree to everything?
– That they will be the first.
When I tried to imagine how I would say this to the director, I was seized by a stupor. I didn't find the words. Is that read written by Max on a piece of paper. I prepared for a week, the director looked at me warily, not understanding what to expect from me. At the appointed time, I entered the conference room, where all the directors were already seated. The report passed in a blur. At the end, in the eyes of the meeting participants, I saw only one question - is it real or have you read fiction? The general spoke first:
- Can you do all of this? I understand that people and time will be needed. But you understood my question.
- I can't. There is a person who can. He is the best in this business, I hardly found him. He knows his worth and simply will not agree to make such a system. You have to go towards him.
- Let's discuss. Well done, the report exceeded my expectations. It is hard to believe, but the goal, probably, should be the maximum.
- If at least part of this can be implemented, we will get a huge effect, I just calculated it.
- Then you will show, we will not detain the others. The meeting is over.

Everyone took turns complimenting me and clapping me on the back as they left. Having stayed with the general, I immediately told him about Max's conditions in his own words. The General thought for a few seconds. “We need to draw up a good contract,” he said in the end. It meant yes. He also asked to speak with each director of his part of the project and draw up a general implementation plan, preferably with deadlines. He will present it to the founders. He did not even ask about the resources, their allocation, apparently, was implied along with the approval of the project. Leaving, I was delighted with my coolness - the project was approved and, together with Max's conditions! I wrote to him right away. He answered succinctly: "I had no doubt who would refuse the profit."

It was necessary to decompose the plan by months and next sprints. Write applications for people. I needed statistics from analysts, documentation on processes in ERP from the development department, and much more. Everything had to be put together in order to understand where to start and what to deal with. Everyone responded kindly to requests, but after a week I realized that no one was going to fulfill my requests. “There was no time, I’ll look tomorrow” is the standard answer. And it is not clear whether this is on purpose or really everyone is busy. In response, I myself began to receive some absurd requests. “Could you send a presentation on the digitalization of our interaction with suppliers, we have a conference tomorrow.” At first I was at a loss from such requests, but in the end I began to calmly do the same as they did with my requests. Ignore. There was no documentation, the data was only in the form of reports, not raw. Excel was the only analytics program. There was no talk of any uploads in BigQuery. Everything had to be done from scratch and by ourselves. The only thing that could be done quickly was to find people. And only thanks to the fact that I myself went to hh.ru and called guys with the competencies we needed for interviews. But I had no idea how to agree with the rest on the interaction on the project.

- Max, there are problems, for a week I ask you to give me data and documentation, but for now it’s solid breakfasts. This is not a company, but some kind of swamp. Nobody needs anything, everyone is busy with their own affairs.
“Don’t worry, we don’t need anyone other than the team you assembled. And you need an API for raw data on customers, goods and sales, all transactions, as well as mail to customer addresses, telephony to their numbers, and that's it for now. Get it done, go straight to the IT director. It seems that in the company only the management needs the project.
“Unfortunately, you are right,” I answered Max with sad emoticons.
Before that, I had only worked in small companies where everyone was practically in the same room and everyone was eager to help the other. It's not like that in big corporations. Managers at all levels are trying to portray vigorous activity by the number of orders to others. But no one immediately undertakes to do what they ask. They will first ask others if they can do it. And it seemed to me that they were competing in who would come up with the most, as if they were being paid money for it. No one is thinking about implementation anymore, the main thing is to hold a meeting and plan something. Since no one reduces or monitors plans, 90% of such initiatives are simply forgotten in the stream of new ones. Behind this self-sufficient flow of internal information, continuously generated by managers, no one sees the client anymore. Instead of customer reports and presentations. Kafka wrote that a large number of papers and laws are characteristic of dying empires. It was then that I got the idea that some managers have something to cut. Now I understand why Max did not agree to go to the office.

Client analysis

The team is assembled, and now it's time to plan sprints. At the command of the IT director, we were provided with some documentation and made an API. With a new team, we deployed a cluster in the data center on Hadoop and began to receive data.
- Where do we start? - I wrote to Max not without optimism.
- From what is simpler, to work well in a team. We will do customer analysis. The most understandable topic so far, and the data is there. How do you currently organize advertising on the site? How is mailing done? I don’t ask about the rest, there is hardly anything else.
- Until I fully understood it myself, the webmaster puts banners on the sites on the instructions of the one who asks. Banners are made by marketing. The webmaster made himself an admin panel in order to somehow keep track of banners and quickly remove them if asked. Letters are sent through a cloud application, analytics are uploaded with addresses, the content manager writes the text, the advertising manager sends letters after approval by his manager, he approves from others. Somehow, as I understand it.
What do you do with your hands? And how many different letters are sent out per month?
- Two three.
“I just don’t understand how a company with such an ancient approach has taken a significant share of the market. Last century. Let's start with this. I will find a suitable Java framework for creating interaction chains. Let's take a bourgeois cloud service as an analogue, register for now and analyze what is useful for us there. Let's start cutting into tasks.
– And what will be in the core of the system?
- Mashob, of course. I already told you that everything will be built on one core of a neuron that is self-learning according to goals. Marketing requires client analysis in order to quickly, right online, cluster users according to their parameters and actions on the site or in the mail. We will build an RFM analysis in order to track the stages. We will put tracking codes in letters and on the website, we will write everything to the database for each client. And then we wrap everything that is needed for automatic interaction with the client on this - a script for building a drag & drop interaction chain with automatic selection of a communication channel with the client, depending on where he sits. Or we send the task to the assigned manager by letter, if the client is completely deaf.
- A big plan, we have to do this for six months.
- No, I'm not an idiot to do everything myself. Let's do it faster.

A month later, the first prototype appeared. And it was fantastic for marketing. In the system, it was possible to make hundreds of segments for hundreds of collected data on customers, to build a guaranteed contact interaction chain for each segment. This is when the chain first tries to show the banner to the client, if it fails, it sends a letter, if it didn’t open, then it sends pushes to the application, if it didn’t look there either, then it sends a task to the manager assigned to the client with the text what needs to be done. All clients for which actions are needed fell into the network from such segments. At the same time, even the life cycle of the client was taken into account as a dynamic feature, whether he is a beginner or an experienced one, how often he makes purchases, whether he has already bought everything and whether he is going to back off. And this was also a sign for segmenting into chains. Clients' actions in response to a banner or a click in an email were also recorded in the database, and he could immediately get into the next chain. So the client could not leave the chains for months, the main thing was not to overdo it. We built the first welcome chains and abandoned carts ourselves.

The only thing that marketing had to do was to build such segments and chains and write a lot of texts and draw many hundreds of banners. Which, of course, they couldn't do right away. Max said that a little later he would make a system for automatically generating email texts and product banners from the product database. But while it was necessary to strain marketers. I was responsible in the team for interaction with other departments, and not only led the project.
But the real focus of the client analysis system was in the abilities based on the machine. Max presented them to the team personally. The system analyzed the customer's behavior and purchases and could tell in advance that the customer might leave. And sent the task to the manager for hold. The system knew better than managers what the client had already bought, and what he was most likely to buy, based on the typical basket of such clients. This is what we call the "basket approach". Moreover, the system itself calculated which banner or text of the letter was better to send, as it knew which text caused the most response from similar ones. It was like magic for me, I saw for the first time what a machine can do in a real business. The team started up, we worked like crazy, because the results delighted us.

- There is little data about clients in your corporate system, you know nothing about them, except for the company, position, industry and email. It's nothing. We do integration with external data providers. Request an agreement with SPARK. And I'll take care of the API with social networks.
- Exactly. Let's enrich the data. I recently saw another service here, which, according to comments on the social network, determines the psychotype of a person. It seems to me that this may be useful to us until I understand why, but I feel that it will not be superfluous.
- We will make recommendations to managers based on them. Give me the address. Just need to check how accurate it is. Something is hard to believe that they can determine without special tests.
- Better than the tests, they determine, I read. Temperament is at least better determined by the reaction to people's comments, and there is plenty of this on the net. Statistically, and not some kind of mood. And you can’t fake it, as in the tests.
- Okay, let's connect, give me the address. And pull up SPARK, by legal entities we will take information on the number in the state, turnover, founders, payments to the budget. Lots of interesting things there, too, useful. Even the contacts and addresses of your managers, as it turned out, cannot be trusted. They write all sorts of crap, just not to give out the contacts of their customers. Very dirty data from them.

Although there was still a lot to debug, after 3 months we made a wonderful system for marketing, but for some reason no one was in a hurry to use it. I wrote letters, held a meeting through the marketing director, approached personally, but no one made segments and chains, especially letters and banners. This was the first sabotage of the system, and I did not understand why. Until I was prompted by one analyst girl who works with marketers. We have made the system too transparent. Client analysis immediately showed how much each mailing brought in sales, which banner was clicked on, and which one was useless for customers. Previously, no one could calculate the effect of a mailing list or a banner on the fly, there were not even click statistics. And now everything is in full view - on the online dashboard, you can clearly see how sales are going on by mailing list. If they go. And this is the problem - no one had the practice of such online marketing, and everyone was afraid to expose their competencies. I wrote to Max.
“I said that they should all be fired,” Max answered expectedly. - It's okay, we'll have to make it more difficult, but we can do without them.
- Any thoughts on how?
– We cluster clients taking into account the type of activity and contacts before buying, so that all clients fall into some segment. And we will make a universal chain that will work through all channels - in the mail, on the website or in the application. Accounting for contacts will allow you to close chains into chains. And we will include the most important predictors - upsells, brand and series recommendations, churn with discounts for returns.
- And who will write the texts, they do not want to do them in such quantity.
- You need a lot of texts and banners, otherwise there will be no sense. Therefore, we will make automatic product banners and texts filled with products. Like widgets in Emarsys. Clients do not need especially artistic texts, marketing texts are only annoying.
So marketers will be completely out of work.
“And do not forget to report this to the management that the system works by itself. Without them. As we promised. And tell marketers: "to the labor exchange, baby."

This has been Max's favorite slogan since some time, when he himself believed in the viability of his algorithms. He had a goal that was the subject of an agreement with management - to reduce costs by reducing manual operations. If we automate the creation of letters and banners, this will be the first major success of the project.

Extension next post ...
(c) Alexander Khomyakov [email protected]

Source: habr.com