In 2008, BigData was a new term and fashionable trend. In 2019, BigData is an object of sale, a source of profit and a reason for new bills.
Last fall, the Russian government initiated a bill to regulate big data. Individuals may not be identified from information, but may do so at the request of federal authorities. Processing BigData for third parties is only after notification of Roskomnadzor. Companies that have more than 100 thousand network addresses fall under the law. And, of course, where without registers - it is supposed to create one with a list of database operators. And if before this Big Data was not taken seriously by everyone, now it will have to be taken into account.
I, as the director of a billing developer company that processes this very Big Data, cannot ignore the database. I will think about big data through the prism of telecom operators, through whose billing systems flows of information about thousands of subscribers pass every day.
Theorem
Let's start, as in a mathematics problem: first we prove that the data of telecom operators can be called BigDat. Typically, big data is characterized by three VVV characteristics, although in free interpretations the number of “Vs” reached seven.
Volume. Rostelecom's MVNO alone serves more than a million subscribers. Key host operators handle data for 44 to 78 million people. Traffic is growing every second: in the first quarter of 2019, subscribers have already accessed 3,3 billion GB from mobile phones.
Velocity. No one can tell you about the dynamics better than statistics, so I’ll go through Cisco’s forecasts. By 2021, 20% of IP traffic will go to mobile traffic - it will almost triple in five years. A third of mobile connections will be M2M – the development of IoT will lead to a sixfold increase in connections. The Internet of Things will become not only profitable, but also resource-intensive, so some operators will focus only on it. And those who develop IoT as a separate service will receive double traffic.
Variety. Diversity is a subjective concept, but telecom operators really know almost everything about their subscribers. From name and passport details to phone model, purchases, places visited and interests. According to the Yarovaya law, media files are stored for six months. So let’s take it as an axiom that the data collected is varied.
Software and methodology
Providers are one of the main consumers of BigData, so most big data analysis techniques are applicable to the telecom industry. Another question is who is ready to invest in the development of ML, AI, Deep Learning, invest in data centers and data mining. Full-fledged work with a database consists of infrastructure and a team, the costs of which not everyone can afford. Enterprises that already have a corporate warehouse or are developing a Data Governance methodology should bet on BigData. For those who are not yet ready for long-term investments, I advise you to gradually build up the software architecture and install components one by one. You can leave the heavy modules and Hadoop for last. Few people buy a ready-made solution for problems such as Data Quality and Data Mining; companies generally customize the system to their specific specifications and needs - themselves or with the help of developers.
But not every billing can be modified to work with BigData. Or rather, not only everything can be modified. Few people can do this.
Three signs that a billing system has a chance to become a database processing tool:
- Horizontal scalability. Software must be flexible - we are talking about big data. An increase in the amount of information should be treated by a proportional increase in hardware in the cluster.
- Fault tolerance. Serious prepaid systems are usually fault-tolerant by default: billing is deployed in a cluster in several geolocations so that they automatically insure each other. There should also be enough computers in the Hadoop cluster in case one or more fail.
- Locality. Data must be stored and processed on one server, otherwise you can go broke on data transfer. One of the popular Map-Reduce approach schemes: HDFS stores, Spark processes. Ideally, the software should seamlessly integrate into the data center infrastructure and be able to do three things in one: collect, organize and analyze information.
Team
What, how and for what purpose the program will process big data is decided by the team. Often it consists of one person – a data scientist. Although, in my opinion, the minimum package of employees for Big Data also includes a Product Manager, Data Engineer, and Manager. The first one understands the services, translates technical language into human language and vice versa. Data Engineer brings models to life using Java/Scala and experiments with Machine Learning. The manager coordinates, sets goals, and controls the stages.
Problems
It is on the part of the BigData team that problems usually arise when collecting and processing data. The program needs to explain what to collect and how to process it - in order to explain this, you first need to understand it yourself. But for providers, things are not so simple. I’m talking about the problems using the example of the task of reducing subscriber churn - this is what telecom operators are trying to solve with the help of Big Data in the first place.
Setting goals. Well-written technical specifications and different understandings of terms have been a centuries-old pain not only for freelancers. Even “dropped” subscribers can be interpreted in different ways - as those who have not used the operator’s services for a month, six months or a year. And to create an MVP based on historical data, you need to understand the frequency of returns of subscribers from churn - those who tried other operators or left the city and used a different number. Another important question: how long before the subscriber is expected to leave should the provider determine this and take action? Six months is too early, a week is too late.
Substitution of concepts. Typically, operators identify a client by phone number, so it is logical that the signs should be uploaded using it. What about your personal account or service application number? It is necessary to decide which unit should be taken as a client so that the data in the operator’s system does not vary. Assessing the value of a client is also questionable - which subscriber is more valuable for the company, which user requires more effort to retain, and which ones will “fall off” in any case and there is no point in spending resources on them.
Lack of information. Not all provider employees are able to explain to the BigData team what specifically affects subscriber churn and how possible factors in billing are calculated. Even if they named one of them - ARPU - it turns out that it can be calculated in different ways: either by periodic client payments, or by automatic billing charges. And in the process of work, a million other questions arise. Does the model cover all clients, what is the price for retaining a client, is there any point in thinking through alternative models, and what to do with clients who have been mistakenly artificially retained.
Goal setting. I know of three types of outcome errors that cause operators to become frustrated with the database.
- The provider invests in BigData, processes gigabytes of information, but gets a result that could have been obtained cheaper. Simple diagrams and models, primitive analytics are used. The cost is many times higher, but the result is the same.
- The operator receives multifaceted data as output, but does not understand how to use it. There is analytics - here it is, understandable and voluminous, but it is of no use. The end result, which cannot consist of the goal of “processing data,” has not been thought through. It’s not enough to process – analytics should become the basis for updating business processes.
- Obstacles to the use of BigData analytics can be outdated business processes and software unsuitable for new purposes. This means that they made a mistake at the preparation stage - they did not think through the algorithm of actions and the stages of introducing Big Data into work.
Why
Speaking of results. I’ll go over the ways of using and monetizing Big Data that telecom operators are already using.
Providers predict not only the outflow of subscribers, but also the load on base stations.
- Information about subscriber movements, activity and frequency services is analyzed. Result: reduction in the number of overloads due to optimization and modernization of problem areas of the infrastructure.
- Telecom operators use information about the geolocation of subscribers and traffic density when opening points of sale. Thus, BigData analytics are already used by MTS and VimpelCom to plan the location of new offices.
- Providers monetize their own big data by offering it to third parties. The main customers of BigData operators are commercial banks. Using the database, they monitor suspicious activities of the subscriber’s SIM card to which the cards are linked, and use risk scoring, verification and monitoring services. And in 2017, the Moscow government requested movement dynamics based on BigData data from Tele2 to plan technical and transport infrastructure.
- BigData analytics are a gold mine for marketers, who can create personalized advertising campaigns for as many as thousands of subscriber groups if they choose. Telecom companies aggregate social profiles, consumer interests and behavioral patterns of subscribers, and then use the collected BigData to attract new customers. But for large-scale promotion and PR planning, billing does not always have enough functionality: the program must simultaneously take into account many factors in parallel with detailed information about clients.
While some still consider BigData an empty phrase, the Big Four are already making money on it. MTS earns 14 billion rubles from big data processing in six months, and Tele2 increased revenue from projects by three and a half times. BigData is turning from a trend into a must have, under which the entire structure of telecom operators will be rebuilt.
Source: habr.com