In 2008, Big Data was a new term and a fashionable trend. By 2019, Big Data is a product for sale, a source of profit, and a reason for new legislation.
Last autumn, the Russian government initiated a bill to regulate big data. Identifying individuals from data is prohibited, but it is allowed upon request from federal authorities. Processing Big Data for third parties is only permitted after notifying Roskomnadzor. The law applies to companies with more than 100,000 network addresses. And of course, there will be registries – it is proposed to create one listing the data operators. If Big Data was not always taken seriously before, it now has to be considered.
As the director of a billing development company that processes Big Data, I cannot overlook Big Data either. I will reflect on big data through the lens of telecom operators, whose billing systems handle streams of information about thousands of subscribers every day.
Theorem
Let’s start like in a math problem: first, we will prove that telecom operators' data can be referred to as Big Data. Typically, big data is characterized by three V’s, although in free interpretations, the number of ‘V’s has reached even seven.
Volume. For instance, the MVNO Rostelecom serves over a million subscribers. Key host operators process data from 44 to 78 million people. Traffic grows every second: in the first quarter of 2019, subscribers already consumed 3.3 billion GB from mobile phones.
Velocity. No one tells the dynamics better than statistics, so I will reference Cisco's forecasts. By 2021, 20% of IP traffic will belong to mobile traffic, which will grow almost threefold over five years. A third of mobile connections will be M2M – IoT development will lead to a sixfold increase in connections. The Internet of Things will not only become profitable but also resource-intensive, so some operators will focus solely on it. Those who develop IoT as a separate service will benefit from double traffic.
Variety. Diversity is a subjective concept, but telecom operators do know almost everything about their customers. From names and ID data to phone models, purchases, visited locations, and interests. Media files must be stored under the Yarovaya law for at least six months. So let's take it as a given that the data collected is diverse.
Software and Methodology
Providers are among the main consumers of Big Data, which is why most data analysis techniques are applicable to the telecom industry. The question is who is willing to invest in the development of ML, AI, Deep Learning, and to invest in data centers and data mining. Effective work with databases consists of infrastructure and a team, the costs of which not everyone can afford. Companies that already have a corporate storage system or are developing Data Governance methodologies should invest in Big Data. For those still unprepared for long-term investments, I recommend gradually building the software architecture and implementing components step by step. Heavy modules and Hadoop can be left for last. Few purchase ready-made solutions for tasks like Data Quality and Data Mining; mostly companies tailor systems to their specific needs, either themselves or with the help of developers.
However, not every billing system can be modified for Big Data. In fact, not all can be modified. Few are capable of doing this.
Three signs that a billing system has a chance of becoming a database processing tool:
- Horizontal Scalability. The software must be flexible—after all, we are talking about large data. An increase in the amount of information should be met with a proportional increase in 'hardware' in the cluster.
- Fault Tolerance. Serious prepaid systems are typically fault-tolerant by default: billing operates in a cluster across several geolocations to automatically back each other up. There should also be enough computers in the Hadoop cluster in case one or more fail.
- Locality. Data must be stored and processed in one server, otherwise, data transmission can become very costly. One of the popular Map-Reduce approaches: HDFS stores, Spark processes. Ideally, the software should integrate seamlessly into the data center infrastructure and be capable of three in one: collecting, organizing, and analyzing information.
The command
The decision on what, how, and for what purpose the program will process big data is made by the team. Often it consists of a single person—the data scientist. However, in my opinion, the minimum set of employees for Big Data should also include a Product Manager, a Data Engineer, and a Lead. The first understands the services, translating technical language into layman's terms and vice versa. The Data Engineer brings models to life using Java/Scala and experiments with Machine Learning. The Lead coordinates, sets goals, and monitors the stages.
Issues
Problems often arise from the Big Data team regarding data collection and processing. The program needs to be told what to collect and how to process it—before that can be explained, one must understand it personally. And it’s not so straightforward with providers. I illustrate these problems using the example of reducing subscriber churn—this is precisely the issue that telecommunications operators try to tackle with Big Data first.
Task Setting. A well-drafted technical specification and the differing interpretations of terms have been a longstanding pain not only for freelancers. Even ‘lost’ subscribers can be interpreted in various ways—whether they haven't used the operator's services for a month, six months, or a year. To create an MVP based on historical data, one needs to understand the frequency of subscribers returning from churn—those who tried other operators' services or left the city and used a different number. Another important question is: how early before a subscriber's expected departure should the provider identify it and take action? Six months in advance is too early, and a week prior is already too late.
Misunderstanding Terms. Typically, operators identify a client by their phone number, so it makes sense to extract characteristics based on it. But what about the account number or the application number? It's essential to decide which unit should be considered a client to ensure data consistency within the operator's system. The assessment of a client's value is also questionable—what subscriber has more value to the company, for whom should more effort be made to retain, and which ones will inevitably churn, making it futile to spend resources on them?
Lack of Information. Not all provider employees are capable of explaining to the Big Data team what specifically affects subscriber churn and how potential factors are calculated in billing. Even if one of them is named – ARPU – it turns out that it can be calculated in different ways: either by periodic customer payments or by automatic billing accruals. And during the process, a million other questions arise. Does the model cover all customers, what is the cost of customer retention, is there any point in considering alternative models, and what to do with customers who are mistakenly being held back artificially?
Goal setting. I know three types of result-related errors that make operators frustrated with Big Data.
- The provider invests in Big Data, processes gigabytes of information, but receives a result that could have been achieved cheaper. Simple schemes and models are used, alongside primitive analytics. The cost is several times higher, yet the result is the same.
- The operator receives multifaceted data, but does not understand how to use it. Analytics is available – here it is, clear and comprehensive, but the outcome is zero. The end result, which cannot consist solely of the goal 'process the data,' has not been thought through. Processing is not enough – analytics must become the basis for updating business processes.
- Outdated business processes and unsuitable software for new goals can become an obstacle to utilizing Big Data analytics. This means that they failed at the preparation stage – they did not think through the action algorithm and the stages of implementing Big Data in operations.
Why
Speaking of results. I will briefly go over the ways to use and monetize Big Data that telecom operators are already employing.
Providers predict not only subscriber churn but also loads on base stations.
- Information about subscriber movements, activities, and frequency services is analyzed. The result: a reduction in the number of overloads due to the optimization and modernization of problematic areas of the infrastructure.
- Telecom operators use information about subscriber geolocation and traffic density when opening sales points. For example, MTS and VimpelCom are already using Big Data analytics for planning the location of new offices.
- Providers monetize their own big data by offering it to third-party companies. The main clients of Big Data operators are commercial banks. They use databases to track suspicious activities of subscriber SIM cards, to which the cards are linked, and utilize services for risk scoring, verification, and monitoring. In 2017, the government of Moscow requested movement dynamics from Tele2 based on Big Data for planning technical and transportation infrastructure.
- Big Data analytics is a goldmine for marketers who can create personalized advertising campaigns for thousands of subscriber groups if they wish. Telecom companies aggregate social profiles, consumer interests, and behavioral patterns of subscribers, and then use the collected Big Data to attract new customers. However, for large-scale promotion and PR planning, billing systems often lack the necessary functionality: the program must simultaneously consider numerous factors alongside detailed customer information.
While some still consider Big Data to be just a buzzword, the 'big four' are already making money from it. Over six months, MTS earns 14 billion rubles from big data processing, while Tele2 has increased revenue from its projects by three and a half times. Big Data is turning from a trend into a must-have, under which the entire structure of telecom operators will be restructured.
Source: habr.com
