Hello, Habr!
Data is the most valuable asset of a company. This is a claim made by nearly every digitally-focused company. It's hard to argue with that: no major IT conference passes without discussing approaches to data management, storage, and processing.
Data comes to us from outside, but it is also generated within the company. Speaking of data in a telecom company, it serves as a treasure trove of client information, their interests, habits, and locations for internal employees. With proper profiling and segmentation, advertising offers can become highly effective. However, in practice, it is not all that rosy. The data that companies store can often be hopelessly outdated, excessive, redundant, or known only to a narrow circle of users. ¯_(ツ)_/¯
In short, data must be managed effectively – only then will it become an asset that brings real benefits and profits to the business. Unfortunately, addressing data management issues requires overcoming quite a few challenges. These challenges are primarily due to historical legacies in the form of system 'zoo parks' and a lack of unified processes and approaches to data management. But what does it mean to 'manage data'?
This is exactly what we will discuss under the fold, as well as how the open-source stack has helped us.
The concept of strategic data management, Data Governance (DG), is already quite well known in the Russian market, and the goals that businesses achieve as a result of its implementation are clear and well-defined. Our company is no exception and has set the task of implementing the data management concept.
So, where did we start? First, we identified our key objectives:
- Ensure the availability of our data.
- Ensure the transparency of the data lifecycle.
- Provide users with consistent, non-contradictory data.
- Provide users with verified data.
Today, there are about a dozen data governance tools available on the software market.
However, after detailed analysis and study of the solutions, we noted a number of critical remarks:
- Most manufacturers offer a comprehensive set of solutions that we find excessive and redundant, duplicating existing functionalities. Additionally, integration into the current IT landscape is resource-intensive and costly.
- The functionality and interface are designed for technologists, not for end business users.
- Low product adoption rates and the lack of successful implementations in the Russian market.
- High costs of software and ongoing support.
The criteria and recommendations mentioned above regarding software import substitution for Russian companies convinced us to pursue our own development using an open-source stack. We chose Django as the platform – a free and open framework written in Python. Thus, we identified key modules that will contribute to the stated goals:
- Report registry.
- Business glossary.
- Module for describing technical transformations.
- Module for describing the data lifecycle from source to BI tool.
- Module for data quality control.
Report registry
According to internal research in large companies, employees spend 40–80% of their time searching for data when addressing data-related tasks. Therefore, we aimed to make information about existing reports publicly accessible, which were previously available only to clients. This reduces the time required to generate new reports and democratizes data access.
The report registry has become a single reporting window for internal users from various regions, departments, and divisions. It consolidates information about infocervices created in several corporate data warehouses, of which there are many at Rostelecom.
However, the registry is not just a dry list of developed reports. For each report, we provide the necessary information for users to familiarize themselves independently:
- a brief description of the report;
- data availability depth;
- customer segment;
- visualization tool;
- name of the corporate data warehouse;
- business functional requirements;
- link to the report;
- link to the access request;
- implementation status.
Reports provide insights into usage levels, with the reports ranking at the top based on log analytics of unique user counts. And that’s not all. In addition to general characteristics, we have included a detailed description of the attribute composition of reports with example values and calculation methods. Such detail immediately gives the user an answer on whether the report is useful for them or not.
The development of this module has been a significant step towards democratizing data and has greatly reduced the time spent searching for needed information. Along with reducing search times, there has also been a decrease in the number of requests to the support team for consultations. Another beneficial result achieved by developing a unified report registry is the prevention of duplicative report development for different structural units.
Business Glossary
As you all know, even within the same company, business speaks different languages. Yes, they use the same terms, but they mean entirely different things. The business glossary is intended to address this issue.
For us, the business glossary is not just a reference book with term descriptions and calculation methodologies. It is a comprehensive environment for developing, agreeing upon, and approving terminology, as well as establishing interconnections between terms and other informational assets of the company. Before being added to the business glossary, a term must go through all stages of approval with business stakeholders and the data quality center. Only after that does it become available for use.
As I mentioned above, the uniqueness of this tool lies in its ability to connect business terms to specific user reports in which they are used, as well as to the level of physical database objects.
This has been made possible by the use of glossary term identifiers in the detailed descriptions of reports from the registry and in the descriptions of physical database objects.
The Glossary currently includes over 4000 defined and agreed-upon terms. Its use simplifies and accelerates the processing of incoming requests for changes in the company's information systems. If the required metric is already implemented in any report, the user will immediately see a set of ready-made reports where this metric is used and can decide on the effective reuse of existing functionality or minimal modifications without initiating new requests for the development of a new report.
Module for describing technical transformations and Data Lineage
You may ask, what are these modules? It's not enough to just implement the Report Registry and Glossary; it's also necessary to map all business terms to the physical model of the databases. In this way, we were able to complete the process of forming the data lifecycle from source systems to BI visualization through all layers of the data warehouse. In other words, to build Data Lineage.
We developed an interface based on the format for describing the rules and logic of data transformation previously used in the company. The same information is entered through the interface as before, but it became mandatory to define the identifier of the term from the business glossary. This is how we establish a connection between the business and physical layers.
Who needs this? What was wrong with the old format, which we used for several years? How much has the labor cost of forming requirements increased? We faced such questions during the implementation of the tool. The answers here are quite simple — this is necessary for all of us, the data office of our company, and our users.
Indeed, employees had to adapt, and this initially led to slight increases in the time spent preparing documentation, but we addressed this issue. Practices, identification, and optimization of problem areas yielded results. We achieved our main goal — we enhanced the quality of the developed requirements. Mandatory fields, standardized directories, input masks, and built-in checks have significantly increased the quality of transformation descriptions. We moved away from the practice of delivering scripts as development requirements, sharing knowledge that was previously available only to the development team. The established metadata base dramatically reduces the time required for regression analysis and allows for quick assessment of the impact of changes across any layer of the IT landscape (showcase reports, aggregates, sources).
But what about regular report users, what benefits do they have? Thanks to the ability to construct Data Lineage, our users, even those unfamiliar with SQL and other programming languages, can quickly obtain information about the sources and objects on which a specific report is based.
Data Quality Control Module
Everything we've discussed above regarding data transparency is meaningless without understanding that the data we provide to users is correct. One of the key modules of our Data Governance concept is the data quality control module.
At the current stage, this is a catalog of checks for selected entities. The nearest goal for product development is to expand the list of checks and integrate with the reports registry.
What will this provide and for whom? For the end user of the registry, information will be available regarding the planned and actual readiness dates of the report, the results of the conducted checks with dynamics, and details about the sources loaded into the report.
For us, an integrated module for data quality within workflows means:
- Prompt formation of customer expectations.
- Decision-making regarding the further use of data.
- Obtaining a preliminary set of problem points at the early stages of work for the development of regular quality control.
Undoubtedly, these are the initial steps in establishing a full-fledged data governance process. However, we are confident that by purposefully engaging in this work and actively integrating Data Governance tools into our workflows, we will provide our clients with insightful information, a high level of trust in data, transparency in data acquisition, and enhance the speed of delivering new functionalities.
DataOffice Team
Source: habr.com
