Hello everyone, my name is Alexander, and I am a Data Quality engineer who focuses on data quality verification. In this article, I will discuss how I arrived at this point and why, in 2020, this field of testing gained significant momentum.
Global Trend
Today’s world is experiencing another technological revolution, one aspect of which is the utilization of accumulated data by various companies to boost their sales, profits, and public relations. It seems that having good (quality) data, along with skilled minds capable of monetizing it (correctly processing, visualizing, building machine learning models, etc.), has become the key to success for many. Whereas 15-20 years ago, only large companies were heavily involved in data accumulation and monetization, today it's a concern for nearly every sensible organization.
As a result, several years ago, job portals worldwide became flooded with Data Scientist vacancies, as everyone believed that hiring such specialists would lead to the creation of super machine learning models, predicting the future, and achieving a "quantum leap" for the company. Over time, people realized that this approach rarely works, as not all data available to these specialists is suitable for training models.
And the requests from Data Scientists began: "Let’s buy more data from this and that source...", "We need more data...", "We need some additional data, and it should be high quality...". Based on these requests, numerous interactions began to form between companies that owned various data sets. Naturally, this required a technical organization of the process — connecting to data sources, extracting them, ensuring they were fully loaded, etc. The number of such processes started to increase, and today we have a huge demand for a different type of specialist — Data Quality engineers — those who would monitor the flow of data in the system (data pipelines), check data quality at both input and output, and make conclusions about their sufficiency, integrity, and other characteristics.
The trend of Data Quality engineers has come to us from the USA, where in the midst of a raging era of capitalism, no one is willing to lose the battle for data. Below, I've provided screenshots from two of the most popular job search websites in the USA: and — which display data as of March 17, 2020, on the number of job postings found with the keywords: Data Quality and Data Scientist.
Data Scientists – 21,416 job openings
Data Quality – 41,104 job openings
Data Scientists – 404 job openings
Data Quality – 2,020 job openings
It is obvious that these professions do not compete with each other in any way. The screenshots were intended to illustrate the current job market situation regarding demands for Data Quality engineers, who are currently in much higher demand than Data Scientists.
In June 2019, EPAM, responding to the needs of the modern IT market, separated the Data Quality direction into a distinct practice. Data Quality engineers, in their everyday work, manage data, check its behavior in new conditions and systems, and monitor the relevance, sufficiency, and timeliness of the data. It is important to note that, in practical terms, Data Quality engineers do spend some time on classical functional testing, BUT though this greatly depends on the project (I will provide an example later).
The responsibilities of a Data Quality engineer are not limited to routine manual/automated checks for "nulls, counts, and sums" in database tables, but require a deep understanding of the client's business needs and, accordingly, the ability to transform available data into usable business information.
Data Quality Theory
To fully understand the role of such an engineer, let’s delve into what Data Quality is in theory.
Data Quality — one of the stages of Data Management (a whole world which we will leave for you to explore independently) and is responsible for analyzing data based on the following criteria:
I believe there is no need to decipher each of the points (theoretically referred to as "data dimensions"), they are quite well described in the picture. However, the testing process does not imply a strict copying of these attributes into test cases and their verification. In Data Quality, as in any other type of testing, it is essential to focus primarily on data quality requirements agreed upon with project participants who make business decisions.
Depending on the project, a Data Quality engineer can perform various functions, from a regular test automator with a superficial assessment of data quality to a person conducting deep profiling based on the above-mentioned attributes.
A very detailed description of Data Management, Data Quality, and related processes is excellently covered in the book titled “DAMA-DMBOK: Data Management Body of Knowledge: 2nd Edition”. I highly recommend this book as an introduction to the topic (you will find a link to it at the end of the article).
My story
In the IT industry, I have progressed from a Junior tester in product companies to a Lead Data Quality Engineer at EPAM. After about two years of working as a tester, I was firmly convinced that I had covered all types of testing: regression, functional, stress, stability, security, UI, etc. — and I tried a multitude of testing tools while working in three programming languages: Java, Scala, Python.
Looking back, I understand why my set of professional skills turned out to be so diverse — I participated in projects involving data work, both big and small. This is what led me into a world of many tools and growth opportunities.
To appreciate the variety of tools and opportunities for acquiring new knowledge and skills, just take a look at the picture below, which depicts the most popular ones in the world of "Data & AI."
Such illustrations are compiled annually by a well-known venture capitalist, Matt Turck, who comes from software development. Here is his blog and , where he works as a partner.
I grew professionally especially quickly when I was the only tester on the project, or at least at the beginning of the project. It is at such moments that you have to take responsibility for the entire testing process, and you have no option to retreat, only to move forward. At first, this was intimidating, but now all the advantages of such a trial are clear to me:
- You start communicating with the entire team like never before, as there is no proxy for communication: no test manager, no fellow testers.
- The immersion in the project becomes incredibly deep, and you have information about all components both in general and in detail.
- Developers do not see you as 'that guy from testing who does who knows what,' but rather as an equal, providing incredible value to the team with your automated tests and anticipation of bugs appearing in specific areas of the product.
- As a result, you become more effective, more qualified, and more sought after.
As the project grew, I became a mentor for the newly arrived testers in 100% of cases, teaching them and passing on the knowledge I had gained myself. Depending on the project, I did not always receive top-level automation specialists from management and there was a need to either train them in automation (for those interested) or create tools for them to use in their daily activities (data generation tools and their loading into the system, a tool for conducting stress testing/quick stability testing, etc.).
Example of a specific project
Unfortunately, due to confidentiality obligations, I cannot go into detail about the projects I have worked on, but I will provide examples of typical tasks for a Data Quality Engineer on one of the projects.
The essence of the project was to implement a platform for preparing data to train machine learning models. The customer was a large pharmaceutical company from the USA. Technically, it was a cluster , running on instances, with several microservices and based on the open-source project by the company EPAM — , adapted to the specific needs of the client (the project has now evolved into ). ETL processes were organized using and transferred data from the client's system to Buckets. Then, a Docker image of the machine learning model was deployed on the platform, which was trained on fresh data and provided predictions of interest to the business through a REST API interface, solving specific tasks.
Visually, it looked something like this:
Functional testing for this project was abundant, and considering the speed of feature development and the need to maintain the release cycle pace (bi-weekly sprints), it was essential to start thinking about automating the testing of the most critical system nodes right away. Most of the platform based on Kubernetes was covered by automated tests implemented in + Python, but maintaining and expanding them was also necessary. Additionally, for the client's convenience, a GUI was created to manage the machine learning models deployed on the cluster, along with the ability to specify from where and to where data needed to be transferred for training the models. This extensive addition led to an expansion of automated functional checks, which were mostly performed through REST API calls and a small number of end-to-end UI tests. Around the midpoint of this process, a manual tester joined us, who excelled in acceptance testing of product versions and communicating with the client regarding the acceptance of the next release. Furthermore, with the arrival of the new specialist, we were able to document our work and add several very important manual checks that were challenging to automate right away.
Finally, after achieving stability with the platform and the GUI built on top of it, we began constructing ETL pipelines using Apache Airflow DAGs. Automated data quality checks were performed by writing specific Airflow DAGs that validated data based on the results of the ETL process. In this project, we were fortunate to have access to anonymized datasets provided by the client, which we used for testing. Data was checked line by line for type conformity, presence of corrupted data, total record counts before and after, and comparisons of transformations made by the ETL process such as aggregation, changes to column names, and more. Additionally, these checks were scaled across different data sources, including both SalesForce and MySQL.
Final data quality checks were conducted at the S3 level, where the data was stored and ready-to-use for training machine learning models. To retrieve data from the final CSV file stored in the S3 Bucket and validate it, code was written using .
The client also required part of the data to be stored in one S3 Bucket and part in another. This necessitated writing additional checks to ensure the accuracy of such sorting.
General experience from other projects.
An example of the most generalized list of activities for a Data Quality engineer includes:
- Prepare test data (valid, invalid, large, small) using an automated tool.
- Upload the prepared dataset to the source system and check its readiness for use.
- Run ETL processes to handle the dataset from the source storage to the final or intermediate state using a specific set of settings (if configurable parameters for the ETL task are available).
- Verify the processed data from the ETL process for quality and compliance with business requirements.
The main focus of checks should not only be on whether the data stream in the system has been processed and completed (which is part of functional testing) but mainly on the verification and validation of data to ensure it meets expected requirements, identifies anomalies, and more.
Tools
One technique for such data control can be the organization of chain checks at each stage of data processing, known in literature as the 'data chain' — monitoring data from the source to the final usage point. This kind of check is most often implemented through the writing of validating SQL queries. It is clear that these queries should be as lightweight as possible and check specific pieces of data quality (tables metadata, blank lines, NULLs, syntax errors — other required quality attributes).
In the case of regression testing, which uses already prepared (unchanged or slightly modified) datasets, the code of automated tests may store ready-made templates for checking data quality (descriptions of expected table metadata; sample objects that can be randomly selected during the test, and more).
It is also necessary to write test ETL processes during testing, using frameworks such as Apache Airflow, or even black-box cloud tools like , and others. This requires the test engineer to delve into the principles of operation of the aforementioned tools and to conduct functional testing more effectively (for example, of existing ETL processes in the project) and to use them for data verification. In particular, Apache Airflow has ready-made operators for working with popular analytical databases, such as . The most basic example of its use has already been outlined , so I will not repeat it.
In addition to ready-made solutions, no one forbids you from implementing your own techniques and tools. This will not only benefit the project but also for the Data Quality Engineer, who will thereby enhance their technical knowledge and coding skills.
How this works on a real project
A good illustration of the recent paragraphs about 'data chain', ETL, and ubiquitous checks is the following process from one of the real projects:
Here, various data (naturally, prepared by us) enters the input 'funnel' of our system: valid, invalid, mixed, etc.; then they are filtered and placed in an intermediate storage, where they undergo a series of transformations before being placed in the final storage, from which analytics, data visualization, and business insights will be derived. In such a system, without functionally checking the ETL processes, we focus on the quality of the data before and after transformations, as well as on the output for analytics.
To sum up the above, regardless of where I have worked, I have always been involved in Data projects that shared the following features:
- Only through automation can certain cases be validated and an acceptable release cycle for business be achieved.
- A tester on such a project is one of the most respected team members, as they bring immense value to each participant (accelerating testing, providing good data to Data Scientists, identifying defects at early stages).
- It doesn't matter whether you are working on your own hardware or in the cloud — all resources are abstracted in a cluster like Hortonworks, Cloudera, Mesos, Kubernetes, etc.
- Projects are built on a microservices approach, with distributed and parallel computing predominating.
I would like to note that when engaged in testing in the realm of Data Quality, the testing specialist shifts their professional focus to the product code and the tools used.
Distinguishing features of Data Quality testing
In addition, I have identified the following (I must emphasize, VERY generalized and purely subjective) distinctive traits of testing in Data (Big Data) projects (systems) versus other areas:
Useful links
- Theory: .
- EPAM
- Recommended materials for a beginner Data Quality engineer:
- Free course on Stepik: .
- Course on LinkedIn Learning: .
- Articles:
- ;
- ;
- ;
- Video:
- ;
- ;
Conclusion
Data Quality — is a very young and promising field; being part of it means being part of a kind of startup. Entering Data Quality will immerse you in a wealth of modern, in-demand technologies, but most importantly, it will open up huge opportunities for generating and implementing your ideas. You will be able to apply a continuous improvement approach not only on the project but also for yourself, continuously growing as a specialist.
Source: habr.com
