🥇Application of low-code in analytical platforms

Dear readers, good day!

The task of building IT platforms for collecting and analyzing data inevitably arises for any company whose business is based on an intellectually intensive service model or the creation of technically complex products. Building analytical platforms is a complex and labor-intensive task. However, any problem can be simplified. In this article, I want to share my experience with low-code tools that assist in creating analytical solutions. This experience was acquired during the implementation of several projects in the Big Data Solutions division of Neoflex. Since 2005, Neoflex's Big Data Solutions division has been addressing issues related to building data warehouses and lakes, optimizing information processing speed, and working on data quality management methodologies.

No one can avoid the conscious accumulation of weakly and/or strongly structured data. Perhaps even in the case of small businesses. When scaling a business, a prospective entrepreneur will encounter questions regarding loyalty program development, want to analyze the effectiveness of sales points, consider targeted advertising, and become concerned about the demand for supplementary products. Initially, this task might be solved 'on the fly'. However, as the business grows, the transition to an analytical platform becomes inevitable.

But when can data analytics tasks evolve into 'Rocket Science' tasks? Perhaps, when it comes to truly big data.
To simplify the 'Rocket Science' task, you can eat the elephant in parts.

The greater the discreteness and autonomy of your applications/services/microservices, the easier it will be for you, your colleagues, and the entire business to digest the elephant.

This postulate has been accepted by nearly all of our clients, reshaping the landscape based on the engineering practices of DevOps teams.

But even with a 'separate, elephant-like' diet, we face a good chance of 'over-saturating' the IT landscape. At this moment, it is worth stopping, taking a breath, and looking towards low-code engineering platform.

Many developers fear the prospect of hitting a career dead-end when moving from direct coding to 'dragging' elements in low-code UI systems. However, the advent of machines did not eliminate engineers; it elevated their work to a new level!

Let's explore why.

Data analysis in logistics, telecom, media research, and the financial sector is always accompanied by the following questions:

The speed of automated analysis;
The ability to conduct experiments without impacting the main data production stream;
The reliability of prepared data;
Tracking changes and versioning;
Data provenance, Data lineage, CDC;
The speed of delivering new features to the production environment;
And the infamous: the cost of development and support.

That is, engineers have a vast number of high-level tasks that can only be efficiently accomplished by clearing their minds of low-level development tasks.

The prerequisites for developers transitioning to a new level are the evolution and digitalization of business. The value of a developer is also changing: there is a significant shortage of developers capable of diving into the essence of the concepts of automated business.

Let's draw an analogy between low-level and high-level programming languages. The shift from low-level languages to high-level languages is a transition from writing 'direct directives in the language of hardware' to 'directives in the language of people.' This means adding a layer of abstraction. In this case, the transition to low-code platforms from high-level programming languages is a shift from 'directives in the language of people' to 'directives in the language of business.' If there are developers who are saddened by this fact, they may have been disheartened since the advent of JavaScript, which utilizes array sorting functions. And these functions, of course, have software implementations under the hood using other means of the same high-level programming.

Thus, low-code is merely the emergence of yet another level of abstraction.

Practical experience with low-code

The low-code theme is quite broad, but right now I would like to discuss the practical application of "low-code concepts" using one of our projects as an example.

The Big Data Solutions division of "Neoflex" primarily specializes in the financial sector, building data warehouses and lakes and automating various reporting tasks. In this niche, the use of low-code has long been the standard. Among other low-code tools, we can mention solutions for organizing ETL processes: Informatica Power Center, IBM Datastage, Pentaho Data Integration. Or Oracle Apex, which serves as a rapid development environment for data access and editing interfaces. However, the use of low-code development tools is not always associated with building narrowly focused applications on a commercial technology stack with a clear vendor dependency.

Low-code platforms can also be used to orchestrate data streams, create data science environments, or, for example, modules for data quality checking.

One practical example of the use of low-code development tools is the collaboration between "Neoflex" and Mediascope, one of the leaders in the Russian media research market. One of the business tasks of this company is to produce data on which advertisers, online platforms, TV channels, radio stations, advertising agencies, and brands make decisions about advertising purchases and plan their marketing communications.

Media research is a technology-heavy sector of business. Video recognition, data collection from devices analyzing viewing patterns, measuring activity on web resources—this all implies that a company needs a large IT staff and substantial experience in building analytical solutions. However, the exponential growth of information, the number and variety of its sources compel the data IT industry to constantly progress. The simplest solution for scaling the already functioning Mediascope analytical platform could have been to increase the IT staff. But a far more effective solution is to accelerate the development process. One of the steps towards this goal could be the use of low-code platforms.

At the start of the project, the company already had a functioning product solution. However, the implementation of the solution on MSSQL could not fully meet expectations for scaling the functionality while maintaining an acceptable cost of development.

The task before us was truly ambitious – Neoflex and Mediascope had to create an industrial solution in less than a year, with the MVP scheduled for release within the first quarter from the start date of the work.

For the foundation of the new data platform, based on low-code computing, the Hadoop technology stack was chosen. The standard for data storage became HDFS, using parquet file format. Access to the data on the platform was facilitated by Hive, where all available views are represented as external tables. Data loading into the storage was implemented with the help of Kafka and Apache NiFi.

The low-code tool in this concept was applied to optimize the most labor-intensive task in building the analytical platform – the data calculation task.

The main mechanism for data mapping was the low-code tool Datagram. Neoflex Datagram is a tool for developing transformations and data flows.
Using this tool, it is possible to avoid writing Scala code "manually." Scala code is generated automatically using a Model Driven Architecture approach.

The obvious advantage of this approach is the acceleration of the development process. However, in addition to speed, there are also the following benefits:

Viewing the contents and structure of sources/targets;
Tracking the lineage of data stream objects down to individual fields;
Partial execution of transformations with the ability to view intermediate results;
Viewing and editing the source code before execution;
Automatic validation of transformations;
Automatic one-to-one data loading.

The entry barrier to low-code solutions for creating transformations is quite low: a developer needs to know SQL and have experience working with ETL tools. However, it's worth noting that code-driven transformation generators are not ETL tools in the broad sense of the term. Low-code tools may not have their own environment for code execution. This means that the generated code will run in the environment that existed on the cluster prior to installing the low-code solution. And this is perhaps another plus for low-code, as a 'classic' team can work alongside the low-code team, implementing functionality, for example, in pure Scala code. Integrating the enhancements from both teams into production will be straightforward and seamless.

It's also worth mentioning that in addition to low-code, there are no-code solutions. Essentially, these are different things. Low-code allows developers to intervene in the generated code to a greater extent. In the case of Datagram, it is possible to view and edit the generated Scala code, whereas no-code might not offer this capability. This difference is quite significant not only in terms of the flexibility of the solution but also regarding the comfort and motivation of data engineers.

Solution Architecture

Let’s try to understand how exactly a low-code tool helps solve the problem of optimizing the speed of data calculation functionality development. To begin, let's analyze the functional architecture of the system. In this case, we will take a data production model for media research as an example.

The data sources in our case are quite heterogeneous and diverse:

People meters (TV meters) are software and hardware devices that read user behavior from respondents of a television panel — who watched what channel, when, in a household participating in the study. The information provided consists of a stream of viewing intervals tied to the media package and media product. Data during the upload stage to the Data Lake can be enriched with demographic attributes, geographical ties, time zones, and other information necessary for analyzing the television viewing of a particular media product. The measurements obtained can be used for analyzing or planning advertising campaigns, assessing audience activity and preferences, and compiling broadcast schedules.
Data can come from systems monitoring streaming television broadcasts and measuring content views on internet video resources.
Measuring tools in the web environment include both site-centric and user-centric counters. A browser extension research bar and a mobile application with built-in capabilities can serve as data providers for the Data Lake. VPN.
Data can also come from platforms consolidating results from online survey forms and the outcomes of phone interviews conducted in the company's survey research.
Additional enrichment of the data lake can happen through uploading information from the logs of partner companies.

The implementation of as-is loading from source systems into the primary staging of raw data can be organized in various ways. If low-code is used for these purposes, automated generation of loading scripts based on metadata is possible. In this case, there is no need to delve into the development of source to target mappings. To implement automated loading, we need to establish a connection to the source and then define in the upload interface the list of entities to be loaded. The creation of directory structures in HDFS will occur automatically and will correspond to the data storage structure in the source system.

However, in the context of this project, we decided not to use this low-code platform capability because Mediascope has already begun work on creating a similar service based on Nifi + Kafka.

It should be noted from the start that these tools are not interchangeable, but rather complement each other. Nifi and Kafka can work in both direct (Nifi -> Kafka) and reverse (Kafka -> Nifi) configurations. The first option of the connection was used for the media research platform.

In our case, Nifi was required to process various types of data from source systems and send them to the Kafka broker. The direction of messages to a specific Kafka topic was accomplished using Nifi processors called PublishKafka. Orchestration and maintenance of these pipelines occur within a visual interface. The Nifi tool and the use of the Nifi + Kafka combination can also be considered a low-code approach to development, with a low entry threshold into Big Data technologies, accelerating the application development process.

The next stage in the project implementation was bringing the detailed data to a unified semantic layer format. In the case where an entity has historical attributes, the calculation is conducted in the context of the partition being considered. If the entity is not historical, either a recalculation of the entire object's contents may optionally occur, or there may be a complete refusal to recalculate this object (due to no changes). At this stage, keys for all entities are generated. The keys are saved in the corresponding master object directories in Hbase, containing the mapping between keys in the analytics platform and keys from source systems. The consolidation of atomic entities is accompanied by enrichment with the results of preliminary calculations of analytical data. The framework used for data calculation was Spark. The described functionality for bringing data to a unified semantics was also realized based on the mappings of the low-code tool Datagram.

In the target architecture, it was necessary to ensure SQL access to the data for business users. For this option, Hive was used. Registration of objects in Hive occurs automatically when the 'Register Hive Table' option is enabled in the low-code tool.

Management of the calculation flow

Datagram has an interface for designing workflow streams. Mappings can be executed using the Oozie scheduler. In the developer interface, you can create schemas for parallel, sequential, or conditionally dependent data transformation executions. It supports shell scripts and Java programs. The use of server Apache Livy. Apache Livy is used to run applications directly from the development environment.

If the company already has its own process orchestrator, it is possible to use the REST API to integrate mappings into the existing workflow. For example, we had quite a successful experience embedding mappings in Scala into orchestrators written in PLSQL and Kotlin. The REST API of the low-code tool includes operations such as generating an executable year based on the mapping design, calling a mapping, invoking a sequence of mappings, and, of course, passing parameters in the URL to trigger the mappings.

Alongside Oozie, it is possible to organize the calculation stream using Airflow. I won't dwell on comparing Oozie and Airflow; I'll simply say that in the context of the media research project, the choice leaned towards Airflow. The main arguments this time were a more active community developing the product and a more advanced interface + API.

Airflow is also great because it uses Python, a language loved by many, for process description. Moreover, there aren’t too many open-source workflow management platforms out there. The ability to launch and monitor process executions (including Gantt charts) only adds points to Airflow's karma.

The configuration file format for launching the low-code solution mappings became spark-submit. This happened for two reasons. First, spark-submit allows you to directly run a jar file from the console. Second, it can contain all the necessary information for configuring the workflow (which eases the script writing that forms the DAG).
The most commonly encountered element of the Airflow workflow in our case became the SparkSubmitOperator.

SparkSubmitOperator allows you to run jars—packaged Datagram mappings with pre-defined input parameters.

It should be noted that each Airflow task is executed in a separate thread and is unaware of other tasks. As a result, interaction between tasks is performed using control operators such as DummyOperator or BranchPythonOperator.

The combined use of the low-code solution Datagram along with the universalization of configuration files (forming Dag) has led to a significant acceleration and simplification of the data loading stream development process.

Calculation of dashboards

Arguably, the most intellectually demanding stage in the production of analytical data is the dashboard construction phase. In the context of one of the data calculation streams of the research company, this stage involves adapting to a standard broadcasting format, taking time zone adjustments into account. A local broadcasting grid adjustment (for local news and advertisements) is also possible. Among other things, this step includes breaking down intervals of continuous viewing of media products based on viewing interval analysis. Here, the 'weighting' of viewing values is performed based on their significance data (calculating the adjustment coefficient).

A separate step in preparing dashboards is data validation. The validation algorithm is associated with the application of several mathematical science models. However, the use of a low-code platform allows for breaking down a complex algorithm into a series of individual visually readable mappings. Each mapping performs a narrow task, allowing for intermediate debugging, logging, and visualization of the data preparation stages.

The validation algorithm was decided to be discretized into the following sub-stages:

Building regressions of network viewing dependencies in the region against the viewing of all networks in the region over 60 days.
Calculating studentized residuals (the deviations of actual values from those predicted by the regression model) for all regression points and for the calculation day.
Sampling anomalous pairs of region-network, where the studentized residual for the calculation day exceeds the norm (set by the operation configuration).
Recalculation of the adjusted studentized residual for anomalous pairs of region-television network for each respondent who watched the network in the region, defining the contribution of this respondent (the magnitude of the change in the studentized residual) while excluding this respondent's viewing from the sample.
Searching for candidates whose exclusion brings the studentized residual of the reporting day back to normal.

The example above serves as confirmation of the hypothesis that a data engineer has too much on their mind already… And if this is truly an 'engineer' and not a 'coder,' then the fear of professional degradation from using low-code tools should finally subside.

What else can low-code do?

The application area of low-code tools for batch and stream data processing without the need to manually write code in Scala does not end here.

The use of low-code in developing data lakes has already become somewhat of a standard for us. It can be said that solutions based on the Hadoop stack mirror the development path of traditional data warehouses based on RDBMS. Low-code tools on the Hadoop stack can address both data processing tasks and the creation of final BI interfaces. It is worth noting that BI can mean not only data representation but also their editing by business users. This functionality is often utilized by us when building analytical platforms for the financial sector.

Among other things, with the help of low-code and, in particular, Datagram, it is possible to solve the problem of tracking the lineage of data stream objects down to individual fields. For this, the low-code tool has implemented integration with Apache Atlas and Cloudera Navigator. Essentially, the developer needs to register a set of objects in the Atlas dictionaries and reference the registered objects when building mappings. The data lineage tracking mechanism or the analysis of object dependencies saves a significant amount of time when modifications to the calculation algorithms are necessary. For example, when preparing financial reports, this feature allows for a smoother transition during legislative changes. The better we understand the cross-entity dependencies in terms of the detailed layer objects, the fewer

Data Quality & Low-code

Another task implemented by the low-code tool in the Mediascope project was the Data Quality class. A distinctive feature of the data validation pipeline for the research company's project was its lack of impact on the functionality and speed of the main data processing stream. To orchestrate independent validation streams, the familiar Apache Airflow was used. As each step of data production was completed, a separate part of the DQ pipeline was launched in parallel.

It is considered good practice to monitor data quality from the moment of its inception in the analytics platform. With metadata information, we can check compliance with basic conditions — not null, constraints, foreign keys — from the moment the information enters the primary layer. This functionality is based on automatically generated mappings of the data quality family in Datagram. In the Mediascope project, the integration occurred with the product's metadata in Enterprise Architect.

Thanks to the integration of the low-code tool and Enterprise Architect, the following checks were automatically generated:

Check for the presence of null values in fields with the 'not null' modifier;
Check for duplicates of the primary key;
Entity foreign key validation;
Unique row validation based on a set of fields.

For more complex checks of data availability and validity, a mapping was created with Scala Expression, which takes external Spark SQL validation code prepared by analysts in Zeppelin as input.

Of course, the auto-generation of checks needs to be approached gradually. The following steps preceded this within the described project:

DQ implemented in Zeppelin notebooks;
DQ integrated into the mapping;
DQ in the form of separate massive mappings containing a complete set of checks for each individual entity;
Universal parameterized DQ mappings that accept metadata and business check information as input.

Perhaps the main advantage of creating a parameterized check service is reducing the time to deliver functionality to the production environment. New quality checks can bypass the classic code delivery pattern through development and testing environments:

All metadata checks are generated automatically when the model changes in EA;
Data availability checks (determining the presence of any data at a specific time) can be generated based on a reference guide that maintains the expected timing of the next batch of data per object;
Business data validity checks are created by analysts in Zeppelin notebooks. They are then directed straight to the configuration tables of the DQ module in the production environment.

The risks of directly deploying scripts to production are virtually nonexistent. Even in the event of a syntax error, the most we risk is the failure of a single check, as the data calculation stream and the quality check initiation stream are separated.

Essentially, the DQ service is permanently running in the production environment and is ready to start its work upon the arrival of the next batch of data.

In conclusion

The advantage of using low-code is clear. Developers no longer need to build applications from scratch. A programmer relieved of additional tasks delivers results faster. Speed, in turn, frees up extra time for addressing optimization issues. Therefore, in this case, we can expect a more quality and rapid solution.

Of course, low-code is not a panacea, and magic won’t just happen by itself:

The low-code industry is undergoing a phase of 'maturing', and there are currently no uniform industrial standards;
Many low-code solutions are not free, and their purchase should be a conscious step made with complete confidence in the financial benefits of their use;
Many low-code solutions do not always work well with GIT / SVN. They may also be inconvenient to use when the generated code is hidden;
When expanding the architecture, it may require additional work on the low-code solution— which, in turn, provokes the effect of 'dependency' on the low-code solution provider.
The required level of security is possible, but it is quite labor-intensive and complex to implement for low-code system engines. Low-code platforms should be chosen not only based on the principle of searching for benefits from their use. When choosing, one should consider the functionality for access management and delegation/escalation of identification data across the entire IT landscape of the organization.

However, if you are fully aware of all the shortcomings of the chosen system, and the benefits of its use are nonetheless overwhelmingly dominant, then transition to low-code without fear. Especially since the transition is inevitable— as is any evolution.

If one developer on a low-code platform can perform their tasks faster than two developers without low-code, it gives the company an advantage in every aspect. The entry threshold for low-code solutions is lower than for 'traditional' technologies, positively impacting the issue of workforce shortages. Using low-code tools can accelerate interaction between functional teams and speed up decision-making on the correctness of the chosen data science research path. Low-level platforms can be a catalyst for the digital transformation of an organization, as the solutions produced can be understood by non-technical specialists (especially business users).

If you have tight deadlines, heavy business logic, a shortage of technological expertise, and you need to accelerate your time to market, then low-code is one way to meet your needs.

It is important not to underestimate the significance of traditional development tools; however, in many cases, applying low-code solutions is the best way to enhance the efficiency of the tasks at hand.

Source: habr.com