It seems that the field of online advertising should be as technological and automated as possible. Indeed, such giants and experts in their field as Yandex, Mail.Ru, Google and Facebook work there. But, as it turned out, there is no limit to perfection and there is always something to automate.
Communication group Dentsu Aegis Network is the largest player in the digital advertising market and is actively investing in technology, trying to optimize and automate its business processes. One of the unsolved problems of the Internet advertising market was the task of collecting statistics on advertising campaigns from different Internet sites. The solution to this problem eventually resulted in the creation of a product D1.Digital (read as DiVan), the development of which we want to talk about.
What for?
1. At the time of the start of the project, there was not a single ready-made product on the market that could solve the problem of automating the collection of statistics on advertising campaigns. This means that no one except ourselves will cover our needs.
Services such as Improvado, Roistat, Supermetrics, SegmentStream offer integration with sites, social networks and Google Analytics, and also provide the ability to build analytical dashboards for easy analysis and control of advertising campaigns. Before starting to develop our product, we tried to use some of these systems in our work to collect data from sites, but, unfortunately, they could not solve our problems.
The main problem was that the tested products were based on data sources, displaying placement statistics in sections by sites, and did not provide an opportunity to aggregate statistics on advertising campaigns. This approach did not allow to see statistics from different sites in one place and analyze the state of the campaign as a whole.
Another factor was that at the initial stages the products were focused on the Western market and did not support integration with Russian sites. And for those sites with which the integration was implemented, all the necessary metrics were not always uploaded with sufficient detail, and the integration was not always convenient and transparent, especially when it was necessary to get something that is not in the system interface.
In general, we decided not to adapt to third-party products, but started developing our own ...
2. The online advertising market is growing from year to year, and in 2018, in terms of advertising budgets, it overtook the traditionally largest TV advertising market. So there is a scale.
3. In contrast to the TV advertising market, where the sale of commercial advertising is monopolized, on the Internet there are a lot of individual owners of advertising inventory of various sizes with their own advertising cabinets. Since an advertising campaign, as a rule, goes on several sites at once, in order to understand the state of an advertising campaign, it is necessary to collect reports from all sites and combine them into one large report that will show the whole picture. So there is potential for optimization.
4. It seemed to us that the owners of advertising inventory on the Internet already have the infrastructure for collecting statistics and displaying it in advertising accounts, and they will be able to provide an API for this data. So, there is a technical possibility of implementation. Let's just say that it was not so simple.
In general, all the prerequisites for the implementation of the project were obvious to us, and we ran to bring the project to life ...
Grand plan
To begin with, we formed a vision of the ideal system:
- Advertising campaigns from the corporate 1C system with their names, periods, budgets and placements on various sites should be automatically loaded into it.
- For each placement within an advertising campaign, all possible statistics from the sites where the placement is taking place, such as the number of impressions, clicks, views, and so on, should be automatically downloaded.
- Some advertising campaigns are tracked using third party monitoring by so-called adserving systems such as Adriver, Weborama, DCM, etc. There is also an industrial measure of the Internet in Russia - the company Mediascope. According to our idea, the data of independent and industrial monitoring should also be automatically uploaded to the corresponding advertising campaigns.
- Most Internet advertising campaigns are aimed at certain targeted actions (purchase, call, sign up for a test drive, etc.), which are tracked using Google Analytics, and whose statistics are also important for understanding the state of the campaign and should be loaded into our tool .
The first pancake is lumpy
Given our commitment to agile software development principles (agile, everything), we decided to first develop the MVP and then move towards the intended goal iteratively.
MVP we decided to build on the basis of our product DANBo (Dentsu Aegis Network Board), which is a web application with general information on the advertising campaigns of our clients.
For MVP, the project was simplified as much as possible in terms of implementation. We have chosen a limited list of platforms for integration. These were the main platforms, such as Yandex.Direct, Yandex.Display, RB.Mail, MyTarget, Adwords, DBM, VK, FB, and the main adserving systems Adriver and Weborama.
To access statistics on the sites via the API, we used a single account. A client group manager who wanted to use automatic collection of advertising campaign statistics had to first delegate access to the necessary advertising campaigns on the platforms to the platform account.
Next, the user of the system DANBo had to upload a file of a certain format to the Excel system, in which all information about the placement was registered (advertising campaign, site, format, placement period, targets, budget, etc.) and identifiers of the corresponding advertising campaigns on the sites and counters in adserving systems.
It looked, frankly, terrifying:
The uploaded data was saved to the database, and then separate services collected campaign identifiers from them on the sites and downloaded statistics on them.
For each site, a separate windows service was written, which once a day went under one service account in the site API and unloaded statistics for the specified campaign IDs. The same thing happened with adserving systems.
The uploaded data was displayed on the interface in the form of a small self-written dashboard:
Unexpectedly for us, the MVP started working and began to upload up-to-date statistics on advertising campaigns on the Internet. We implemented the system on several clients, but when trying to scale, we ran into serious problems:
- The main problem was the laboriousness of preparing data for loading into the system. Also, the placement data had to be brought to a strictly fixed format before downloading. It was necessary to prescribe identifiers of entities from different sites in the file for loading. We encountered the fact that it is very difficult for technically untrained users to explain where to find these identifiers on the site and where they need to be put in the file. Given the number of employees in the divisions leading the campaigns on the sites, and the turnover, this resulted in a huge amount of support on our side, which we were absolutely not happy with.
- Another problem was that not all advertising platforms had mechanisms for delegating access to advertising campaigns to other accounts. But even if the delegation mechanism was available, not all advertisers were ready to provide access to their campaigns to third-party accounts.
- An important factor was the resentment factor, which caused users that all the planned indicators and placement details that they already enter into our 1C accounting system, they must re-enter into DANBo.
This prompted us to think that our 1C system should serve as the primary source of information about the placement, in which all data is entered accurately and on time KPIs). So a new concept of the system was born ...
Concept
The first thing we decided to do was to separate the system for collecting statistics on advertising campaigns on the Internet into a separate product - D1.Digital.
In the new concept, we decided to upload to D1.Digital information on advertising campaigns and placements within them from 1C, and then pull statistics from sites and from AdServing systems to these placements. This was supposed to greatly simplify the life of users (and, as usual, add work to developers) and reduce the amount of support.
The first problem we encountered was of an organizational nature and was due to the fact that we could not find a key or attribute by which it would be possible to compare entities from different systems with campaigns and placements from 1C. The fact is that the process in our company is arranged in such a way that advertising campaigns are entered into different systems by different people (media planners, buying, etc.).
To solve this problem, we had to invent a unique hashed key, DANBoID, which would link entities in different systems together, and which could be quite easily and unambiguously identified in the downloaded datasets. This identifier is generated in the internal 1C system for each individual placement and is transferred to campaigns, placements and counters on all sites and in all AdServing systems. It took some time to implement the practice of affixing DANBoID to all placements, but we managed it π
Further, we found out that not all sites have an API for automatically collecting statistics, and even those that have an API do not return all the necessary data.
At this stage, we decided to significantly reduce the list of sites for integration and focus on the main sites that are involved in the vast majority of advertising campaigns. This list includes all the largest players in the advertising market (Google, Yandex, Mail.ru), social networks (VK, Facebook, Twitter), major AdServing and analytics systems (DCM, Adriver, Weborama, Google Analytics) and other platforms.
The bulk of the sites we chose had an API that gave us the metrics we needed. In cases where there was no API, or there was no necessary data in it, we used reports that come daily to business mail to download data (in some systems it is possible to configure such reports, in others we agreed on the development of such reports for us).
When analyzing data from different sites, we found that the hierarchy of entities is not the same in different systems. Moreover, information from different systems must be downloaded in different detail.
To solve this problem, the SubDANBoID concept was developed. The idea of ββSubDANBoID is quite simple, we mark the main entity of the campaign on the site with the generated DANBoID, and we unload all nested entities with unique site identifiers and form SubDANBoID according to the DANBoID principle + first-level nested entity identifier + second-level nested entity identifier + ... This approach allowed us to connect advertising campaigns in different systems and download detailed statistics on them.
We also had to solve the problem of access to campaigns on different sites. As we wrote above, the mechanism of delegating access to a campaign to a separate technical account is not always applicable. Therefore, we had to develop an infrastructure for automatic authorization via OAuth using tokens and mechanisms for updating these tokens.
Further in the article, we will try to describe in more detail the architecture of the solution and the technical details of the implementation.
Solution architecture 1.0
Starting the implementation of a new product, we understood that it was necessary to immediately provide for the possibility of connecting new sites, so we decided to follow the path of microservice architecture.
When designing the architecture, we separated into separate services connectors to all external systems - 1C, advertising platforms and adserving systems.
The main idea is that all site connectors have the same API and are adapters that bring the site API to an interface that is convenient for us.
At the heart of our product is a web application, which is a monolith that is designed in such a way that it can be easily disassembled into services. This application is responsible for processing the uploaded data, collating statistics from different systems and presenting them to users of the system.
For connectors to communicate with the web application, we had to create an additional service, which we called Connector Proxy. It performs the functions of Service Discovery and Task Scheduler. This service runs data collection tasks for each connector every night. Writing a service layer was easier than connecting a message broker, and it was important for us to get the result as quickly as possible.
For simplicity and speed of development, we also decided that all services will be a Web API. This allowed us to quickly assemble proof-of-concept and check that the whole construction works.
Setting up access for collecting data from different accounts turned out to be a separate, rather complicated task, which, as we decided, should be carried out by users through the web interface. It consists of two separate steps: first, the user adds a token to access the account via OAuth, and then sets up data collection for the client from a specific account. Obtaining a token via OAuth is necessary because, as we already wrote, it is not always possible to delegate access to the desired account on the site.
To create a universal mechanism for selecting a cabinet from the platforms, we had to add a method to the connectors API that returns a JSON Schema, which is rendered into a form using a modified JSONEditor component. So users could choose accounts from which to download data.
To comply with the request limits that exist on the sites, we combine requests for settings within a single token, but we can process different tokens in parallel.
We chose MongoDB as the storage for the downloaded data for both the web application and the connectors, which allowed us not to worry too much about the data structure at the initial stages of development, when the application object model changes every other day.
We soon found out that not all data fits well in MongoDB and, for example, it is more convenient to store daily statistics in a relational database. Therefore, for connectors, the data structure of which is more suitable for a relational database, we started using PostgreSQL or MS SQL Server as storage.
The chosen architecture and technologies allowed us to build and launch the D1.Digital product relatively quickly. Over the two years of product development, we have developed 23 site connectors, gained invaluable experience working with third-party APIs, learned how to bypass the pitfalls of different sites, which each turned out to have its own, contributed to the development of the API of at least 3 sites, automatically downloaded information on almost 15 campaigns and for more than 000 placements, collected a lot of feedback from users on the operation of the product and managed to change the main process of the product several times based on this feedback.
Solution architecture 2.0
Two years have passed since the start of development D1.Digital. The constant increase in the load on the system and the emergence of new data sources gradually revealed problems in the existing solution architecture.
The first problem is related to the amount of data downloaded from the sites. We faced the fact that collecting and updating all the necessary data from the largest sites began to take too much time. For example, collecting data using the AdRiver adserving system, which we use to track statistics for most placements, takes about 12 hours.
To solve this problem, we started using all kinds of reports to download data from sites, we are trying to develop their API together with the sites so that the speed of its work meets our needs, and to parallelize data loading as much as possible.
Another problem is related to the processing of uploaded data. Now, when new placement statistics are received, a multi-stage process of recalculating metrics is launched, which includes loading raw data, calculating aggregated metrics for each site, comparing data from different sources with each other, and calculating summary metrics for the campaign. This causes a heavy load on the web application that does all the calculations. Several times the application in the process of recalculation ate up all the memory on the server, about 10-15 GB, which had the most detrimental effect on the work of users with the system.
The identified problems and grandiose plans for further development of the product led us to the need to revise the architecture of the application.
We started with connectors.
We noticed that all connectors work according to the same model, so we built a pipeline framework in which, to create a connector, it was only necessary to program the logic of the steps, the rest was universal. If some connector needs to be improved, then we immediately transfer it to the new framework at the same time as the connector is finalized.
In parallel, we started deploying connectors to docker and Kubernetes.
We planned to move to Kubernetes for a long time, experimented with CI / CD settings, but started moving only when one connector, due to an error, began to eat up more than 20 GB of memory on the server, practically killing other processes. During the investigation, the connector was moved to a Kubernetes cluster, where it ended up staying even when the bug was fixed.
Quite quickly, we realized that Kubernetes is convenient, and in six months we transferred 7 connectors and Connectors Proxy to the production cluster, which consume the most resources.
Following the connectors, we decided to change the architecture of the rest of the application.
The main problem was that data comes from connectors to proxies in large batches, and then they are beaten by DANBoID and sent to the central web application for processing. Due to the large number of metrics recalculations, there is a large load on the application.
It also proved quite difficult to monitor the statuses of individual data collection jobs and report errors occurring within the connectors to a central web application so that users can see what is happening and why data is not being collected.
To solve these problems, we developed the 2.0 architecture.
The main difference of the new version of the architecture is that instead of the Web API, we use RabbitMQ and the MassTransit library for messaging between services. To do this, we had to almost completely rewrite Connectors Proxy, making Connectors Hub out of it. The name was changed because the main role of the service is no longer in forwarding requests to connectors and back, but in managing the collection of metrics from connectors.
From the central web application, we separated information about placements and statistics from sites into separate services, which made it possible to get rid of unnecessary recalculations and store only already calculated and aggregated statistics at the level of placements. We also rewrote and optimized the logic for calculating the main statistics based on raw data.
In parallel, we are migrating all services and applications to docker and Kubernetes to make the solution easier to scale and easier to manage.
Where are we now
Proof-of-concept architecture 2.0 product D1.Digital ready and working in a test environment with a limited set of connectors. The only thing left to do is to rewrite another 20 connectors to the new platform, test that the data is loaded correctly, and all metrics are correctly calculated, and roll out the entire structure to production.
In fact, this process will happen gradually and we will have to leave backward compatibility with the old APIs in order for everything to continue to work.
Our immediate plans include the development of new connectors, integration with new systems and the addition of additional metrics to the data set downloaded from connected sites and adserving systems.
We also plan to migrate all applications, including the central web application, to docker and Kubernetes. In combination with the new architecture, this will significantly simplify the deployment, monitoring and control of consumed resources.
There is also an idea to experiment with choosing a database for storing statistics, which is currently stored in MongoDB. We have already transferred several new connectors to SQL databases, but there the difference is almost imperceptible, and for aggregated statistics by days, which can be queried for an arbitrary period, the gain can be quite serious.
In general, grandiose plans, moving on π
Authors of the article R&D Dentsu Aegis Network Russia: Georgy Ostapenko (), Mikhail Kotsik ()
Source: habr.com