TL;DR: Description of the client-server architecture of our internal network configuration management system, QControl. At its core is a two-level transport protocol that operates with gzip-packed messages without decompression between endpoints. Distributed routers and endpoints receive configuration updates, and the protocol allows for the installation of localized intermediate relays. The system is built on the principle of (“recent-stable,” explained below) and utilizes the JMESpath query language along with the Jinja templating engine for rendering configuration files.
Qrator Labs manages a globally distributed anti-attack neutralization network. Our network operates on an anycast principle, with subnets announced via BGP. Being a BGP anycast network physically located in multiple regions of the Earth allows us to process and filter illegitimate traffic closer to the core of the internet — Tier-1 operators.
On the other hand, being a geographically distributed network is not easy. Communication between network points of presence is critically important for a security service provider to maintain a consistent configuration across all network nodes, updating them in a timely manner. Therefore, in order to provide the maximum possible level of core service to the consumer, we needed to find a way to reliably synchronize configuration data across continents.
In the beginning was the Word. It quickly became a communication protocol in need of an update.
The cornerstone of QControl's existence and the main reason for spending a significant amount of time and resources on building such a protocol is the necessity to obtain a single authoritative source of configuration and ultimately synchronize our points of presence with it. The storage itself was just one of several requirements during the development of QControl. In addition, we also needed integrations with existing and planned services at the points of presence (PoPs), intelligent (and customizable) methods for data validation, as well as access control. Furthermore, we wanted to manage such a system through commands rather than modifying files. Prior to QControl, data was sent to the points of presence almost manually. If one of the points of presence was unavailable and we forgot to update it later, the configuration became unsynchronized—resulting in wasted time to bring it back online.
As a result, we devised the following scheme:
The configuration server is responsible for data validation and storage, while the router has several endpoints that receive and transmit configuration updates from clients and the support team to the server, and from the server to the points of presence.
Internet connection quality still varies significantly in different corners of the planet—for illustration of this thesis, let's look at a simple MTR from Prague, Czech Republic to Singapore and Hong Kong.
MTR from Prague to Singapore
The same to Hong Kong
High latency means lower speed. Additionally, there is packet loss. Bandwidth does not compensate for this issue, which must always be taken into account when building decentralized systems.
The complete configuration of a point of presence involves a significant amount of data that needs to be transmitted to many recipients over unreliable connections. Fortunately, although the configuration changes constantly, it does so in small increments.
Design recent-stable
It can be said that building a distributed network based on incremental updates is quite an obvious solution. However, there are numerous problems related to diffs. We need to retain all diffs between reference points and be able to resend them in case someone missed part of the data. Each destination must apply them in a strictly defined sequence. Typically, when there are multiple destinations, such an operation can take considerable time. The recipient must also be able to request the missed parts, and of course, the central part must respond to such a request correctly, sending only the missing data.
As a result, we arrived at a rather interesting solution — we have only one fixed reference layer, which we will call stable, and only one diff for it — recent. Each recent is based on the last constructed stable and is sufficient for reconstructing the configuration data. Once a fresh recent arrives at its destination, the old one is no longer needed.
We only need to periodically send the fresh stable configuration, for example, if the recent becomes too large. An important aspect here is that we send all of these updates in broadcast/multicast mode, without worrying about individual recipients and their ability to piece the data together. Once we are sure that everyone has the correct stable — we only send the new recent. Is it worth mentioning that this works? It works. Stable is cached on the configuration server and recipients, and recent is created as needed.
Two-Level Transport Architecture
Why did we build our transport on two levels? The answer is quite simple — we wanted to separate routing from high-level logic, drawing inspiration from the OSI model with its transport layer and application layer. We chose Thrift as the transport protocol and msgpack serialization format for high-level control messages. This is why the router (performing multicast/broadcast/relay) does not look inside msgpack, does not unpack and repack the content, and only performs data forwarding.
Thrift (from English 'thrift', pronounced as [θrift]) is an interface description language used to define and create services across different programming languages. It is a framework for remote procedure calls (RPC). It combines a software pipeline with a code generation engine for developing services that work effectively and easily between languages.
We chose the Thrift framework due to its RPC and support for multiple languages. As usual, the client and server were manageable components. However, the router turned out to be a tough nut to crack, partly because there was no ready-made solution during our development.
There are other options, like protobuf / gRPC, but when we started our project, gRPC was relatively new, and we hesitated to adopt it.
Of course, we could have (and probably should have) built our own solution. It would have been easier to create a protocol tailored to our needs since a client-server architecture is relatively straightforward to implement compared to building a router in Thrift. Nevertheless, there is a traditional bias against custom protocols and implementations of popular libraries (for good reason), and the question always arises: 'How will we port this to other languages?' So, we quickly discarded the idea of building our own.
Msgpack is an alternative to JSON, but faster and more compact. It is a binary data serialization format allowing data exchange between multiple languages.
At the first level, we have Thrift with the minimum information necessary for the router to forward messages. At the second level, there are packed msgpack structures.
We chose msgpack because it is faster and more compact compared to JSON. More importantly, it supports custom data types, allowing us to use cool features like sending raw binaries or special objects indicating the absence of data, which was important for our 'recent-stable' schema.
JMESPath
JMESPath is a query language for JSON.
This is what the description looks like from the official JMESPath documentation, but in reality, it offers much more. JMESPath allows you to search and filter subtrees in arbitrary tree structures, as well as apply changes to data on the fly. Additionally, it enables you to add special filters and data transformation procedures. Although it definitely requires some mental effort to understand.
Jinja
For some users, we need to convert the configuration into a file — that’s why we use a templating engine, and Jinja is the obvious choice. It allows us to generate a configuration file from a template and the data obtained at the destination.
To generate the configuration file, we need a JMESPath query, a template for the file's location in the file system, and a template for the config itself. At this stage, it’s also good to clarify the file access permissions. All of this can successfully be combined in one file — at the beginning of the configuration template, we place a header in YAML format that describes the rest.
For example:
---
selector: "[@][?@.fft._meta.version == `42`] | items([0].fft_config || `{}`)"
destination_filename: "fft/{{ match[0] }}.json"
file_mode: 0644
reload_daemons: [fft]
...
{{ dict(match[1]) | json(indent=2, sort_keys=True) }}
To create a configuration file for a new service, we only need to add a new template file. No changes to the source code or software at the points of presence are required.
What changed after the introduction of QControl into operational activities? The first and most important is the consistent and reliable delivery of configuration updates across all network nodes. The second is the acquisition of a powerful tool for verifying configurations and making changes by our support team as well as service consumers.
We managed to achieve all this by using the recent-stable update scheme to simplify communication between the configuration server and the configuration recipients. We utilized a two-level protocol to support a content-independent method of data routing. Successfully integrating a Jinja-based configuration generation engine into a distributed filtering network. This system supports a wide range of configuration methods for our distributed and diverse edge devices.
Thank you for your assistance in writing this material. , , .
post.
Source: habr.com
