Hello, Habr!
We remind you that following the book about we have released an equally interesting work about the library .
As the community is just beginning to explore the boundaries of this powerful tool. Recently, an article was published that we would like to introduce you to. The author shares from their own experience how to create a distributed data store using Kafka Streams. Happy reading!
Apache Library is used globally in enterprises for distributed stream processing on top of Apache Kafka. One of the underrated aspects of this framework is that it allows storing local state generated based on stream processing.
In this article, I will share how our company successfully leveraged this capability while developing a product for cloud application security. Using Kafka Streams, we created microservices with shared state, each serving as a fault-tolerant and highly available source of reliable information about the status of objects in the system. For us, this is a step forward in both reliability and ease of maintenance.
If you are interested in an alternative approach that allows the use of a single central database to support the formal state of your objects, take a look; it will be interesting...
Why we felt it was time to change our approaches to working with shared state
We needed to maintain the state of various objects based on reports from agents (for instance, was the site attacked?). Before transitioning to Kafka Streams, we often relied on a single central database (+ service API) for state management. This approach has its drawbacks: in supporting consistency and synchronization becomes a real challenge. The database can become a bottleneck or end up in a and suffer from unpredictability.
Illustration 1: a typical scenario with state separation encountered before transitioning to
Kafka and Kafka Streams: agents report their views through the API, and the updated state is calculated through the central database.
Meet Kafka Streams – it’s now easy to create microservices with shared state.
About a year ago, we decided to thoroughly review our workflows regarding shared state to address certain issues. We immediately decided to try Kafka Streams – known for its scalability, high availability, and fault tolerance, as well as its rich streaming functionality (including stateful transformations). It was exactly what we needed, not to mention how mature and reliable the messaging system has become in Kafka.
Each of the stateful microservices we created was built on an instance of Kafka Streams with a fairly simple topology. It comprised 1) a source 2) a processor with a persistent key-value store 3) a sink:
Illustration 2: The default topology of our stream instances for stateful microservices. Note: there is also a store here that contains metadata about scheduling.
With this new approach, agents create messages that are sent to the source topic, while consumers – say, a mail notification service – receive the computed shared state via the sink (the output topic).
Illustration 3: A new example of a task stream for a scenario with shared microservices: 1) an agent generates a message that goes to the Kafka source topic; 2) a stateful microservice (using Kafka Streams) processes it and writes the computed state to the Kafka sink topic; after which 3) consumers receive the new state.
Hey, that embedded key-value store is really useful!
As mentioned above, our topology with shared state includes a key-value store. We found several use cases for it, two of which are described below.
Option #1: Using the key-value store in computations
Our first key-value store contained auxiliary data that we needed for computations. For example, in some cases, the shared state was determined by a 'majority vote' principle. The store could hold all the latest reports from agents regarding the state of a certain object. Then, upon receiving a new report from a specific agent, we could save it, retrieve reports from all other agents about the same object from the store, and repeat the computation.
Below in Illustration 4, you can see how we opened access to the key-value store for the processor's processing method, allowing us to process a new message.
Illustration 4: Opening access to the key-value store for the processor's processing method (after that, in each scenario involving shared state, it is necessary to implement the method doProcess)
Option #2: Creating a CRUD API on top of Kafka Streams
Having set up our basic task stream, we tried to write a RESTful CRUD API for our microservices with shared state. We wanted to be able to retrieve the state of some or all objects, as well as to set or remove the state of an object (this is useful for supporting the backend).
To support all the Get State APIs, whenever we needed to recompute the state during processing, we would store it in the embedded key-value store for a long time. In this case, it becomes simple enough to implement such an API using a single instance of Kafka Streams, as shown in the listing below:
Illustration 5: Using an embedded key-value store to obtain the precomputed state of an object
Updating the state of an object via the API is also not complicated to implement. Essentially, you only need to create a Kafka producer and use it to make a record that contains the new state. This ensures that all messages generated through the API will be processed in exactly the same way as those coming from other producers (e.g., agents).
Illustration 6: You can set the state of an object using a Kafka producer
A small complication: Kafka has many partitions
Next, we wanted to distribute the processing load and improve availability by providing a cluster of microservices with shared state for each scenario. The setup was straightforward: after configuring all instances to work with the same application ID (and the same boot servers), almost everything else was done automatically. We also specified that each source topic would consist of several partitions so that each instance could be assigned a subset of those partitions.
I should also mention that it's common practice here to back up the state store, so that, for example, in the event of a recovery from failure, this backup can be transferred to another instance. For each state store in Kafka Streams, a replicable topic is created with a change log (which tracks local updates). This way, Kafka continuously ensures the safety of the state store. Therefore, in the event of a failure of any Kafka Streams instance, the state store can be quickly restored on another instance, where the corresponding partitions will be moved. Our tests showed that this can be done in just a few seconds, even if there are millions of records in the store.
Transitioning from a single microservice with shared state to a cluster of microservices makes implementing the Get State API not so trivial. In the new situation, the state store of each microservice contains only a part of the overall picture (those objects whose keys were mapped to a specific partition). We had to determine which instance contained the state of the object we needed, and we did this based on the stream metadata, as shown below:
Figure 7: Using the stream metadata, we determine which instance to request the state of the required object from; this approach was applied with the GET ALL API.
Main Findings
State stores in Kafka Streams can effectively serve as a distributed database,
- constantly replicated in Kafka.
- On top of such a system, a CRUD API can be easily built.
- Handling multiple partitions becomes somewhat more complex.
- It is also possible to add one or more state stores to the streaming topology to store auxiliary data. This option can be used for:
- Long-term storage of data needed for computations during stream processing
- Long-term storage of data that may be useful for subsequent initialization of the stream instance
- many other things…
Thanks to these and other advantages, Kafka Streams is well-suited for supporting global state in a distributed system like ours. Kafka Streams has proven to be very reliable in production (since its deployment, we have practically not lost any messages), and we are confident that its capabilities are not limited to this!
Source: habr.com
