Hello everyone! We have great news, in June OTUS is launching the course again , which is why we traditionally share useful material with you.
If you've encountered all this microservices story without any context, it's understandable to find it a bit strange. Breaking an application into fragments connected by a network inevitably means adding complex fault tolerance modes to the resulting distributed system.
Despite the fact that this approach involves splitting into many independent services, the ultimate goal is much greater than just getting these services to run on different machines. It's about interacting with the surrounding world, which in essence is also distributed. Not in a technical sense, but rather in terms of an ecosystem made up of many people, teams, and programs, each part of which somehow must carry out its function.
Companies, for example, represent a set of distributed systems that collectively contribute to achieving a certain goal. We ignored this fact for decades, trying to achieve unification by transferring files via FTP or using corporate integration tools, all while focusing on our individual isolated goals. But with the advent of services, everything changed. Services helped us look beyond the horizon and see a world of interdependent programs working together. However, to succeed, it is essential to understand and design two fundamentally different worlds: the external world, where we live in an ecosystem of many other services, and our personal, internal world, where we rule alone.
This distributed world differs from the one we grew up in and became accustomed to. The principles of building traditional monolithic architecture are open to critique. Therefore, the right understanding of such systems is more than just creating a cool scheme on a whiteboard or an impressive proof of concept. It's about such a system working successfully over a long time. Fortunately, services have been around for quite some time, although they come in various forms. still relevant, even when spiced up with Docker, Kubernetes, and slightly rumpled hipster beards.
So today we will look at how the rules have changed, why we need to rethink our approach to the services and the data they exchange with each other, and why we will need a completely different toolkit for this.
Encapsulation won't always be your friend.
Microservices can operate independently of one another. This characteristic gives them the most value. This same characteristic allows services to scale and grow. Not in terms of scaling to quadrillions of users or petabytes of data (although they can help with that too), but in terms of scaling from a human perspective, as teams and organizations continuously grow.
However, independence is a double-edged sword. While a service itself can run easily and effortlessly, if a function within the service requires another service, we ultimately have to make changes to both services almost simultaneously. In a monolith, this can be done easily; you just implement the change and release it, but synchronizing independent services poses more challenges. Coordination between teams and release cycles undermines flexibility.
In the standard approach, troublesome cross-cutting changes are simply avoided by clearly separating functionality among services. A gateway service can be a good example here. It has a well-defined role that distinguishes it from other services. This clear separation means that in a world of rapidly changing requirements for the surrounding services, the gateway service is unlikely to change. It exists within a strictly limited context.
The problem lies in the fact that in the real world, business services cannot consistently maintain clear role separation. For example, these same business services largely interact with data coming from other similar services. If you are in online retail, processing a flow of orders, product catalogs, or user information will become a requirement for many of your services. Each of the services will need access to this data in order to function.
Most business services utilize the same data flow, therefore their operations inevitably intertwine.
Thus, we arrive at an important point worth discussing. While services work well for infrastructure components that operate largely separately, most business services end up being intertwined much more closely.
Data Dichotomy
Service-oriented approaches may already exist, but there is still limited information on how to exchange large volumes of data between services.
The main issue is that data and services are inseparable. On one hand, encapsulation urges us to hide data so that services can be separated from one another, facilitating their growth and subsequent changes. On the other hand, we need to have the ability to freely share and exercise control over common data, just like any other. It is about being able to get to work right away, as freely as one would in any other information system.
However, information systems have little in common with encapsulation. In fact, quite the opposite. Databases do everything they can to provide access to the data they hold. They come with a powerful declarative interface that allows you to manipulate the data as you need. Such functionality is important during the preliminary research phase, but not for managing the growing complexity of an ever-evolving service.
And here lies the dilemma. The contradiction. The dichotomy. After all, information systems are about providing data, while services are about concealing it.
These two forces are fundamental. They underpin much of our work, constantly vying for supremacy in the systems we create.
As service systems grow and evolve, we see different manifestations of the consequences of data dichotomy. Either the service interface will expand, providing an increasingly broad set of features and start to resemble a very peculiar homegrown database, or we will face frustration and implement some way to extract or move entire datasets from service to service.
In turn, creating something that looks like a peculiar homegrown database leads to a host of problems. We won’t delve into the specifics of its dangers, shared database, let's just say it poses significant costly engineering and operational for a company trying to utilize it.
Worse yet, the volume of data amplifies the problems with service boundaries. The more shared data resides within a service, the more complex the interface becomes, and the harder it is to unify data sets coming from various services.
An alternative approach of extracting and moving entire datasets also has its issues. The common approach to this challenge looks like simply extracting and storing a dataset in its entirety, and then keeping it locally in each consumer service.
The problem is that different services interpret the data they consume differently. This data is always at hand. It changes and is processed locally. Quite quickly, it stops resembling the data in the source.
The more mutable the copies, the more the data will diverge over time.
What’s even worse is that such data is difficult to correct retrospectively ( might actually come to the rescue here). In fact, some of the intractable technological problems businesses face arise from heterogeneous data proliferating from application to application.
To find a solution to the problem of shared data, we need to think differently. They must become first-class objects in the architectures we build. refers to such data as "external," and this is a very important feature. We need encapsulation to avoid revealing the internal structure of the service, but we must facilitate access to shared data for services so that they can perform their tasks correctly.
The problem is that none of the approaches are currently relevant, as neither service interfaces, messaging, nor Shared Database offer a good solution for working with external data. Service interfaces are poorly suited for data exchange on any scale. Messaging moves data but doesn't keep its history, so over time the data becomes corrupted. Shared Databases are too focused on a single point, which stifles progress. We inevitably get stuck in a cycle of data failure:
Cycle of Data Failure
Streams: A Decentralized Approach to Data and Services
Ideally, we need to change the way services work with shared data. Right now, any approach faces the aforementioned dichotomy, as there is no magical dust that can be sprinkled generously to make it disappear. However, we can rethink the problem and come to a compromise.
This compromise implies a certain degree of centralization. We can take advantage of distributed logging mechanisms, as they provide reliable, scalable streams. Now, services need to be able to join and work with these shared streams; however, we want to avoid complex centralized God Services that perform this processing. Therefore, the best option is to embed stream processing in each service consumer. This way, services can combine datasets from different sources and work with them as needed.
One way to achieve such an approach is through the use of a streaming platform. There are many options available, but today we will specifically look at Kafka, as its Stateful Stream Processing allows us to effectively address the presented problem.
Utilizing the distributed logging mechanism allows us to follow a well-trodden path and use messaging to work with. . This approach is believed to provide better scalability and separation than the 'request-response' mechanism, as it gives control of the flow to the receiver instead of the sender. However, everything in life comes at a price, and you will need a broker. But for large systems, this compromise is worth it (which cannot be said for your average web applications).
If the broker is responsible for distributed logging instead of a traditional messaging system, additional features can be utilized. The transport can be linearly scaled almost as well as a distributed file system. Data can be retained in logs for a considerable time, providing not just messaging but also information storage. Scalable storage without the fear of obtaining a mutable shared state.
Then, a stateful stream processing mechanism can be employed to add declarative database tools to consumer services. This is a very important thought. As long as data is stored in common streams accessible to all services, the combination and processing performed by the service are private. They are isolated within a strictly limited context.
Eliminate the data dichotomy by separating the immutable state stream. Then add this functionality to each service using Stateful Stream Processing.
Thus, if your service needs to work with orders, product catalog, and inventory, it will have complete access: only you will decide which data to combine, where to process it, and how it should change over time. Even though the data is shared, working with it is entirely decentralized. It occurs within each service, in a world where everything goes by your rules.
Share data in a way that maintains its integrity. Encapsulate the function, not the source, in each service that needs it.
It often happens that data needs to be moved in bulk. Sometimes the service requires a local historical dataset in the chosen database engine. The focus is that it can be guaranteed that a copy can be restored from the source by accessing the distributed logging mechanism if needed. Connectors in Kafka handle this task excellently.
So, the approach we discussed today has several advantages:
- Data is utilized in the form of shared streams, which can be stored for a long time in logs, and the mechanism for working with shared data is integrated into each individual context, allowing services to operate easily and quickly. This way, it is possible to balance the data dichotomy.
- Data coming from various services can easily be merged into sets. This simplifies interaction with shared data and eliminates the need to maintain local datasets in the database.
- Stateful Stream Processing only caches the data, with shared logs remaining the source of truth, so the issue of data corruption over time is less critical.
- Essentially, services are driven by data, meaning that despite the constant growth of data volume, services can still respond swiftly to business events.
- Scalability issues fall on the broker, not on the services. Thus, the complexity of writing services is significantly reduced as there is no need to think about scalability.
- Adding new services does not require changing the old ones, thus making it easier to integrate new services.
As you can see, this is more than just REST. We have obtained a set of tools that allows us to work with shared data in a decentralized manner.
In today's article, not all aspects have been revealed. We still need to determine how to balance between the 'request-response' paradigm and the event-driven paradigm. But we'll tackle this next time. There are topics that we need to explore further, such as why Stateful Stream Processing is so advantageous. We'll discuss this in the third article. Additionally, there are other powerful constructs we can utilize if we choose to. It changes the game for distributed business systems, as this architecture provides transactional guarantees for in a scalable manner. This will be discussed in the fourth article. Finally, we will need to run through the details of implementing these principles.
But for now, just remember this: the data dichotomy is the force we face when creating business services. And we must keep that in mind. The focus is on flipping everything on its head and starting to treat shared data as first-class objects. Stateful Stream Processing offers a unique compromise for this. It avoids centralized 'God Components' that hinder the progress of development. Moreover, it ensures the speed, scalability, and fault tolerance of data streaming pipelines, embedding them in every service. This allows us to concentrate on a common stream of consciousness that any service can connect to and work with its data. As a result, services become more scalable, interchangeable, and autonomous. Therefore, they will not only look good on whiteboards and during hypothesis testing but also function and evolve for decades.
Source: habr.com
