Nearly all modern software products consist of several services. Often, long response times between service channels become a source of performance issues. A common solution to these kinds of problems is to bundle several inter-service requests into a single package, known as batching.
If you are using batching, you may not be satisfied with its performance or code clarity. This method is not as straightforward for the caller as one might think. Solutions can vary significantly for different purposes and in various situations. I will demonstrate the advantages and disadvantages of several approaches with specific examples.
Demonstration Project
For clarity, let's consider an example from one of the services in the application I am currently working on.
Explanation for Platform Choice in ExamplesThe problem of poor performance is quite common and does not pertain to any specific languages or platforms. In this article, examples in Spring + Kotlin will be used to demonstrate tasks and solutions. Kotlin is equally comprehensible (or incomprehensible) to Java and C# developers; moreover, the code is more compact and understandable than in Java. To ease understanding for pure Java developers, I will avoid Kotlin's dark magic and use only its white magic (in the spirit of Lombok). There will be a few extension methods, but they are actually familiar to all Java programmers as static methods, so this will be a little sweetness that won't spoil the dish.
There is a document approval service. Someone creates a document and brings it up for discussion, during which edits are made, and ultimately, the document gets approved. The approval service itself knows nothing about the documents; it is merely a chat for approvers with a few additional functions that we will not discuss here.
So, there are chat rooms (corresponding to documents) with a predetermined set of participants in each. As in regular chats, messages contain text and files and may be replies and forwards:
data class ChatMessage(
// nullable так как появляется только после persist
val id: Long? = null,
/** Ссылка на автора */
val author: UserReference,
/** Сообщение */
val message: String,
/** Ссылки на аттачи */
// из-за особенностей связки JPA+СУБД проще поддерживать и null, и пустые списки
val files: List<FileReference>? = null,
/** Если является ответом, то здесь будет оригинал */
val replyTo: ChatMessage? = null,
/** Если является пересылкой, то здесь будет оригинал */
val forwardFrom: ChatMessage? = null
)
Links to files and users are links to others domains. This is how it works for us:
typealias FileReference = Long
typealias UserReference = Long
User data is stored in Keycloak and accessed via REST. The same applies to files: files and their metadata reside in a separate file storage service.
All calls to these services are heavy requests.This means that the overhead for transporting these requests is much greater than the time taken to process them by the external service. In our test environments, the typical response time for such services is 100 ms, so we will use these figures going forward.
We need to create a simple REST controller to retrieve the last N messages with all the necessary information. This means we assume that the message model on the frontend is almost the same, and we need to forward all data. The difference for the frontend model is that the file and user need to be presented in a somewhat decrypted form to make them links:
/** В таком виде отдаются ссылки на сущности для фронта */
data class ReferenceUI(
/** Идентификатор для url */
val ref: String,
/** Видимое пользователю название ссылки */
val name: String
)
data class ChatMessageUI(
val id: Long,
/** Ссылка на автора */
val author: ReferenceUI,
/** Сообщение */
val message: String,
/** Ссылки на аттачи */
val files: List<ReferenceUI>,
/** Если являтся ответом, то здесь будет оригинал */
val replyTo: ChatMessageUI? = null,
/** Если являтся пересылкой, то здесь будет оригинал */
val forwardFrom: ChatMessageUI? = null
)
We need to implement the following:
interface ChatRestApi {
fun getLast(n: Int): List<ChatMessageUI>
}
Postfix UI means DTO models for the frontend, that is, what we should return via REST.
Here, it may seem surprising that we are not passing any chat identifier, and it is even absent in the ChatMessage/ChatMessageUI model. I did this deliberately to avoid cluttering the code examples (chats are isolated, so we can assume we only have one).
Philosophical digressionBoth in the ChatMessageUI class and in the ChatRestApi.getLast method, the data type List is used, while in reality it is an ordered Set. The JDK does not handle this well, so declaring the order of elements at the interface level (preserving order upon addition and extraction) is not feasible. Therefore, it has become common practice to use List in cases where an ordered Set is needed (there's also LinkedHashSet, but that's not an interface).
Important limitation: we will assume that long chains of responses or forwards do not exist. That is, they do exist, but their length does not exceed three messages. On the frontend, the chain of messages must be sent in full.
To retrieve data from external services, there are these APIs:
interface ChatMessageRepository {
fun findLast(n: Int): List<ChatMessage>
}
data class FileHeadRemote(
val id: FileReference,
val name: String
)
interface FileRemoteApi {
fun getHeadById(id: FileReference): FileHeadRemote
fun getHeadsByIds(id: Set<FileReference>): Set<FileHeadRemote>
fun getHeadsByIds(id: List<FileReference>): List<FileHeadRemote>
fun getHeadsByChat(): List<FileHeadRemote>
}
data class UserRemote(
val id: UserReference,
val name: String
)
interface UserRemoteApi {
fun getUserById(id: UserReference): UserRemote
fun getUsersByIds(id: Set<UserReference>): Set<UserRemote>
fun getUsersByIds(id: List<UserReference>): List<UserRemote>
}
It is evident that external services initially provide for batch processing, both through Set (without preserving the order of elements, with unique keys) and through List (duplicates may exist—the order is preserved).
Simple implementations
Naive implementation
The initial naive implementation of our REST controller will generally look something like this:
class ChatRestController(
private val messageRepository: ChatMessageRepository,
private val userRepository: UserRemoteApi,
private val fileRepository: FileRemoteApi
) : ChatRestApi {
override fun getLast(n: Int) =
messageRepository.findLast(n)
.map { it.toFrontModel() }
private fun ChatMessage.toFrontModel(): ChatMessageUI =
ChatMessageUI(
id = id ?: throw IllegalStateException("$this must be persisted"),
author = userRepository.getUserById(author).toFrontReference(),
message = message,
files = files?.let { files ->
fileRepository.getHeadsByIds(files)
.map { it.toFrontReference() }
} ?: listOf(),
forwardFrom = forwardFrom?.toFrontModel(),
replyTo = replyTo?.toFrontModel()
)
}
Everything is perfectly clear, which is a big plus.
We use batch processing and retrieve data from an external service in batches. But what does that mean for our performance?
For each message, there will be one call to UserRemoteApi to get data for the author field and one call to FileRemoteApi to get all attached files. Seems simple enough. Let's assume that the fields forwardFrom and replyTo for ChatMessage are obtained in such a way that additional calls are not required. However, transforming them into ChatMessageUI will lead to recursion, which means the call counter metrics may increase significantly. As we noted earlier, let's assume we don’t have a lot of nesting and the chain is limited to three messages.
As a result, we will have between two to six calls to external services per message and one JPA call for the entire message batch. The total number of calls will range from 2*N+1 to 6*N+1. What does this translate to in real terms? Suppose that 20 messages are needed to render the page. To obtain them, it will take between 4 to 10 seconds. Horrible! We’d like to keep it under 500 ms. And since the frontend aimed for seamless scrolling, the performance requirements for this endpoint can be doubled.
Pros:
- The code is concise and self-documenting (a developer's dream).
- The code is straightforward, so there are almost no chances of missteps.
- Batch processing does not seem alien and fits organically into the logic.
- Logic changes will be easy and local.
Cons:
Terrible performance due to the fact that the batches are very small.
This approach is often seen in simple services or prototypes. If speed of changes is important, complicating the system is likely unnecessary. At the same time, for our very simple service, performance turns out to be horrendous, so the applicability of such an approach is very limited.
Naive concurrent processing
You can run the processing of all messages concurrently — this will eliminate linear growth in time depending on the number of messages. This is not a particularly good way, as it will lead to a high peak load on the external service.
Implementing parallel processing is very simple:
override fun getLast(n: Int) =
messageRepository.findLast(n).parallelStream()
.map { it.toFrontModel() }
.collect(toList())
By using message parallel processing, we can ideally achieve 300–700 ms, which is significantly better than naive implementations, but still not fast enough.
With this approach, requests to userRepository and fileRepository will be executed synchronously, which is not very efficient. To fix this, we need to significantly change the call logic. For example, using CompletionStage (aka CompletableFuture):
private fun ChatMessage.toFrontModel(): ChatMessageUI =
CompletableFuture.supplyAsync {
userRepository.getUserById(author).toFrontReference()
}.thenCombine(
files?.let {
CompletableFuture.supplyAsync {
fileRepository.getHeadsByIds(files).map { it.toFrontReference() }
}
} ?: CompletableFuture.completedFuture(listOf())
) { author, files ->
ChatMessageUI(
id = id ?: throw IllegalStateException("$this must be persisted"),
author = author,
message = message,
files = files,
forwardFrom = forwardFrom?.toFrontModel(),
replyTo = replyTo?.toFrontModel()
)
}.get()!!
It is evident that the initially straightforward mapping code has become less clear. This is due to the need to separate external service calls from the places where the results are used. This in itself isn't bad. However, the combination of calls looks rather inelegant and resembles typical reactive 'spaghetti'.
If we use coroutines, everything will look much better:
private fun ChatMessage.toFrontModel(): ChatMessageUI =
join(
{ userRepository.getUserById(author).toFrontReference() },
{ files?.let { fileRepository.getHeadsByIds(files)
.map { it.toFrontReference() } } ?: listOf() }
).let { (author, files) ->
ChatMessageUI(
id = id ?: throw IllegalStateException("$this must be persisted"),
author = author,
message = message,
files = files,
forwardFrom = forwardFrom?.toFrontModel(),
replyTo = replyTo?.toFrontModel()
)
}
Where:
fun <A, B> join(a: () -> A, b: () -> B) =
runBlocking(IO) {
awaitAll(async { a() }, async { b() })
}.let {
it[0] as A to it[1] as B
}
Theoretically, using such parallel processing, we can achieve 200–400 ms, which is already close to our expectations.
Unfortunately, such efficient parallelism is rare, and the trade-off is quite harsh: with only a few users simultaneously working on the services, a flood of requests will crash through, which still won't be processed in parallel, so we'll return to our unfortunate 4 seconds.
My result when using such a service is 1300–1700 ms for processing 20 messages. This is faster than in the first implementation, yet it still does not solve the problem.
Alternative application of parallel requestsWhat if the third-party services do not provide batch processing? For example, we can hide the lack of batch processing implementation within the interface methods:
interface UserRemoteApi {
fun getUserById(id: UserReference): UserRemote
fun getUsersByIds(id: Set<UserReference>): Set<UserRemote> =
id.parallelStream()
.map { getUserById(it) }.collect(toSet())
fun getUsersByIds(id: List<UserReference>): List<UserRemote> =
id.parallelStream()
.map { getUserById(it) }.collect(toList())
}
This makes sense if there is hope for the emergence of batch processing in future versions.
Pros:
- Easy implementation of message parallel processing.
- Good scalability.
Cons:
- The need to separate data retrieval from their processing when making parallel requests to different services.
- Increased load on third-party services.
The applicability of the framework is similar to the naive approach. Using parallel request methods makes sense if you want to significantly boost the performance of your service at the expense of heavy reliance on external resources. In our example, performance increased by 2.5 times, but that is clearly not enough.
Caching
You can implement caching in the spirit of JPA for external services, which means storing obtained objects within a session to avoid fetching them again (including during batch processing). You can create such caches yourself, or use Spring with its @Cacheable, plus there's always the option to manually use a ready-made cache like EhCache.
The general problem is that caches are only beneficial if there are hits. In our case, hits are likely on the author field (let's say, 50%), while there won't be any hits on files at all. This approach will offer some improvements, but won't dramatically change performance (and we need a breakthrough).
Inter-session (long) caches require complex invalidation logic. In general, the later you wait to address performance issues using inter-session caches, the better.
Pros:
- Implementing caching without changing code.
- Performance gains by several times (in some cases).
Cons:
- Potential decrease in performance if misused.
- High memory overhead, especially with long caches.
- Complicated invalidation, where errors can lead to hard-to-reproduce issues at runtime.
Caches are often used merely to quickly patch design issues. This doesn't mean they shouldn't be used, but one should always approach them with caution and first assess the performance gain achieved, and only then make a decision.
In our example, the performance gain from caches will be about 25%. However, the downsides of caching are quite numerous, so I wouldn’t recommend using them here.
Summary
So, we've looked at the naive implementation of a service using batch processing and a few simple ways to speed it up.
The main advantage of all these methods is simplicity, which has many pleasant consequences.
A common issue with these methods is poor performance, primarily related to packet size. Therefore, if these solutions are not suitable for you, it is worth considering more radical methods.
There are two main directions to look for solutions:
- asynchronous data processing (requires a paradigm shift, so it is not discussed in this article);
- batching while maintaining synchronous processing.
Batching will significantly reduce the number of external calls while keeping the code synchronous. This topic will be addressed in the next part of the article.
Source: habr.com
