🥇BPM-style integration

Hello, Habr!

Our company specializes in developing ERP-class software solutions, with a significant portion comprising transactional systems that handle extensive business logic and document flow, similar to EDMS. Modern versions of our products are based on JavaEE technologies, but we are also actively experimenting with microservices. One of the most challenging aspects of such solutions is integrating various subsystems related to adjacent domains. Integration tasks have always caused us considerable headaches, regardless of the architectural styles, technology stacks, and frameworks we employ; however, recently there has been progress in addressing these challenges.

In the article presented to you, I will discuss the experiences and architectural explorations at Krista NPO in this field. We will also examine an example of a simple solution to an integration task from the perspective of an application developer and uncover what lies behind this simplicity.

Disclaimer

The architectural and technical solutions described in the article are based on my personal experience in the context of specific tasks. These solutions do not claim universality and may be suboptimal under different usage conditions.

What does BPM have to do with this?

To answer this question, we need to delve a little into the specifics of the application tasks of our solutions. The main portion of business logic in our typical transactional system involves entering data into a database through user interfaces, manual and automated data verification, processing them through a certain workflow, publishing them to another system / analytical database / archive, and generating reports. Thus, the key function of the system for clients is automating their internal business processes.

For convenience, we use the term "document" in our discussions as an abstraction of a set of data unified by a common key, to which a specific workflow can be "tied".
But how do we address the integration logic? After all, the integration task arises from the system architecture, which is "split" into parts NOT at the client's request, but under the influence of entirely different factors:

under the influence of Conway's law;
as a result of reusing subsystems previously developed for other products;
based on the architect's decision, considering non-functional requirements.

There is a strong temptation to separate the integration logic from the business logic of the main workflow in order to avoid cluttering the business logic with integration artifacts and to relieve the application developer from needing to delve into the architectural landscape of the system. While this approach has its advantages, practice shows its inefficiency:

the solution to integration tasks typically devolves into the simplest options in the form of synchronous calls due to limited extension points in the implementation of the main workflow (more about the shortcomings of synchronous integration below);
integration artifacts still infiltrate the core business logic when feedback from another subsystem is required;
the application developer ignores the integration and can easily break it by modifying the workflow;
the system stops being a cohesive whole from the user's perspective, with noticeable "joints" between subsystems and redundant user operations that initiate data transfer from one subsystem to another.

Another approach is to consider integration interactions as an essential part of the core business logic and workflow. To keep the qualification requirements for application developers from skyrocketing, the creation of new integration interactions must be done smoothly and effortlessly, with minimal options for choosing a solution. This is more challenging than it seems: the tool must be powerful enough to provide the user with a necessary variety of application options while not allowing them to 'shoot themselves in the foot'. There are many questions that an engineer must address in the context of integration tasks, but that the application developer should not have to think about in their daily work: transaction boundaries, consistency, atomicity, security, scalability, load and resource distribution, routing, marshalling, context propagation and switching, etc. It is essential to offer application developers simple solution templates that already embed answers to all such questions. These templates must be sufficiently safe: business logic changes very often, increasing the risks of introducing errors, and the cost of mistakes should remain at a relatively low level.

But what does BPM have to do with it? There are many ways to implement workflow...
Indeed, in our solutions, another implementation of business processes is very popular – through the declarative specification of state transition diagrams and attaching handlers with business logic to transitions. Here, the state that defines the current position of the 'document' in the business process is an attribute of the 'document' itself.

This is what the process looks like at the start of the project.

The popularity of this implementation is due to the relative simplicity and speed of creating linear business processes. However, as software systems become increasingly complex, the automated part of the business process expands and complicates. There arises a need for decomposition, reusing parts of processes, and branching processes so that each branch is executed in parallel. In such conditions, the tool becomes inconvenient, and the state transition diagram loses its informativeness (integration interactions are not reflected at all in the diagram).

This is what the process looks like after several iterations of requirement refinement.

The solution to this situation was the integration of the engine jBPM into some products with the most complex business processes. In the short term, this solution had some success: it became possible to implement complex business processes while maintaining a sufficiently informative and up-to-date diagram in the notation BPMN2.

A small part of a complex business process.

In the long term, the solution did not meet expectations: the high labor intensity of creating business processes through visual tools did not allow for achieving acceptable productivity rates, and the tool itself became one of the least loved among developers. There were also complaints about the internal structure of the engine, which led to the appearance of numerous patches and workarounds.

The main positive aspect of using jBPM was the recognition of the benefits and drawbacks of having a persistent state for an instance of a business process. We also saw the opportunity to apply a process-oriented approach to implement complex integration protocols between various applications using asynchronous interactions through signals and messages. The presence of a persistent state plays a crucial role in this.

Based on the above, we can conclude that: the process-oriented approach in the style of BPM allows us to address a wide range of tasks for automating increasingly complex business processes, harmoniously integrating these processes with integration activities, and retaining the possibility of visually representing the implemented process in an appropriate notation.

Disadvantages of Synchronous Calls as an Integration Pattern

Synchronous integration refers to the simplest blocking call. One subsystem acts as the server side and exposes an API with the necessary method. Another subsystem acts as the client side and at the right moment makes a call while waiting for the result. Depending on the system architecture, the client and server sides can be located either in the same application and process or in different ones. In the latter case, some implementation of RPC is required and the marshaling of parameters and the call result must be ensured.

This integration pattern has a significant number of drawbacks, but it is widely used in practice due to its simplicity. The speed of implementation is appealing and leads to its repeated use under tight deadlines, recording the solution as technical debt. However, inexperienced developers sometimes use it unwittingly, simply unaware of the negative consequences.

In addition to the most obvious increase in subsystem coupling, there are less evident problems with 'spreading' and 'stretching' transactions. Indeed, if business logic introduces changes, transactions become unavoidable, and the transactions themselves block certain resources of the application affected by these changes. This means that while one subsystem is waiting for a response from another, it cannot complete the transaction and release the locks. This significantly increases the risk of various effects:

the system's responsiveness is lost, users wait a long time for responses to their requests;
the server completely stops responding to user requests due to an overloaded thread pool: most threads are 'stuck' on a resource lock occupied by the transaction;
deadlocks begin to appear: the likelihood of their occurrence strongly depends on the duration of transactions, the amount of business logic involved in the transaction, and the locks;
transaction timeout errors occur;
the server 'crashes' due to OutOfMemory if the task requires processing and modifying large volumes of data, and the presence of synchronous integrations makes it very difficult to break the processing into more 'lightweight' transactions.

From an architectural perspective, the use of blocking calls during integration leads to a loss of control over the quality of individual subsystems: it is impossible to ensure the target quality metrics of one subsystem independently from those of another subsystem. If the subsystems are developed by different teams, this poses a significant challenge.

Things get even more interesting when the integrated subsystems are in different applications and synchronous changes need to be made from both sides. How can we ensure the transactionality of these changes?

If changes are made with separate transactions, then reliable exception handling and compensation will need to be ensured, which completely undermines the main advantage of synchronous integrations – simplicity.

Distributed transactions come to mind as a solution, but we do not use them in our solutions: ensuring reliability is challenging.

The Saga as a solution to transaction problems

With the growing popularity of microservices, the Saga Pattern.

This pattern effectively addresses the long transaction issues mentioned above and expands the capabilities for managing system state from the business logic side: compensation after a failed transaction can not revert the system to its original state, but can provide an alternative route for data processing. It also allows for successful steps in data processing to be omitted during repeated attempts to achieve a 'good' ending.

Interestingly, in monolithic systems, this pattern is also relevant when it comes to integrating loosely coupled subsystems, where negative effects arise from long transactions and the resulting resource locks.

Regarding our BPM style business processes, implementing Sagas is very straightforward: individual steps of a Saga can be defined as activities within the business process, and the persistent state of the business process determines, among other things, the internal state of the Saga. Thus, no additional coordinating mechanism is required. Only a message broker with 'at least once' delivery guarantees will be needed as the transport.

But even this solution has its 'cost':

Business logic becomes more complex: compensation processing is required;
It will be necessary to give up full consistency, which can be particularly sensitive for monolithic systems;
The architecture becomes slightly more complicated, creating an additional need for a message broker;
Additional monitoring and administration tools will be required (although overall this is even good: the quality of system service will improve).

For monolithic systems, the justification for using a Saga is not so evident. For microservices and other SOA, where there is likely already a broker and full consistency was sacrificed at the project's inception, the benefits of using this pattern may significantly outweigh the drawbacks, especially if there is a convenient API at the business logic level.

Encapsulation of business logic in microservices

When we started experimenting with microservices, a reasonable question arose: where to place the domain business logic relative to the service that ensures the persistence of domain data?

Looking at the architecture of various BPMS, it may seem reasonable to separate business logic from persistence: create a layer of platform-independent microservices that form the environment and container for executing domain business logic, while configuring the persistence of domain data as a separate layer of very simple and lightweight microservices. Business processes in this case orchestrate the services of the persistence layer.

This approach has a significant advantage: one can continuously enhance the functionality of the platform, and only the corresponding layer of platform microservices will become 'thicker'. Business processes from any domain immediately gain the ability to use new platform functionality as soon as it is updated.

A more detailed analysis revealed significant drawbacks of this approach:

The platform service executing the business logic of many domains carries significant risks as a single point of failure. Frequent changes in business logic increase the risk of errors leading to failures that propagate across the entire system;
Performance issues: business logic operates with its data through a narrow and slow interface:
- Data will be marshaled and processed through the network stack multiple times;
- Domain services often return more data than the business logic requires for processing due to insufficient request parameterization capabilities at the external API service level;
- Multiple independent parts of the business logic may re-request the same data for processing (this issue can be mitigated by adding session components that cache data, but this complicates the architecture and creates challenges with data relevance and cache invalidation);
Transactionality issues:
- Business processes with persistent state, managed by the platform service, may become inconsistent with domain data, with no straightforward solutions in sight;
- Isolating domain data locking outside of transactions: if the domain business logic needs to make changes while first validating the correctness of current data, it is necessary to prevent any concurrent modifications of the data being processed. External data locking can help address this issue, but such a solution carries additional risks and decreases overall system reliability;
Additional complexities during updates: in some cases, it is necessary to update the persistence service and business logic synchronously or in a strict sequence.

Ultimately, we had to return to the basics: encapsulating domain data and domain business logic into a single microservice. This approach simplifies the perception of the microservice as a cohesive component within the system and avoids the aforementioned issues. However, this is not without cost:

Standardization of the API for interaction with business logic is required (particularly to support user activities within business processes) and API platform services; there needs to be a more careful approach to API changes, ensuring both forward and backward compatibility;
Additional runtime libraries need to be added to ensure the functioning of business logic within each such microservice, which creates new requirements for these libraries: lightweight and minimal transitive dependencies;
Developers of business logic need to keep track of library versions: if a microservice hasn't been updated for a while, it's likely to have an outdated library version. This can become an unexpected obstacle when adding new features and may require migrating the old business logic of that service to new library versions if there were incompatible changes between versions.

The platform services layer in such an architecture is also present, but this layer does not form a container for executing domain business logic; it merely provides an environment for it, offering auxiliary 'platform' functions. This layer is necessary not only to maintain the lightweight nature of domain microservices but also to centralize management.

For example, user activities in business processes generate tasks. However, when dealing with tasks, a user should see tasks from all domains in a single list, meaning there should be a corresponding platform service for task registration, stripped of domain business logic. Maintaining encapsulation of business logic in such a context is quite challenging, and this is another compromise of this architecture.

Integrating business processes from the perspective of an application developer

As previously mentioned, an application developer should be abstracted from the technical and engineering aspects of implementing interaction between multiple applications to ensure good development productivity.

Let's attempt to solve a rather complex integration task, specifically designed for this article. It will be a 'game' task involving three applications, each defining a certain domain name: 'app1', 'app2', 'app3'.

Inside each application, business processes are initiated that start to 'play ball' through an integration bus. The ball will be represented by messages named 'Ball'.

Game rules:

the first player is the initiator. They invite other players to the game, start the game, and can end it at any moment;
other players declare their participation in the game, 'get to know' each other and the first player;
upon receiving the ball, a player selects another participating player and passes the ball to them. The total number of passes is counted.
Each player has "energy" that decreases with each pass made by that player. When their energy runs out, the player exits the game, announcing their departure;
if a player is left alone, they immediately announce their exit;
when all players have exited, the first player announces the end of the game. If they exited the game earlier, they continue to monitor the game to conclude it.

To solve this task, I will use our DSL for business processes, which allows for compact logic description in Kotlin with minimal boilerplate.

In the app1 application, the business process of the first player (the game initiator) will run:

class InitialPlayer

import ru.krista.bpm.ProcessInstance
import ru.krista.bpm.runtime.ProcessImpl
import ru.krista.bpm.runtime.constraint.UniqueConstraints
import ru.krista.bpm.runtime.dsl.processModel
import ru.krista.bpm.runtime.dsl.taskOperation
import ru.krista.bpm.runtime.instance.MessageSendInstance

data class PlayerInfo(val name: String, val domain: String, val id: String)

class PlayersList : ArrayList()

// This is the process instance class: encapsulates its internal state
class InitialPlayer : ProcessImpl(initialPlayerModel) {
    var playerName: String by persistent("Player1")
    var energy: Int by persistent(30)
    var players: PlayersList by persistent(PlayersList())
    var shotCounter: Int = 0
}

// This is the process model declaration: created once, used by all
// instances of the corresponding class
val initialPlayerModel = processModel(name = "InitialPlayer",
                                                     version = 1) {

    // According to the rules, the first player is the game initiator and should be the only one
    uniqueConstraint = UniqueConstraints.singleton

    // Declare activities that make up the business process
    val sendNewGameSignal = signal("NewGame")
    val sendStopGameSignal = signal("StopGame")
    val startTask = humanTask("Start") {
        taskOperation {
            processCondition { players.size > 0 }
            confirmation { "${players.size} players connected. Shall we begin?" }
        }
    }
    val stopTask = humanTask("Stop") {
        taskOperation {}
    }
    val waitPlayerJoin = signalWait("PlayerJoin") { signal ->
        players.add(PlayerInfo(
                signal.data!!,
                signal.sender.domain,
                signal.sender.processInstanceId))
        println("... player ${signal.data} joined ...")
    }
    val waitPlayerOut = signalWait("PlayerOut") { signal ->
        players.remove(PlayerInfo(
                signal.data!!,
                signal.sender.domain,
                signal.sender.processInstanceId))
        println("... player ${signal.data} is out ...")
    }
    val sendPlayerOut = signal("PlayerOut") {
        signalData = { playerName }
    }
    val sendHandshake = messageSend("Handshake") {
        messageData = { playerName }
        activation = {
            receiverDomain = process.players.last().domain
            receiverProcessInstanceId = process.players.last().id
        }
    }
    val throwStartBall = messageSend("Ball") {
        messageData = { 1 }
        activation = { selectNextPlayer() }
    }
    val throwBall = messageSend("Ball") {
        messageData = { shotCounter + 1 }
        activation = { selectNextPlayer() }
        onEntry { energy -= 1 }
    }
    val waitBall = messageWaitData("Ball") {
        shotCounter = it
    }

    // Now construct the process graph from the declared activities
    startFrom(sendNewGameSignal)
            .fork("mainFork") {
                next(startTask)
                next(waitPlayerJoin).next(sendHandshake).next(waitPlayerJoin)
                next(waitPlayerOut)
                        .branch("checkPlayers") {
                            ifTrue { players.isEmpty() }
                                    .next(sendStopGameSignal)
                                    .terminate()
                            ifElse().next(waitPlayerOut)
                        }
            }
    startTask.fork("afterStart") {
        next(throwStartBall)
                .branch("mainLoop") {
                    ifTrue { energy < 5 }.next(sendPlayerOut).next(waitBall)
                    ifElse().next(waitBall).next(throwBall).loop()
                }
        next(stopTask).next(sendStopGameSignal)
    }

    // Attach additional handlers to activities for logging
    sendNewGameSignal.onExit { println("Let's play!") }
    sendStopGameSignal.onExit { println("Stop!") }
    sendPlayerOut.onExit { println("$playerName: I'm out!") }
}

private fun MessageSendInstance.selectNextPlayer() {
    val player = process.players.random()
    receiverDomain = player.domain
    receiverProcessInstanceId = player.id
    println("Step ${process.shotCounter + 1}: " +
            "${process.playerName} >>> ${player.name}")
}

In addition to executing business logic, the provided code can generate an object model of the business process, which can be visualized as a diagram. We have not yet implemented the visualizer, so we had to spend some time drawing it (here I slightly simplified the BPMN notation regarding the use of gates to improve the diagram's consistency with the provided code):

The app2 application will include the business process of another player:

class RandomPlayer

import ru.krista.bpm.ProcessInstance
import ru.krista.bpm.runtime.ProcessImpl
import ru.krista.bpm.runtime.dsl.processModel
import ru.krista.bpm.runtime.instance.MessageSendInstance

data class PlayerInfo(val name: String, val domain: String, val id: String)

class PlayersList: ArrayList()

class RandomPlayer : ProcessImpl(randomPlayerModel) {

    var playerName: String by input(persistent = true, 
                                    defaultValue = "RandomPlayer")
    var energy: Int by input(persistent = true, defaultValue = 30)
    var players: PlayersList by persistent(PlayersList())
    var allPlayersOut: Boolean by persistent(false)
    var shotCounter: Int = 0

    val selfPlayer: PlayerInfo
        get() = PlayerInfo(playerName, env.eventDispatcher.domainName, id)
}

val randomPlayerModel = processModel(name = "RandomPlayer", 
                                                   version = 1) {

    val waitNewGameSignal = signalWait("NewGame")
    val waitStopGameSignal = signalWait("StopGame")
    val sendPlayerJoin = signal("PlayerJoin") {
        signalData = { playerName }
    }
    val sendPlayerOut = signal("PlayerOut") {
        signalData = { playerName }
    }
    val waitPlayerJoin = signalWaitCustom("PlayerJoin") {
        eventCondition = { signal ->
            signal.sender.processInstanceId != process.id 
                && !process.players.any { signal.sender.processInstanceId == it.id}
        }
        handler = { signal ->
            players.add(PlayerInfo(
                    signal.data!!,
                    signal.sender.domain,
                    signal.sender.processInstanceId))
        }
    }
    val waitPlayerOut = signalWait("PlayerOut") { signal ->
        players.remove(PlayerInfo(
                signal.data!!,
                signal.sender.domain,
                signal.sender.processInstanceId))
        allPlayersOut = players.isEmpty()
    }
    val sendHandshake = messageSend("Handshake") {
        messageData = { playerName }
        activation = {
            receiverDomain = process.players.last().domain
            receiverProcessInstanceId = process.players.last().id
        }
    }
    val receiveHandshake = messageWait("Handshake") { message ->
        if (!players.any { message.sender.processInstanceId == it.id}) {
            players.add(PlayerInfo(
                    message.data!!, 
                    message.sender.domain, 
                    message.sender.processInstanceId))
        }
    }
    val throwBall = messageSend("Ball") {
        messageData = { shotCounter + 1 }
        activation = { selectNextPlayer() }
        onEntry { energy -= 1 }
    }
    val waitBall = messageWaitData("Ball") {
        shotCounter = it
    }

    startFrom(waitNewGameSignal)
            .fork("mainFork") {
                next(sendPlayerJoin)
                        .branch("mainLoop") {
                            ifTrue { energy < 5 || allPlayersOut }
                                    .next(sendPlayerOut)
                                    .next(waitBall)
                            ifElse()
                                    .next(waitBall)
                                    .next(throwBall)
                                    .loop()
                        }
                next(waitPlayerJoin).next(sendHandshake).next(waitPlayerJoin)
                next(waitPlayerOut).next(waitPlayerOut)
                next(receiveHandshake).next(receiveHandshake)
                next(waitStopGameSignal).terminate()
            }

    sendPlayerJoin.onExit { println("$playerName: I'm here!") }
    sendPlayerOut.onExit { println("$playerName: I'm out!") }
}

private fun MessageSendInstance.selectNextPlayer() {
    val player = if (process.players.isNotEmpty()) 
        process.players.random() 
    else 
        process.selfPlayer
    receiverDomain = player.domain
    receiverProcessInstanceId = player.id
    println("Step ${process.shotCounter + 1}: " +
            "${process.playerName} >>> ${player.name}")
}

Diagram:

In the app3 application, we will make the player behave a bit differently: instead of randomly selecting the next player, it will act according to the round-robin algorithm:

class RoundRobinPlayer

import ru.krista.bpm.ProcessInstance
import ru.krista.bpm.runtime.ProcessImpl
import ru.krista.bpm.runtime.dsl.processModel
import ru.krista.bpm.runtime.instance.MessageSendInstance

data class PlayerInfo(val name: String, val domain: String, val id: String)

class PlayersList: ArrayList()

class RoundRobinPlayer : ProcessImpl(roundRobinPlayerModel) {

    var playerName: String by input(persistent = true, 
                                    defaultValue = "RoundRobinPlayer")
    var energy: Int by input(persistent = true, defaultValue = 30)
    var players: PlayersList by persistent(PlayersList())
    var nextPlayerIndex: Int by persistent(-1)
    var allPlayersOut: Boolean by persistent(false)
    var shotCounter: Int = 0

    val selfPlayer: PlayerInfo
        get() = PlayerInfo(playerName, env.eventDispatcher.domainName, id)
}

val roundRobinPlayerModel = processModel(
        name = "RoundRobinPlayer", 
        version = 1) {

    val waitNewGameSignal = signalWait("NewGame")
    val waitStopGameSignal = signalWait("StopGame")
    val sendPlayerJoin = signal("PlayerJoin") {
        signalData = { playerName }
    }
    val sendPlayerOut = signal("PlayerOut") {
        signalData = { playerName }
    }
    val waitPlayerJoin = signalWaitCustom("PlayerJoin") {
        eventCondition = { signal ->
            signal.sender.processInstanceId != process.id 
                && !process.players.any { signal.sender.processInstanceId == it.id}
        }
        handler = { signal ->
            players.add(PlayerInfo(
                    signal.data!!, 
                    signal.sender.domain, 
                    signal.sender.processInstanceId))
        }
    }
    val waitPlayerOut = signalWait("PlayerOut") { signal ->
        players.remove(PlayerInfo(
                signal.data!!, 
                signal.sender.domain, 
                signal.sender.processInstanceId))
        allPlayersOut = players.isEmpty()
    }
    val sendHandshake = messageSend("Handshake") {
        messageData = { playerName }
        activation = {
            receiverDomain = process.players.last().domain
            receiverProcessInstanceId = process.players.last().id
        }
    }
    val receiveHandshake = messageWait("Handshake") { message ->
        if (!players.any { message.sender.processInstanceId == it.id}) {
            players.add(PlayerInfo(
                    message.data!!, 
                    message.sender.domain, 
                    message.sender.processInstanceId))
        }
    }
    val throwBall = messageSend("Ball") {
        messageData = { shotCounter + 1 }
        activation = { selectNextPlayer() }
        onEntry { energy -= 1 }
    }
    val waitBall = messageWaitData("Ball") {
        shotCounter = it
    }

    startFrom(waitNewGameSignal)
            .fork("mainFork") {
                next(sendPlayerJoin)
                        .branch("mainLoop") {
                            ifTrue { energy < 5 || allPlayersOut }
                                    .next(sendPlayerOut)
                                    .next(waitBall)
                            ifElse()
                                    .next(waitBall)
                                    .next(throwBall)
                                    .loop()
                        }
                next(waitPlayerJoin).next(sendHandshake).next(waitPlayerJoin)
                next(waitPlayerOut).next(waitPlayerOut)
                next(receiveHandshake).next(receiveHandshake)
                next(waitStopGameSignal).terminate()
            }

    sendPlayerJoin.onExit { println("$playerName: I'm here!") }
    sendPlayerOut.onExit { println("$playerName: I'm out!") }
}

private fun MessageSendInstance.selectNextPlayer() {
    var idx = process.nextPlayerIndex + 1
    if (idx >= process.players.size) {
        idx = 0
    }
    process.nextPlayerIndex = idx
    val player = if (process.players.isNotEmpty()) 
        process.players[idx] 
    else 
        process.selfPlayer
    receiverDomain = player.domain
    receiverProcessInstanceId = player.id
    println("Step ${process.shotCounter + 1}: " +
            "${process.playerName} >>> ${player.name}")
}

Otherwise, the player's behavior remains unchanged from the previous version, so the diagram does not change.

Now we need a test to run all of this. I will only provide the code for the test itself to avoid cluttering the article with boilerplate (actually, I used a test environment created earlier for testing the integration of other business processes):

testGame()

@Test
public void testGame() throws InterruptedException {
    String pl2 = startProcess(app2, "RandomPlayer", playerParams("Player2", 20));
    String pl3 = startProcess(app2, "RandomPlayer", playerParams("Player3", 40));
    String pl4 = startProcess(app3, "RoundRobinPlayer", playerParams("Player4", 25));
    String pl5 = startProcess(app3, "RoundRobinPlayer", playerParams("Player5", 35));
    String pl1 = startProcess(app1, "InitialPlayer");
    // Now we need to wait a bit for the players to "get to know" each other.
    // Waiting with sleep is a bad solution, but it's the simplest one.
    // Don't do this in serious tests!
    Thread.sleep(1000);
    // Starting the game, closing user activity
    assertTrue(closeTask(app1, pl1, "Start"));
    app1.getWaiting().waitProcessFinished(pl1);
    app2.getWaiting().waitProcessFinished(pl2);
    app2.getWaiting().waitProcessFinished(pl3);
    app3.getWaiting().waitProcessFinished(pl4);
    app3.getWaiting().waitProcessFinished(pl5);
}

private Map playerParams(String name, int energy) {
    Map params = new HashMap();
    params.put("playerName", name);
    params.put("energy", energy);
    return params;
}

Running the test, watching the log:

console output

Lock acquired for key lock://app1/process/InitialPlayer
Let's play!
Lock released for key lock://app1/process/InitialPlayer
Player2: I'm here!
Player3: I'm here!
Player4: I'm here!
Player5: I'm here!
... join player Player2 ...
... join player Player4 ...
... join player Player3 ...
... join player Player5 ...
Step 1: Player1 >>> Player3
Step 2: Player3 >>> Player5
Step 3: Player5 >>> Player3
Step 4: Player3 >>> Player4
Step 5: Player4 >>> Player3
Step 6: Player3 >>> Player4
Step 7: Player4 >>> Player5
Step 8: Player5 >>> Player2
Step 9: Player2 >>> Player5
Step 10: Player5 >>> Player4
Step 11: Player4 >>> Player2
Step 12: Player2 >>> Player4
Step 13: Player4 >>> Player1
Step 14: Player1 >>> Player4
Step 15: Player4 >>> Player3
Step 16: Player3 >>> Player1
Step 17: Player1 >>> Player2
Step 18: Player2 >>> Player3
Step 19: Player3 >>> Player1
Step 20: Player1 >>> Player5
Step 21: Player5 >>> Player1
Step 22: Player1 >>> Player2
Step 23: Player2 >>> Player4
Step 24: Player4 >>> Player5
Step 25: Player5 >>> Player3
Step 26: Player3 >>> Player4
Step 27: Player4 >>> Player2
Step 28: Player2 >>> Player5
Step 29: Player5 >>> Player2
Step 30: Player2 >>> Player1
Step 31: Player1 >>> Player3
Step 32: Player3 >>> Player4
Step 33: Player4 >>> Player1
Step 34: Player1 >>> Player3
Step 35: Player3 >>> Player4
Step 36: Player4 >>> Player3
Step 37: Player3 >>> Player2
Step 38: Player2 >>> Player5
Step 39: Player5 >>> Player4
Step 40: Player4 >>> Player5
Step 41: Player5 >>> Player1
Step 42: Player1 >>> Player5
Step 43: Player5 >>> Player3
Step 44: Player3 >>> Player5
Step 45: Player5 >>> Player2
Step 46: Player2 >>> Player3
Step 47: Player3 >>> Player2
Step 48: Player2 >>> Player5
Step 49: Player5 >>> Player4
Step 50: Player4 >>> Player2
Step 51: Player2 >>> Player5
Step 52: Player5 >>> Player1
Step 53: Player1 >>> Player5
Step 54: Player5 >>> Player3
Step 55: Player3 >>> Player5
Step 56: Player5 >>> Player2
Step 57: Player2 >>> Player1
Step 58: Player1 >>> Player4
Step 59: Player4 >>> Player1
Step 60: Player1 >>> Player4
Step 61: Player4 >>> Player3
Step 62: Player3 >>> Player2
Step 63: Player2 >>> Player5
Step 64: Player5 >>> Player4
Step 65: Player4 >>> Player5
Step 66: Player5 >>> Player1
Step 67: Player1 >>> Player5
Step 68: Player5 >>> Player3
Step 69: Player3 >>> Player4
Step 70: Player4 >>> Player2
Step 71: Player2 >>> Player5
Step 72: Player5 >>> Player2
Step 73: Player2 >>> Player1
Step 74: Player1 >>> Player4
Step 75: Player4 >>> Player1
Step 76: Player1 >>> Player2
Step 77: Player2 >>> Player5
Step 78: Player5 >>> Player4
Step 79: Player4 >>> Player3
Step 80: Player3 >>> Player1
Step 81: Player1 >>> Player5
Step 82: Player5 >>> Player1
Step 83: Player1 >>> Player4
Step 84: Player4 >>> Player5
Step 85: Player5 >>> Player3
Step 86: Player3 >>> Player5
Step 87: Player5 >>> Player2
Step 88: Player2 >>> Player3
Player2: I'm out!
Step 89: Player3 >>> Player4
... player Player2 is out ...
Step 90: Player4 >>> Player1
Step 91: Player1 >>> Player3
Step 92: Player3 >>> Player1
Step 93: Player1 >>> Player4
Step 94: Player4 >>> Player3
Step 95: Player3 >>> Player5
Step 96: Player5 >>> Player1
Step 97: Player1 >>> Player5
Step 98: Player5 >>> Player3
Step 99: Player3 >>> Player5
Step 100: Player5 >>> Player4
Step 101: Player4 >>> Player5
Player4: I'm out!
... player Player4 is out ...
Step 102: Player5 >>> Player1
Step 103: Player1 >>> Player3
Step 104: Player3 >>> Player1
Step 105: Player1 >>> Player3
Step 106: Player3 >>> Player5
Step 107: Player5 >>> Player3
Step 108: Player3 >>> Player1
Step 109: Player1 >>> Player3
Step 110: Player3 >>> Player5
Step 111: Player5 >>> Player1
Step 112: Player1 >>> Player3
Step 113: Player3 >>> Player5
Step 114: Player5 >>> Player3
Step 115: Player3 >>> Player1
Step 116: Player1 >>> Player3
Step 117: Player3 >>> Player5
Step 118: Player5 >>> Player1
Step 119: Player1 >>> Player3
Step 120: Player3 >>> Player5
Step 121: Player5 >>> Player3
Player5: I'm out!
... player Player5 is out ...
Step 122: Player3 >>> Player5
Step 123: Player5 >>> Player1
Player5: I'm out!
Step 124: Player1 >>> Player3
... player Player5 is out ...
Step 125: Player3 >>> Player1
Step 126: Player1 >>> Player3
Player1: I'm out!
... player Player1 is out ...
Step 127: Player3 >>> Player3
Player3: I'm out!
Step 128: Player3 >>> Player3
... player Player3 is out ...
Player3: I'm out!
Stop!
Step 129: Player3 >>> Player3
Player3: I'm out!

Several important conclusions can be drawn from all of this:

With the necessary tools, application developers can create integration interactions between applications without disrupting business logic;
The complexity of the integration task, which requires engineering expertise, can be hidden within the framework if it is initially built into the framework's architecture. However, the difficulty of the task cannot be concealed, so the solution to a difficult task in code will reflect that complexity;
When developing integration logic, it is essential to consider eventual consistency and the absence of linearizability in the state changes of all integration participants. This necessitates complicating the logic to make it insensitive to the order of external events. For instance, a player must participate in the game only after declaring their exit: other players will continue passing the ball to them until the information about their exit is communicated and processed by all participants. This logic does not derive from the rules of the game and represents a compromise within the chosen architecture.

Next, we will discuss various nuances of our solution, compromises, and other factors.

All messages are in one queue.

All integrated applications operate using a single integration bus, represented as an external broker, one BPMQueue for messages, and one BPMTopic for signals (events). Routing all messages through a single queue is, in itself, a compromise. At the business logic level, new message types can be introduced without making alterations to the system structure. This is a significant simplification, but it carries certain risks that, in the context of our typical tasks, we found not to be particularly significant.

However, there is one nuance: each application filters its own messages from the queue right at the entry point, based on its domain name. The domain can also be specified in signals if you need to limit the visibility of a signal to a single application. This should increase the throughput of the bus, but the business logic now must operate with domain names: mandatory for message addressing, preferable for signals.

Ensuring the reliability of the integration bus

Reliability consists of several factors:

the chosen message broker is a critically important component of the architecture and a single point of failure: it must be sufficiently fault-tolerant. Only time-tested implementations with good support and a large community should be used;
high availability of the message broker must be ensured, which means it should be physically separated from the integrated applications (ensuring high availability of applications with business logic is significantly more complex and expensive);
the broker must ensure "at least once" delivery guarantees. This is a mandatory requirement for the reliable operation of the integration bus. There is no need for "exactly once" guarantees: business processes are generally not sensitive to the repeated arrival of messages or events, and in special tasks where this is important, it is easier to add an additional check in the business logic than to constantly rely on sufficiently "expensive" guarantees;
the sending of messages and signals must be involved in a general transaction with the state change of business processes and domain data. The preferred option would be to use the pattern Transactional Outbox, but it will require an additional table in the database and a relay. In JEE applications, this can be simplified using a local JTA manager, but the connection to the chosen broker must be able to operate in " XA;
handlers of incoming messages and events must also work with the transaction of changing the business process state: if such a transaction is rolled back, then the reception of the message must be canceled;
messages that failed to deliver due to errors need to be stored in a separate repository DLQ (Dead Letter Queue). We have created a dedicated platform microservice to store such messages in its storage, index them by attributes (for quick grouping and searching), and provide an API for viewing, resending to the destination, and deleting messages. System administrators can interact with this service through its web interface;
in the broker settings, you need to adjust the number of retry attempts and delays between deliveries to reduce the likelihood of messages ending up in the DLQ (calculating optimal parameters is practically impossible, but you can act empirically and adjust them as you go along);
the DLQ storage should be continuously monitored, and the monitoring system should alert system administrators to react as quickly as possible when undelivered messages appear. This will help minimize the 'impact zone' of any failures or business logic errors;
the integration bus should be insensitive to the temporary absence of applications: subscriptions to topics should be durable, and the application domain name should be unique so that no one else attempts to process its messages from the queue during the application's absence.

Ensuring thread safety of business logic

The same instance of a business process may receive multiple messages and events simultaneously, which will be processed in parallel. At the same time, everything should be simple and thread-safe for the application developer.

The business logic of the process handles each external event affecting this business process separately. Such events may include:

starting an instance of the business process;
a user action related to activities within the business process;
receiving a message or signal that the instance of the business process is subscribed to;
the triggering of a timer set by the instance of the business process;
controlling action via API (e.g., emergency interruption of the process).

Each such event can change the state of a business process instance: some activities may end while others may begin, and the values of persistent properties may change. The completion of any activity can trigger one or several subsequent activities. These, in turn, may pause waiting for other events or, if they do not need any additional data, may finalize in the same transaction. Before closing the transaction, the new state of the business process is saved in the database, where it will await the occurrence of the next external event.

Persistent data of the business process, stored in a relational database, serves as a very convenient synchronization point for processing when using SELECT FOR UPDATE. If one transaction manages to retrieve the business process state from the database for modification, no other transaction can simultaneously obtain the same state for a different change; after the first transaction is completed, the second will surely receive the already modified state.

By utilizing pessimistic locks on the database side, we fulfill all necessary requirements ACID, while also preserving the ability to scale the application with business logic by increasing the number of active instances.

However, pessimistic locks pose the risk of deadlocks, thus SELECT FOR UPDATE should indeed be limited to a reasonable timeout to account for deadlocks in glaring cases within the business logic.

Another problem is the synchronization of the business process start. While there is no instance of the business process, there is no state in the database, hence the described method is not suitable. If unique instances of a business process need to be ensured within a specific scope, then a synchronization object associated with the process class and corresponding scope will be required. To solve this issue, we employ a different locking mechanism that allows us to lock an arbitrary resource specified by a key in URI format via an external service.

In our examples, the business process InitialPlayer contains a declaration

uniqueConstraint = UniqueConstraints.singleton

Therefore, the log contains messages about acquiring and releasing the lock for the corresponding key. Other business processes do not have such messages: the uniqueConstraint is not set.

Issues with business processes involving persistent state

Sometimes the presence of persistent state not only helps but also complicates development significantly.
Problems arise when changes need to be made to the business logic and/or the business process model. Not every change proves compatible with the old state of the business processes. If there are many "live" instances in the database, introducing incompatible changes can cause a lot of issues, which we have often encountered while using jBPM.

Depending on the depth of the changes, there are two approaches:

create a new type of business process to avoid making incompatible changes to the old one, and use it instead at the launch of new instances. The old instances will continue to operate "as before";
migrate the persistent state of the business processes when updating the business logic.

The first approach is simpler but has its limitations and drawbacks, such as:

duplication of business logic across many business process models, leading to increased complexity of the business logic;
often a seamless transition to new business logic is required (almost always in integration tasks);
the developer does not know when it is safe to remove outdated models.

In practice, we use both approaches, but we have made several decisions to simplify our work:

the persistent state of the business process in the database is stored in an easily readable and processable format: in a JSON string. This allows migrations to be carried out both within the application and externally. In extreme cases, manual edits can also be made (especially useful during debugging during development);
the integration business logic does not use the names of business processes so that at any moment one can replace the implementation of one of the involved processes with a new one, with a new name (for example, "InitialPlayerV2"). Binding occurs through message and signal names;
The process model has a version number that we increment when we make incompatible changes to this model, and this number is preserved along with the process instance's state;
The persistent state of the process is read from the database first into a convenient object model that the migration procedure can work with if the model version number has changed;
The migration procedure is placed alongside the business logic and is called 'lazily' for each instance of the business process at the moment it is restored from the database;
If it is necessary to migrate the state of all process instances quickly and synchronously, more classical database migration solutions are used, but there you have to work with JSON.

Is another framework for business processes needed?

The solutions described in the article have allowed us to significantly simplify our lives, expand the range of issues addressed at the application development level, and make the ideas of isolating business logic into microservices more attractive. A lot of work has been done for this, culminating in the creation of a very 'lightweight' framework for business processes, as well as service components for addressing the indicated problems in the context of a wide range of application tasks. We are eager to share these results and open our general components to the public under a free license. This will require certain efforts and time. Understanding the demand for such solutions could serve as an additional incentive for us. The proposed article pays very little attention to the capabilities of the framework itself, but some of them are evident from the examples presented. If we do publish our framework, a separate article will be dedicated to it. For now, we would appreciate if you could leave some feedback by answering the question:

Only registered users can participate in the survey. Please log in, please.

Is another framework for business processes needed?

18,8%Yes, we've been looking for something like this for a long time.
12,5%I'm interested in learning more about your implementation; it might come in handy.
6,2%We are using one of the existing frameworks but are considering a replacement.
18,8%We are using one of the existing frameworks; everything is satisfactory.
18,8%We are managing without a framework.
25,0%We are writing our own.