🥇How we made the mechanics of ballistic calculation for a mobile shooter with a network delay compensation algorithm

Hi, I'm Nikita Brizhak, server developer at Pixonic. Today I would like to talk about lag compensation in mobile multiplayer.

Many articles have been written about server lag compensation, including in Russian. This is not surprising, because this technology has been actively used in the creation of multiplayer FPS since the late 90s. For example, we can recall the QuakeWorld mod, which was one of the first to use it.

We also use it in our mobile multiplayer shooter Dino Squad.

In this article, my goal is not to repeat what has already been written a thousand times, but to tell how we implemented lag compensation in our game, taking into account our technology stack and core gameplay features.

In a few words about our core and technologies.

Dino Squad is an online mobile PvP shooter. Players control dinosaurs equipped with a variety of weapons and fight each other in 6v6 teams.

Both the client and the server are based on Unity. The architecture is quite classic for shooters: the server is authoritarian, and client-side prediction works on clients. The game simulation is written using in-house ECS and is used both on the server and on the client.

If you have heard about lag compensation for the first time, here is a brief digression into the issue.

In multiplayer FPS games, the match is usually simulated on a remote server. Players send their input (information about the keys pressed) to the server, and in response, the server sends them an updated game state based on the received data. With this interaction scheme, the delay between pressing the "forward" key and the moment when the player's character moves on the screen will always be more than a ping.

If on local networks this delay (popularly referred to as input lag) may not be noticeable, then when playing via the Internet it creates a feeling of “sliding on ice” when controlling a character. This problem is doubly relevant for mobile networks, where the case when the player has a ping of 200ms is considered to be still a great connection. Often the ping is 350, and 500, and 1000 ms. Then it becomes almost impossible to play a fast shooter with an input lag.

The solution to this problem is client-side simulation prediction. Here the client itself applies the input to the player's character, without waiting for a response from the server. And when the answer is received, it simply checks the results and updates the positions of the opponents. The delay between pressing a key and displaying the result on the screen in this case is minimal.

It is important to understand the nuance here: the client always draws itself according to its last input, and enemies - with a network delay, according to the previous state from the data from the server. That is, when shooting at the enemy, the player sees him in the past relative to himself. More about client prediction we wrote earlier.

Thus, the client prediction solves one problem, but creates another: if the player shoots at the point where the enemy was in the past, on the server, when shooting at the same point, the enemy may no longer be in that place. Server lag compensation tries to solve this problem. When a weapon is fired, the server restores the state of the game that the player saw at the time of the shot locally, and checks whether he really could have hit the enemy. If the answer is "yes", the hit counts, even if the enemy is no longer on the server at that point.

Armed with this knowledge, we started implementing server lag compensation in Dino Squad. First of all, it was necessary to understand how to restore on the server what the client saw? And what exactly needs to be restored? In our game, weapon and ability hits are calculated through raycasts and overlaps - that is, through interactions with the enemy's physical colliders. Accordingly, the position of these colliders, which the player "saw" locally, we needed to reproduce on the server. At that time, we were using Unity version 2018.x. The physics API is static there, the physical world exists in a single copy. There is no way to save its state, so that later it can be restored out of the box. So what to do?

The solution was on the surface, all its elements have already been used by us to solve other problems:

For each client, we need to know at what time he saw opponents when he pressed the keys. We already wrote this information to the input package and used it to correct the work of the client prediction.
We need to be able to store a history of game states. It is in it that we will hold the positions of opponents (and hence their colliders). We already had a state history on the server, we used it to build deltas. Knowing the right time, we could easily find the right state in history.
Now that we have the state of the game from history in our hands, we need to be able to synchronize the player data with the state of the physical world. Move the existing colliders, create the missing ones, destroy the extra ones. This logic was also already written by us and consisted of several ECS systems. We used it to keep several game rooms in one Unity process. And since the physical world is one per process, it had to be reused between rooms. Before each tick of the simulation, we "reset" the state of the physical world and re-initialized it with data for the current room, trying to reuse Unity game objects as much as possible through a tricky pooling system. It remained to call the same logic for the game state from the past.

By bringing all these elements together, we got a "time machine" that could roll back the state of the physical world to the right moment. The code turned out to be uncomplicated:

public class TimeMachine : ITimeMachine
{
     //История игровых состояний
     private readonly IGameStateHistory _history;

     //Текущее игровое состояние на сервере
     private readonly ExecutableSystem[] _systems;

     //Набор систем, расставляющих коллайдеры в физическом мире 
     //по данным из игрового состояния
     private readonly GameState _presentState;

     public TimeMachine(IGameStateHistory history, GameState presentState, ExecutableSystem[] timeInitSystems)
     {
         _history = history; 
         _presentState = presentState;
         _systems = timeInitSystems;  
     }

     public GameState TravelToTime(int tick)
     {
         var pastState = tick == _presentState.Time ? _presentState : _history.Get(tick);
         foreach (var system in _systems)
         {
             system.Execute(pastState);
         }
         return pastState;
     }
}

It remained to figure out how to use this machine to compensate for shots and abilities.

In the simplest case, when the mechanics are built on a single hitscan, everything seems to be clear: before the player shoots, you need to roll back the physical world to the desired state, make a raycast, count a hit or miss, and return the world to its initial state.

But there are very few such mechanics in Dino Squad! Most of the weapons in the game create projectiles - long-lived bullets that fly for several ticks of simulation (in some cases, dozens of ticks). What to do with them, at what time should they fly?

В ancient article about the Half-Life networking stack, the folks at Valve asked the same question, and their answer was that projectile lag compensation is problematic and best avoided.

We didn't have this option: projectile-based weapons were a key feature of the game design. So we had to come up with something. After a bit of brainstorming, we formulated two options that seemed to work for us:

1. We bind the projectile to the time of the player who created it. Each tick of the server simulation for each bullet of each player, we roll back the physical world to the client state and perform the necessary calculations. This approach made it possible to have a distributed load on the server and a predictable projectile flight time. Predictability was especially important to us, since we have all projectiles, including enemy projectiles, predicted on the client.

In the picture, the player in the 30th tick shoots a rocket in advance: he sees in which direction the enemy is running, and knows the approximate speed of the rocket. Locally, he sees that he hit the target in the 33rd tick. Thanks to lag compensation, he will also get on the server

2. We do everything the same as in the first option, but after counting one bullet simulation tick, we do not stop, but continue to simulate its flight within the same server tick, each time bringing its time closer to the server one by one tick and updating collider positions. We do this until one of two things happens:

The bullet has expired. This means that the calculations are over, we can count a miss or a hit. And this is in the same tick in which the shot was fired! For us, this was both a plus and a minus. Plus, because for the shooting player, this significantly reduced the delay between hitting and reducing the health of the enemy. The downside is that the same effect was observed when opponents fired at the player: the enemy, it would seem, only fired a slow rocket, and the damage was already counted.
The bullet has reached server time. In this case, its simulation will continue in the next server tick without lag compensation. For slow projectiles, this could theoretically reduce the number of physics "rollbacks" compared to the first option. At the same time, the uneven load on the simulation increased: the server either stood idle, then calculated a dozen simulation ticks for several bullets in one server tick.

The same scenario as in the previous picture, but calculated according to the second scheme. The missile “caught up” with the server time in the same tick that the shot occurred, and the hit can be counted on the next tick. In the 31st tick, in this case, lag compensation is no longer applied.

In our implementation, these two approaches differed in just a couple of lines of code, so we both washed down, and for a long time they existed in parallel with us. Depending on the mechanics of the weapon and the speed of the bullet, we chose one or another option for each dinosaur. The turning point here was the appearance in the game of a mechanic like “if you hit the enemy so many times in such and such time, get such and such a bonus.” Any mechanic where the timing at which the player hit an enemy was important refused to be friends with the second approach. So in the end we settled on the first option, and now it applies to all weapons and all active abilities in the game.

Separately, it is worth raising the issue of performance. If it seemed to you that all this would slow down, I answer: it is so. Moving colliders, turning them on and off, Unity is pretty slow. In Dino Squad, in the "worst" case, several hundred projectiles can exist in combat at the same time. Moving colliders to count each projecttile individually is a luxury. Therefore, it was absolutely necessary for us to minimize the number of "rollbacks" of physics. To do this, we created a separate component in ECS, in which we record the player's time. We hung it on all entities that require lag compensation (projectiles, abilities, etc.). Before we start processing such entities, we cluster them by this time and process them together, rolling back the physical world once for each cluster.

At this stage, we got a generally working system. Her code is somewhat simplified:

public sealed class LagCompensationSystemGroup : ExecutableSystem
{
     //Машина времени
     private readonly ITimeMachine _timeMachine;

     //Набор систем лагкомпенсации
     private readonly LagCompensationSystem[] _systems;
     
     //Наша реализация кластеризатора
     private readonly TimeTravelMap _travelMap = new TimeTravelMap();

    public LagCompensationSystemGroup(ITimeMachine timeMachine, 
        LagCompensationSystem[] lagCompensationSystems)
     {
         _timeMachine = timeMachine;
         _systems = lagCompensationSystems;
     }

     public override void Execute(GameState gs)
     {
         //На вход кластеризатор принимает текущее игровое состояние,
         //а на выход выдает набор «корзин». В каждой корзине лежат энтити,
         //которым для лагкомпенсации нужно одно и то же время из истории.
         var buckets = _travelMap.RefillBuckets(gs);

         for (int bucketIndex = 0; bucketIndex < buckets.Count; bucketIndex++)
         {
             ProcessBucket(gs, buckets[bucketIndex]);
         }

         //В конце лагкомпенсации мы восстанавливаем физический мир 
         //в исходное состояние
         _timeMachine.TravelToTime(gs.Time);
     }

     private void ProcessBucket(GameState presentState, TimeTravelMap.Bucket bucket)
     {
         //Откатываем время один раз для каждой корзины
         var pastState = _timeMachine.TravelToTime(bucket.Time);

         foreach (var system in _systems)
         {
               system.PastState = pastState;
               system.PresentState = presentState;

               foreach (var entity in bucket)
               {
                   system.Execute(entity);
               }
          }
     }
}

It remains only to configure the details:

1. Understand how much to limit the maximum range of movement in time.

It was important for us to make the game as accessible as possible in conditions of poor mobile networks, so we limited the story with a margin of 30 ticks (with a tick rate of 20 Hz). This allows players to hit opponents even at very high pings.

2. Determine which objects can be moved in time and which cannot.

We, of course, move the opponents. But installed energy shields, for example, are not. We decided it was better to prioritize the defensive ability here, as is often done in online shooters. If the player already placed the shield in the present, lag-compensated bullets from the past should not fly through it.

3. Decide if dinosaur abilities should be lag compensated: bite, tail strike, etc. We decided that it is necessary and process them according to the same rules as bullets.

4. Determine what to do with the colliders of the player for whom lag compensation is being performed. In a good way, their position should not shift to the past: the player should see himself at the same time in which he is now on the server. However, we also roll back the colliders of the shooting player, and there are several reasons for this.

First, it improves clustering: we can use the same physics state for all players with close pings.

Secondly, in all raycasts and overlaps, we always exclude the colliders of the player who owns the ability or projecttile. In Dino Squad, players control dinosaurs, which have a rather non-standard geometry by the standards of shooters. Even if the player fires at an unusual angle and the bullet's trajectory goes through the player's dinosaur collider, the bullet will ignore it.

Thirdly, we calculate the position of the dinosaur's weapon or the point of application of the ability from the data from the ECS even before the start of the lag compensation.

As a result, the real position of the colliders of the lag-compensated player is not important for us, so we took a more productive and at the same time simpler path.

Network delay cannot simply be removed, it can only be masked. Like any other method of disguise, server lag compensation has its own tradeoffs. It improves the game experience of the shooting player at the expense of the player who is being shot at. For Dino Squad, however, the choice here was obvious.

Of course, all this had to be paid for by the increased complexity of the server code as a whole - both for programmers and game designers. If earlier the simulation was a simple sequential call of systems, then with lag compensation, nested loops and branches appeared in it. We also spent a lot of effort to make it convenient to work with.

In the 2019 version (or maybe a little earlier), Unity introduced plus or minus full support for independent physics scenes. We implemented them on the server almost immediately after the update, because we wanted to get rid of the physical world common to all rooms as soon as possible.

We gave each game room its own physical scene and thus got rid of the need to “clear” the scene from the data of the neighboring room before calculating the simulation. First, it gave a significant performance boost. Secondly, it allowed to get rid of a whole class of bugs that arose if the programmer made a mistake in the scene clearing code when adding new game elements. Such bugs were difficult to debug, and they often resulted in the state of physical objects from the scene of one room "flowing" into another room.

In addition, we did some research on whether physical scenes can be used to store the history of the physical world. That is, conditionally, to allocate to each room not one scene, but 30 scenes, and make a cyclic buffer out of them, in which to store the history. In general, the option turned out to be working, but we did not implement it: it did not show any crazy performance gain, but it required rather risky changes. It was difficult to predict how the server would behave when working with so many scenes for a long time. So we followed the rule:If it ain't broke, don't fix it».

Source: habr.com

How we made the mechanics of ballistic calculation for a mobile shooter with a network delay compensation algorithm

Add a comment Отменить ответ