Continuing a topic previously raised in our blog , we will discuss the applicability of machine learning to it and in what form. With our experiences and the solutions chosen based on them expert in AI issues at Apex Game Tools, Jacob Rasmussen.
In recent years, there has been much discussion about how machine learning will fundamentally change the gaming industry, as this technology has already proven groundbreaking in many other digital applications. However, it is essential to remember that games are much more complex than a driving simulator, drone control software, or facial recognition algorithms.
While the gaming industry continues to rely on traditional AI methods, such as finite state machines, behavior trees, and — more frequently in recent times — Utility-based AI (utility-based systems), these AIs are also referred to as design-based (artificial design intelligence) or expert systems. However, it is becoming increasingly clear — primarily to the players — that these systems are becoming less suitable for creating truly advanced opponents capable of mimicking player behavior. This is particularly true for creative solutions. This can be explained by the fact that AI developers cannot take into account all possible tactics and behavioral strategies and successfully implement them in traditional AI systems. For players, this often leads to a boring and predictable experience when facing an opponent whose behavior is easily memorized.
This outcome is due to multiple reasons, but one of the main ones is the AI's inability to learn. Therefore, when creating opponent AI, it naturally leads to considering a transition to machine learning, which has proven highly effective in many other applications. However, there are several nuances to consider. For instance, gaming AI must be able to adapt to any situation and leverage the advantages it presents, as well as adjust to various opponent play styles — both live players and other AIs.
Current Status
The British company DeepMind, which is involved in the development of artificial intelligence, recently demonstrated how AI can learn to play games autonomously, understand their rules, and find ways to complete or win them—though currently only with simple games, such as early ones. — for example, chess and the Japanese strategy game go. The results obtained for these games show that artificial intelligence can form an adequate assessment of what is happening on the board. However, when it comes to adapting AI to different opponent play styles, the results are not yet so impressive.
Nowadays, neural networks have learned to recognize images and drive cars. However, these functions can also be implemented using relatively simple architectures, even if the end result turns out to be quite deep and extensive. For instance, the image recognition AI on Facebook has a depth of about 100 layers, making it reminiscent of a biological brain—in terms of the number and complexity of the connections between neurons, forming one large network.
Gaming AI
Regarding the application of machine learning in the gaming industry, there are several limitations that prevent this type of architecture from being used universally. These include system requirements, particularly related to CPU capabilities, which determine a computer's ability to process complex gaming structures and its suitability for storytelling and gameplay.
Thus, in many games, the implementation of a complex artificial intelligence system may not have the necessary hardware infrastructure, let alone a server cluster, which exists, for example, for image recognition networks at Facebook. Sometimes multiple AIs must operate simultaneously—not only on computers, but also on mobile devices and other less powerful platforms. All this imposes restrictions on the size and complexity of the machine learning architecture, as all computations must also be executed within a frame duration of about 1 or 2 milliseconds. Of course, various optimization and load distribution technologies can be used between frames, but it still does not eliminate these limitations completely.
Serious problems for AI can arise from the complexity of the game. In games like StarCraft II, for example, the game mechanics are many times more complex than in Atari games. Therefore, it is unrealistic to expect that at a given frame rate and with known system requirements, machine learning will successfully learn and interact with the entire state of the game. Just as a player often relies on intuition in the early stages of the game, AI must learn to process the game state preliminarily to simplify its further progression. For example, in one of the latest only displays the information that developers deemed important: in one case, the AI operated with a reduced view of the entire map area, while in another case, similar to the player, it could move the camera, and then its perception was limited to the information on the screen.
Visualization of AI AlphaStar playing against a player in StarCraft II: the screenshot shows the 'raw' original observations, neural network activity, some of its possible actions and coordinates, as well as the predicted match outcome.
This is especially relevant in the case of games. Often, conventional machine learning problem-solving methods do not apply to game AI. For example, it is usually not required to win or do everything possible to secure victory, as was the case with Atari games. More often, the role of AI is to make gameplay more engaging. It may need to act and behave as the character it is responsible for suggests. Thus, game AIs are more tied to game design and storytelling and must have the necessary tools to manage their behavior to achieve their goals. Pure machine learning isn't always suitable for this — which means alternative solutions need to be explored.
Practical machine learning issues
These issues arose during the development of machine learning-based AI for , where AIs need to behave like regular players — that is, be as flexible and resourceful.
Like Starcraft II, Unleashed is much more complex than chess and Go for Atari. Its gameplay is intuitive and easy to learn, but to truly excel, players must develop specific skills in managing the meta. Players need to build mazes, unleash monsters on enemies, and devise strategies for economics, offense, and defending structures throughout the game. To do this, they must bluff and anticipate their opponents' moves, as well as manage the psychological meta — it is this that elevates poker from being merely a statistical game.
Screenshot from Unleashed
In search of the most suitable architecture for these purposes, technologies like and deep learning were initially implemented into the game in a virtually unchanged form, testing how they would perform raw as an enemy AI.
It was terrible.
It quickly became clear that Unleashed had many global problems that were challenging to adapt to machine learning.
One of them is building an effective maze. As in many tower defense games, players need to construct a maze around the tower to stop monsters from breaking through. To tackle these monsters, players must use weapons placed throughout the maze. Ideally, the maze should be as long as possible to inflict sufficient damage on the monsters before they reach the tower. Certain weapons are more effective against specific types of monsters, so they should be positioned earlier in the maze for greater efficiency. The unique aspect of Unleash is that there is no perfect maze; the game features so many types of monsters that inevitably, some will pass through any section of the maze. Each maze must be adapted to accommodate new monsters introduced by other players. Therefore, it was essential not only to teach artificial intelligence how to build mazes but also to create efficient mazes for various scenarios that players might encounter both in the early and late versions of the game.
Additionally, the AI had to learn to calculate which monsters would appear in the maze. This is a kind of inverse problem relative to maze construction. As in many other games, in Unleash it's not enough to just build up an army and send it against the enemy; it is also necessary to spy on the enemy's defenses and structure the army in such a way that it can effectively strike at the opponent's weak points. The monster army must interact with each other in a way that maximizes their chances of breaking through the maze. Sometimes, it is also necessary to release monsters in a specific order depending on their functions and roles. This adds to the various combinations available.
Finally, since the player must both create mazes and gather a monster army, the AI also needs to learn to find a balance between attack and defense. It's worth considering that the more the player grows the monster army and builds the maze, the more resources are needed for this. Therefore, an effective attack strategy is crucial for both the game's economy and achieving victory. To ensure competitiveness, the AI must be able to allocate resources in a way that creates a powerful monster army without jeopardizing the strength of the maze. Investing heavily in monsters may be economically beneficial, but it increases the risk of the opposing monsters capturing the maze. Conversely, focusing on strengthening the maze's defense can undermine your economy. None of these scenarios will lead to victory. Thus, the optimization problem in Unleashed turns out to be more complex than in chess or Starcraft, involving the need to sacrifice something and calculating benefits several steps ahead.
As the artificial intelligence learns, various previously unaccounted issues arise. Initially, the AI often reached a certain level of development where it began to understand some aspects of the game — for instance, which weapons in the maze are effective against specific types of monsters or which monsters are best suited for certain maze sections. However, the learning process was slow and led to the development of repetitive strategies.
The necessity for parallel approaches
While the development of AI based on machine learning progressed slowly and wasn't particularly successful, better AI and more reliable competitive AI became necessary for other stages of testing and development. The Utility architecture was utilized to create specialized AI for game quality testing, in-game tests, weapon and monster balancing, and for designing specific mazes and monsters. However, during the development of Unleash, the creators also honed their skills in navigating through it and later decided to use the knowledge gained to create a more advanced Utility AI. It became clear that many problems encountered in machine learning-based AI systems could be easily resolved using Utility systems that leverage their embedded knowledge, and vice versa.
For instance, more effective mazes are better created using Utility AI based on knowledge bases compiled from the results of internal tests. It's straightforward to describe and program an algorithm for constructing a maze and placing weapons in such a way that it would be easier for a live player to defend a tower against specific monsters. However, creating an army of monsters based on knowledge of an opponent's base proved to be a challenging task for such AIs, as the number of different conditions and combinations to consider was overwhelming. With this type of AI architecture, finding suitable monster sets would take an infinitely long time. Therefore, given the constraints, deep learning would be an ideal solution for this problem.
Creating hybrid AIs
Thus, it was decided to combine two approaches to create a hybrid artificial intelligence system based on machine learning and Utility. The idea was that where it was necessary to process a vast number of combinations and game states, or where training was needed, machine learning would be utilized. For other tasks, where it’s better to rely on the developers' personal experience, Utility systems were employed. The advantage of this approach also lies in the ability to better control the AI's behavior when necessary, ensuring it more accurately follows the assigned goal. For example, Utility AI can be used to ensure a balance between offense and defense, thus creating varying levels of aggression, or different maze configurations for different AIs, allowing them to develop unique play styles. Specific value systems can also be assigned to neural networks to shape different preferences for selecting aerial or ground monsters, adding individuality to separate AIs. There are many other options for implementing project solutions, all emphasizing the strengths of different types of AI architecture.
The hybrid approach also addressed another question that arose for the development team during the creation of artificial intelligence for Unleash: should a single global deep neural network based on machine learning be applied to account for all input and output data, or is it better to design the AI with a hierarchical structure?
Two architectures used in Unleash: on the left — a large deep neural network with its own unified architecture, on the right — a hierarchical system where each network has its own task.
Nonetheless, there is a desire to create a unified approach to artificial intelligence systems, one in which developers do not impose their own experiences into the architecture. However, the more entries there were into the game, the more the neural network grew. It was not possible to separate AI training and focus it on a single aspect: either defense or attack. Concerns arose that a more generalized approach would lead to a substantial increase in computational requirements.
From this, the idea emerged to create a hierarchical architecture, where each specific task would be handled by a specialized neural network. According to this concept, the artificial intelligence must first decide how to allocate resources for attack (increasing the army of monsters) and defense (building the maze). Once it does this, it moves to the next layer based on its choice and gains access to the necessary part of the game state, after which it makes detailed decisions about which monsters to choose and what weapons to place in the maze.
Conclusion and Next Steps
In a hybrid approach, Utility AI incorporating machine learning-based networks resembles a hierarchical architecture. This, in turn, is similar to a biological brain, where different neural centers are responsible for distinct tasks.
Currently, the AI opponents in Unleash are very difficult to defeat: they can adapt to any situation in the game, yet developers can adjust their settings at will. The author of the article believes that, over time, the hybrid approach should gain more traction and appear in many other games. Perhaps one day, it will be possible to integrate pure machine learning-based artificial intelligence into the gameplay. However, it is clear that this will take time. For now, the goal is to find an architecture that can adapt to the tasks it faces and discover optimal ways to solve them.
Source: habr.com
