When we talk about artificial intelligence in games, we usually picture smarter or more realistic enemies that don’t come off as mindless automatons. New research, though, is showing how an AI powered by a neural network could revolutionize the way player avatars animate realistically through complicated game environments in real time.
Phase-Functioned Neural Networks for Character Control is a fundamentally new way of handling character animation that will be presented at the ACM’s upcoming SIGGRAPH conference this summer. In most games, character animation is handled through “canned,” pre-recorded motion capture. This means an average player will see precisely the same motion cycled repeated thousands of times in a single play-through. “Our system works completely differently,” University of Edinburgh researcher Daniel Holden told Ars in a recent interview.
“We start by making a huge database of animation data,” he said. “And we use machine learning to produce a system which maps directly from the user input to the animation of the character. So, instead of storing all the data and selecting which clip to play with, [we] have a system which actually generates animations on the fly, given the user input.”
You can instantly see the value of this new method by watching a video that accompanies his company’s recent paper on The Orange Duck. The video shows off a tricorn-clad avatar making a wide variety of freeform, natural-looking movements through extremely complex terrain.
The most impressive bit of the demo comes at about 2:27, when the avatar moves over the rocky terrain “climbing, balancing, and jumping where required,” as the narrator puts it. You can see the avatar realistically swing his arms for momentum and put down hands for additional balance before swinging his knees and legs with tiny variations depending on the specifics of the desired movement direction and the environment. The character even naturally slows to a stop when approaching walls, rather than just running full speed and abruptly stopping.
These small variations based on circumstance really make the animation stand out from the usual canned motion capture. The avatar may throw out a hand to steady himself after a dip or push up extra hard with his left leg to get up a tall boulder. In just a minute of watching, you’ll see enough of these small, lifelike details to make the animation stand out.
Learning how to move
For this demo, Holden and his colleagues at the University of Edinburgh and Method Studios captured about 1.5GB of motion-capture data in a single two-hour session. After that, a neural network trained itself on the data for about 30 hours. In essence, the network learned how to blend those specific mo-cap animations into totally new combinations that can be applied to generalized in-game situations.
“The weights of the neural network represent something like different components that make up a pose, and the input ends up producing something like a weighted sum of these components,” Holden explains.
This leads to a much wider range of possible animation outputs than what has been directly recorded. To step down from a ledge, for instance, a traditional animation system might just load up one standardized “step down from ledge” animation file and be done with it. The neural network, on the other hand, extrapolates specifically how to move each individual body part by combining data from similar situations found in the initial motion capture.
What started as 1.5GB of training data ends up as just dozens of megabytes in memory when stored in this form as a neural network, Holden explains. “Essentially what happens is that the animation data gets ‘compressed’ into the weights of the neural network. If every pose in the database can be made up of weighted sums of just a few components (called basis), then the neural network learns this, and it can compress the data greatly.”
Pros and cons
While other animation methods can blend different motion capture “scenes” into combined animation for new situations, those methods tend to require storing large databases locally, and they can slow down a system. Recent work on other neural-network based animation models can end up looking rough and unnatural without the addition of a cyclical “phase” that helps guide the neural network through the blending process more precisely.
Using neural networks to modify your motion-captured animation can lead to some unexpected results, Holden said. For instance, the researchers didn’t provide any motion-capture data specifically tuned for a character crouching while walking over rough terrain. The system managed to learn how to handle this situation though, Holden said, by combining animations from crouching on flat terrain with those of walking or running over rough terrain.
While Holden says these crouching-on-uneven-ground animations “aren’t as good as the other results,” they “probably would be acceptable in most cases.” And though the system “is probably learning more than we expected in some areas,” there are limits to how far the neural network can go. The animation breaks altogether if the terrain is too steep, for instance, “unless you give it the data you basically have no guarantee it will work.”
For all the benefits of this AI-driven approach to animation, Holden admits to some drawbacks. Having to wait through 30 hours of training time after motion capture can be a significant imposition, especially if you want to re-record something after the fact. Artists can’t easily go in and “touch up” the output of a neural network, either, as they can with more traditional, canned animation. And while the neural network’s real-time decisions operate quickly (~1ms), they still take up much more processor time than simply playing a pre-recorded animation (but Holden hopes “there will probably be ways to speed it up in the future.”)
While Holden has “just joined Ubisoft to do more R&D,” he couldn’t comment on whether we’d see his animation methods in any specific upcoming games. We hope this research (or something like it) comes to games soon, though, just to save us from seeing those same canned animations running endlessly in our heads.
This research also has us wondering what other areas of game development might be revolutionized by neural networks in the near future.
Listing image by Holden et al