On Photo-Realism In Games

Below is an interview I gave a few months ago for a UK magazine on the topic of photo-realistic graphics that I have kindly been given permission to reprint. I provided an animator’s perspective as it also featured insights from developers working on racing games that are at the forefront of realistic mechanical rendering – something I hadn’t considered before. It would be interesting to know others’ thoughts on the matter, as it touches on that timeless classic, the Uncanny Valley.

From an animation perspective, how do you view the prospect of photo-realistic graphics?

I believe that years ago the game animation community realised it was something of a false god – that rather than realism we should instead be pushing for believability, (not realism), in our characters. It’s a lot easier to make a convincing performance with a character with exaggerated features when you can concentrate solely on acting instead of worrying about details such as the perfect movement of skin and muscle across a skull or the most accurate rendering of wetness in the eye. Ultimately it depends what kind of experience you’re hoping to achieve with your game. Assassin’s Creed simply wouldn’t work as a cartoony game, and the reverse is true for the Mario series.

Having worked on games like Assassin’s Creed and Uncharted, which already have a large number of complex animations, what are the major challenges in getting those animations to look “right? in a real-world environment?”

The primary challenge I’ve faced is to give the character enough weight in the world so that they naturally fit in a realistic setting, without making it unresponsive for the player to the point that the connection is lost with the avatar – that sense of embodying the player character is central to every decision a game animator makes. This means we spend a lot of time trying to avoid sudden changes in momentum or unnatural weight-shifts while at the same time balancing that with the controls. We can’t resort to having crazy-high jumps or incredibly reactive locomotion with characters turning on a dime, or performing ridiculously fast combos during combat that “conventional design wisdom” says are easy ways to make a character feel better. That said, restrictions like this can be liberating too – real-world physics are a great guiding principle to maintain stylistic consistency throughout a game.

On a technological level, how does current motion-capture tech stand with regard to photo-realism? Is it sufficient to capture nuanced enough animations? Or would you need further advances for it to be possible?

The capturing of motion itself hasn’t improved drastically in the decade I’ve been using it, but the results we are getting are so good to the point that we no longer fight with it to get it working on our characters. This frees us up to use all our animator know-how on the creative side – improving posing, exaggerating actions and timing among other tricks that make up for the “real” actors’ limitations. We have seen improvements in full performance-capture though, with the simultaneous recording of body, face and voice allowing for more wholistic performances rather than being cobbled together at a later stage via multiple actors. In a way, a lot of the advances I can see in the future will be aiming to reduce the physical awkwardness an actor must endure on a shoot so they can better get into character and deliver the best performance possible. Putting that in the hands of a skilled animator is the winning combo for successful game animation.

What about alternative animation methods such as keyframes or the use of physics to drive more emergent character movement. Would those play a role in photo-realism?

Keyframing will never go away. Even with the best performance-capture solutions, there are simply some actions that cannot physically be performed by an actor, such as very acrobatic moves, death-defying stunts, or feats of super-human strength. Physics has been on my radar for years but is for now just a little out of reach from providing a complete solution for a realistic character. I’ve seen it work greatly on cartoony or non-human characters though, and the sense of self-preservation that comes from artificial intelligence using dynamic behaviours to try something as simple as stand up after being prodded can be quite a joy to watch. For now we use phsyics as a tool to supplement animation, such as on cloth or flowing hair, and most recently to provide extra weight on characters as they make quick changes in momentum, again without sacrificing either weight or responsiveness.

A big issue with the idea of photo-realism is that it’s impossible to fully predict what a player will do in a realtime scenario, and thus how the game will “look” at all times. This is especially the case with animations. How do you try to tackle this?

This is a major difference between interactive games and more passive linear mediums such as film or television, where scenes are pre-determined and therefore created only to the specifications as are needed for each section of the narrative. We have a variety of tricks we’ve learned over the years such as parametric blending – the process of creating multiple animations for one action, and blending between them depending on the player’s input, (how far or high the player jumps etc). A new approach we’ll be seeing in the future that I’m very excited about is the use of “Motion Graphs”. Where in the past a game character had a collection of short animation cycles that were created to loop or hit pre-determined poses, in the future large banks of non-looping motion capture will be curated and maintained by the animator, with the player input affecting which sections to play back on the player character on the fly. This not only makes characters more responsive as they can interrupt animations at any point during a move, but the visual quality is unmatched as we’re no longer seeing noticeable loops or popping between pre-created animations – instead different animations are dynamically pose-matching whenever required.

L.A. Noire

One area that is particularly problematic is facial animations. As we saw with LA Noire, the “uncanny valley” effect is very difficult to overcome. Is it possible to have highly facially expressive characters without making the player feel uncomfortable?

It is absolutely possible, but again requires a LOT of work and skill. Animators must work in concert with character artists and technical artists to ensure faces never go “off model”, (look unusual or creepy). The real challenge is to do this consistently for a 30+ hour game with hours of close-up cinematics on more characters than a typical television series. I loved LA Noire – it was a fascinating experiment in inter-character dynamics and for the most part avoided the uncanny valley, but the nature and sheer scale of an open-world game ensured they couldn’t apply the same level of quality to every character. Imagine if that whole game centred only on conversations between Cole Phelps and his wife in their home, (as an example of focus) – that could have been visually amazing!

I’ve referred mostly to character animation so far, but there are many other aspects to a game that are animated. Animals, mechanisms, even things like trees swaying in the breeze. Things move all the time in our environment. How granular can animation be before it becomes unrealistic to develop?

I believe that we, as viewers, don’t apply the same standards to how a pig might walk as a human walks because we don’t see it on a daily basis. Therefore, they are far easier to say “that’s good enough”. Big-budget videogame development is all about choosing where to spend your efforts and so the vast majority of time is always going to be on your player character, then the NPCs you interactive with, then the ones you don’t and so on. I expect that even as main characters increase in quality, a door just needs to work. Environments are becoming much more alive, again with the help of physics. Simulating wind through a natural environment is enough to make trees sway and grass ripple, with little to no effort being taken by an animator. One of the things I love about game development is we’re always finding easier ways to automate complex tasks – it’s how game teams nowadays, while clearly larger than when I started a decade and a half ago, haven’t had to scale up to the same ratio as the level of quality has increased.

And of course there’s the fact that games often feature beings that are completely fictional, aliens, monsters and so forth. How do you “realistically” animate something that isn’t real?

This is the most fun part of our job! Animators can really let loose their imaginations on non-existent creatures and such. The trick is to always find a real-world equivalent to base it on and use as reference. From jellyfish to bats to microscopic bacteria – the more outlandish the better. Importantly, fantastical creatures in a realistic setting must still obey the same laws of physics as their human counterparts so as not to seem out of place. If a creature is large, it needs to move in a slow and heavy manner consistent with its surroundings. It also buys us a lot of leeway that we don’t get with humans – I recall that even the same facial animations played on Mass Effect’s blue-skinned Asasri looked instantly more “believable” than on their human female counterparts.

You mentioned working on more stylised games recently, and noted a significant difference between animating for that and games that attempt a more realistic aesthetic. What insights has this approach provided?

What surprised me most about the challenge of a non-realistic game was that working towards a realistic style gives a great yard-stick for your measurement of success. You know when it’s not working, and can break it down and work towards what you know to be correct. Creating stylised characters on the other hand can take any of an infinite amount of directions so it’s much harder to figure out exactly what is and isn’t working. Ultimately, you’re searching for what we in animation call “appeal”. Pixar characters have it. Team Fortress 2 characters have it in spades, where others fall flat and are forgettable – the level of artistry required to achieve that kind of quality is what inspires me to keep trying harder.

Is there anything you’d like to add?

I get excited about graphics every time we take a technological leap forward in our hardware, but one only needs to look at film to estimate how long it’ll take before we achieve true photorealism with characters. Film has only recently achieved that which is indistinguishable from reality, but they are usually only on screen for as long as required and not seen from all angles at the same level of scrutiny afforded by games. A while back I attended a lecture on the creation of the tiger Richard Parker from Life of Pi, and it would take them around 30 hours to render just one frame whereas in games we target 30-60 frames every second. As such, I don’t expect true photo-realistic characters in games for at least a decade or two. In the meantime though, our efforts are better served pursuing advances in AI and deep character interaction – it won’t matter what an NPC looks like if he/she can perfectly react to me in a completely believable manner. I’m not talking about enemies perfectly flanking my cover – imagine instead an RPG party with all the social dynamics of a real group of people – that’s what I look forward to the most.