This is an ‘HD Remaster’ of a talk I gave at the Twitch Animation Exchange in February 2017.
Find a complete transcript of the talk below.
Interactive Cinematics (IGCs)
So what are Interactive Cinematics?
Well basically they’re any time we’re in a unique moment in the game and it’s neither full gameplay nor is it a full cutscene, but there’s something for the player to do.
We refer to these internally as In-Game Cinematics (IGCs for short) although that’s something of a misnomer because, while PS3-era Naughty Dog cinematics were all pre-rendered, all cinematics from Uncharted 4 onwards are real-time in-game so the distinction is now whether they are interactive or not.
So today we’re talking about interactive cinematics, but to keep with habit I’m still going to refer to them as IGCs as that’s how they’re referred to in the studio.
Introduction
A little bit about my background, I’ve worked on several big games typically heavy on story and I’ve worked in different roles; cinematics or in-game, and usually they’re quite separate.
Most games have either distinct gameplay or cutscenes, but Uncharted 4 was the first time I had the opportunity to blend the two as that’s something that Naughty Dog really pushes. It was a lot of fun – essentially today I’m just gonna talk to you about how fun my job is.
Currently I’m creating these kinds of interactive scenes for The Last of Us: Part 2 so just bear in mind that Uncharted 4, Uncharted The Lost Legacy and The Last of Us: Part 2 all share a similar process and philosophies.
Overview
So the meat of this presentation is split into 3 parts. First I’m gonna go over my process. The way I typically set up a scene and take it all the way through to final.
Next I’m going to show a selection of examples that illustrate the various technical tricks and techniques that are used commonly throughout the games.
And finally I’m gonna finish up with a deep-dive on Uncharted 4’s Madagascar chase sequence that also doubled as our E3 2015 demo. That was one of the most complex scenes to work on and it really used every trick we have.
Process
So first up, Process. Every scene has to be set up in Maya correctly, then we’re going to bring in props.
I’m going to take you through the previz stage – that’s a lengthy process where we’re figuring out the issues with design via lots of iteration.
Once we’re done there we go to motion-capture, and things can still change after that because the very nature of game development is non-linear, but once we’re confident in a scene then we’ll just hammer on the polish.
Setup
So, starting with a proverbial blank-slate, the first thing I’m gonna do is bring in a character, in this case the player Nathan Drake, and the next thing we always require is what we call an AP reference.
That’s short for action-pack and that allows every required animation in the ‘pack’ to be played relative to this position. This is how we perfectly align our animations in the game world, but importantly allows us to play animations off of moving objects by aligning the AP ref with the position of things like collapsing buildings or moving vehicles.
There I’ve slapped on a pose from our library on the character and have brought in an NDI camera, (short for Naughty Dog Inc), that’s automatically exported with the player-character’s animation.
The way I like to set up my scenes is I create a pivot on the character so I can rotate the camera around and it behaves a lot like it would if you were rotating with gameplay camera via the PS4’s dualshock thumbstick.
Now I don’t start every scene from scratch, I have a bunch of prefabs with various combinations of characters and sometimes vehicles like the jeep, so here’s Nate’s wife Elena in this scene with him.
I’m gonna drop in what we call the Designer Blockmesh. Now this was still early on in the game so the designers were still building out the levels. There I created a Master Mover that can move the entire scene over to the correct location. This allows us to easily move everything in a scene because this happens a lot during production as we’re iterating on the level layouts.
Props
So the scene I’m gonna use as an example is simply going through a doorway, and in order to do that I need a door! Now we have a full team that will build props for us but for something simple like this I’ll use my fantastic animator modelling skills to build a door. As long as I give it a texture so it shows in-game and move the pivots to the correct place we have a tool that will automatically build it along with its collision and put it in the game with animatable bones etc.
Previz
Now the fun part – actually pre-vizing the actions. Now we spent a long time at the start of the project figuring out how much animation is too much, because we don’t want to waste time on it here – there’s still a high chance the scene will be cut. But we also don’t want to just have T-posed characters passing through a door as we need to playtest this and that requires players to understand what’s going on in the game.
Here I’m mostly just pasting poses, using the time-slider there at the bottom of the maya scene to quickly adjust timing. I even mirror Drake’s animation across to Elena – just any trick that we can use to save time and get this playable in the game as quickly as possible.
Finishing off there with a camera pass there. And now I’ve set it up for export into the game. Again at the time-slider in the bottom you see I’ve chopped it up into an entrance into an idle, door bashes each with an idle in case the player stops mid-way through the action by not hitting the action button, and then a success at the end.
Mocap
For mocap, we’re gonna start prepping before we even arrive at the stage, so here we are now further on, we’ve got a more finished environment and I’m bringing in a replica of one of our two mocap stages we have down the road at the Sony Santa Monica studio.
This allows me to align our stage in the world and I can bring up our full inventory of building props we have down at the stage. Having this lets me to pre-plan the prop build at the stage for each individual scene.
Now here I don’t care too much about the actual doors themselves, it’s more about where the doors are placed. I’ll be able to export this scene and bring it back up again when we’re at the mocap stage for us to measure everything out.
Here we are shooting that scene and we ended up using human doors – that’s because the team at the stage know the props best, and always have the best idea of how to actually build them out. I like to keep it quite loose when planning – I just care about the measurements.
What you’ll notice here is once they go through the door we’ve added on an extra story beat to the door bash whereby Nate & Elena will fall onto a sofa and find a note that delivers some important story information to the player.
That happens often because we don’t want to keep adding IGCs, so we’ll take IGCs that already exist and rework them to get more story info to the player. That way we’re becoming as efficient as possible with the storytelling.
So we’ll get our mocap back on a hard-drive the same day. The next day at work we’ll then run a batch process to retarget it and add it to our animation library. Then I’ll make my selects of the takes I want and will just drop it into the scene.
I’ll use Maya’s Trax editor to sequence it. I’ll make my loops and blends at this point plus any re-adjustments to have it aligned in the world and an updated camera pass to match. This is just another quick pass that allows us to get this into the game because this has to be playable all the time.
This is now a more finished version, still with blocky props, and this will stay in until later in the project when we are confident enough it’s not gonna change at all, and we can begin the polish process.
Polish
At that stage when we’re good to go, there’s going to be a tonne of iteration. This is a snapshot of my version-control here. A reminder for animators to always add notes when checking in changes because often we’re jumping back and forth based on changes.
At this point I’m doing things like polishing up the face after adding facial, polishing the camera motion, adding Depth-of-Field, bringing in final props and animating them. Making sure all the contact points work with the environment – just everything to bring that Naughty Dog level of polish.
So here it is with the final environment. The doors are much better than the prop I made earlier. We’ve added extra props in there to explain why the door needs to be bashed open in the first place. We’ve added the pile of books, (with a physics simulation on it), for visual polish.
I don’t know if you’ve noticed but now Elena ends on the other side of Nate. That was done after the mocap stage because we found that the scene just works better as she walks towards a door on the right.
We’ve added to the scene further. Nate has an upper-body gesture so he’s intently listening. Elena stands and stops in front of the door to prevent the player going through before Elena finishes. Importantly this is all under the player control so we don’t know where the player is throughout this.
If the player chooses not to follow immediately then Elena will wait on him. We have a directional gesture on Elena so she’ll always hand the note to him regardless of where he stands.
So that’s an example of a relatively simple scene that gets more complex over time – and illustrates our process for building these scenes. I had 120 scenes of various complexity on Uncharted 4 and there were around 1000 in the game overall.
Exploration
One part of the job that’s even more enjoyable than this is the coming up with ideas for each sequence long before we even figure out how we’re gonna do it. This is using the same animation style of quick pose-to-pose. It’s all exploratory previs of what COULD be awesome.
How CAN we put the player in these situations. Here I was trying a riff on the classic Raiders of The Lost Ark moving under a truck during the chase sequence. What are the elements that might push the player into that situation, and once there, what are some of the extra awesome moves that you could perform.
We ended up not going for this because we didn’t want to give the player the impression they could drive the truck so the drivers remained inaccessible. But some things do make it like jumping into the jeep so there’s always something you can take out of these exploratory exercises. I’ll show quite a bit more of that later.
Examples
So that was the full start-to-finish process of a complete IGC. Next I’ll go through some of the examples of tricks that we use in order to get them working in game.
But why we even bother making all these scenes unique? You’ll notice that if you play any Naughty Dog game there are lots of one-off moments..
How we align animations in the world and play actions on moving objects such as vehicles, as well as creating the gestures I mentioned earlier that were playing on Nate & Elena.
And some common techniques that we apply to animations such as playing them back and forwards as well as blending between them based on the player’s input.
And last of all, the process for Naughty Dog’s trademark seamless transitions in and out of IGCs and cinematics.
Unique Scenes
So first off, why do we keep keep creating unique scenes where other games copy-and-paste and often take a more systemic approach. Essentially why do we make it so difficult for ourselves?
Well I learned this the hard way. When I first came to Naughty Dog, as I mentioned I mostly came from an in-game/gameplay animation background and was given the task of creating the buddy-boosts.
This is any time you boost a friend or they boost you up to a previously unreachable ledge. We use this to gate the player in combat arenas, bring them together or separate them sometimes if the boost ledge breaks beneath them.
You’ll see here that my initial pass was to create all the gameplay animations I expected it to require. Entering from different directions, different speeds, handling varying ledge heights to build a system that we could use anywhere.
However, I was warned that that’s not the way Naughty Dog works. They had tried it before and I learned why. The reason being, and we ended up with over 20 unique scenes for buddy boosts, is because you can see already we’re using a variety of character combinations.
We also have completely unique backgrounds and we have a level of polish where we want all the hands and fingers to perfectly touch everything they grab so we can’t simply re-use the same animations.
We also have different props we jump up to. We standardised the language of boosting up to ladders, but there are different styles of ladder that have to make sense in each level.
Sometimes the characters are even carrying props, and they’re often talking to one another. As I said earlier, every time we have an IGC it’s an opportunity to deliver story to the player. We’ll even use boosts as transitions into cutscenes.
All these mean we can’t just use one animation repeatedly, and why it’s actually easier to maintain on our end if we just have 20 unique scenes to work on rather than trying to standardise backgrounds, props and characters across the game.
Alignment
Alignment in the world is something required for every scene, and I showed earlier as one of the first things we include in each scene setup, is our AP ref – again short for an Action Pack of animations. In this complex example I’ve got three in the scene (Uncharted 4 was the first Naughty Dog game to allow multiple in each scene – enabling us to up the complexity).
These locators essentially allow each animation we play in the game to play relative to one of these positions. In this scene there are three – a static one at the top of the hill, one on the jeep, and one attached to the player character.
Both the falling rock and the jeep are playing relative to the AP on top of the hill there because that part of the IGC always plays out exactly the same. However when we go back into gameplay we don’t know where the jeep is because it’s under player control.
As such all the character’s animations play relative to the AP on the jeep. That means they get exported as if that position in Maya is the origin, and are then played back relative to wherever that point on the jeep is.
Lastly we have one that moves with the player character because we have an animated camera here that’s still playing as we start climbing out. We’re giving control back to the player and we don’t know where they’re gonna climb to as we’re blending out of that animation. Therefore we play the final camera move relative to the player character wherever they are.
This is the technique we use all the time on order to have our characters placed in the world correctly, and attached to moving objects.
Gestures
For gestures, here’s a relatively simple scene where we’re revealing a volcano. We’re taking control of the camera but not the jeep, (although we are limiting the speed). So the jeep can still be driven by the player and rotated and turned around.
For the action on Nate’s brother Sam as he stands up in the back we set it up as a directional gesture by animating the same action three times to cover three directions, and blending between them based on the jeep’s orientation relative to the volcano.
We have other simpler kinds of gestures on the other two, so their buddy Sully sitting in the front passenger seat has a full-body gesture that will take over whatever the gameplay has him doing at that time.
Nate however, has a point, but we still want him to be driving while pointing so that’s what we call an additive partial animation. He has additive animation on his spine so he leans forward, retaining the underlying spine movement, and a partial on his head and left arm only so the point and head gesture take over 100%.
This leaves his right arm to continue playing the underlying driving anims such as steering and gearshifts. We break the body up into multiple partial sets so we can get really granular with how we break up and combine our animations, and thee are set at export.
So those were three different examples covering the most common types of gestures we use.
Scrubbing & Blending
Something we’ll incorporate a lot is scrubbing animations forward and backward, most often when the player must use effort to lift up an obstacle to get under, and the obstacle can lower back down if the player stops hitting the action button.
Here we’re using this approach for what we call a squeeze-through. As the player pushes forward on a stick we’re playing an animation forwards with a speed up then slow down to stop when they let go of the stick.
But it’s not as simple as that. We’ll also have an IK track on the hands that run along the wall that we’ll turn on if the player stops so the hands rest on the wall. When stopped we’ll also layer an additive breathing animation on the player so while the squeeze-forward animation has come to a stop, the characters still appear alive.
And last of all, via script here we offset the animations of the player and Sully following behind so Sully appears to be reacting to the player’s movements. It helps to make him that little bit more convincing as they both move through the gap.
Using that technique, here’s an example of a complex prop scene. We don’t always take control of the camera. Sometimes IGCs are just environmental such as collapsing buildings where we’re primarily animating a prop.
Here’s a basic bridge prop I made with a spline rig inside it, and what we’re doing here is scrubbing the animation forward as we drive across. If the player turns around and drives back across we’ll scrub it backwards. Essentially the frame of the animation is dependant on the position of the jeep on the bridge.
But like Sam’s jeep-standing gesture before, we set it up so that we have we have three animations, (a centre, left and right version), and we’ll play all three in sync but blend the animation displayed on the bridge depending on whether the player drives to the left or the right edge of the bridge so it appears to tilt.
I said earlier we have a full team that works exclusively on the props, and they’ll take something like that from me and make it much prettier. Here we’ve got physics on every single plank that bend under the weight of the jeep. Some loose planks will fall for effect – it’s all about conveying to the player that it’s a very rickety old bridge.
Transitions
Now one thing we’re big on at Naughty Dog is smooth transitions in and out of these scenes. One of the key tricks we use to achieve this is to record animation playing in the game and bring them back into maya to get our in-and-out points.
Here you can see in my edited version I can push the camera in and create a scene from climbing over the wall, knowing the exact motion and camera position that action creates. This can be used for revealing what’s beyond the wall in a dramatic fashion.
I can just export that section as an IGC, and I’ll have the before and after poses for a seamless transition into and out of gameplay, or more often just keep the start and end poses and insert something entirely different in the scene – as long as it starts from that motion it’ll be seamless.
Here’s an example of how we keep smooth transitions while offering player choice. In interacting with this tree here we don’t want the player to have to move to the one entry point we’ve designated with a single AP to start our scene.
We found in playtesting that players would often run past the tree then double back and approach from the other side, so we simply placed multiple AP-Refs around the tree so the same scene could be played from several angles, with the scene just picking the closest one to the player’s current position. As long as we made sure that the hand always touched it worked.
This was also our first scene to prototype a new technique we used for smoother camera transitions. Notice that as the player gets closer to the interact point we’re pushing the camera in closer into a pre-set camera that matches the start of the IGC.
Then we play the scene with a fully-animated camera and come back out to another pre-set gameplay camera that as we walk away from the interact point we pull out to the regular gameplay camera. This avoids unsightly fast camera movements in and out as we interact with objects in the world.
It overcomes the conundrum of cameras needing to be far and wide for gameplay purposes but requiring to be close when we wish to show specific detail in our IGCs.
We ended up with around 30 different gameplay camera pre-sets that allowed us to better transition in and out of scenes. We would also use these during regular gameplay to affect certain moods so we had some loose cinematography complementing the gameplay.
Because for the first time in the Uncharted series we had real-time cinematics we could also do these kinds of transitions directly into the full cinematic system (which essentially turns off many gameplay systems so we can increase the quality of lighting as we go really close on the characters).
Here’s an example where I’d go back to mocap after the cinematic has been shot and recapture the start with a stuntman to get the hard land on the ground from the rope, and in maya blend it into the full cinematic we’re seeing here.
We also had a cool trick here where the player could aim for the left or right of the ledge and we’d slide the AP-Ref of the cinematic to match the landing position. We’d then play just the first shot of the cinematic from that variable position, going back to the pre-authored position for the rest of the cinematic on the first camera cut.
So while we try to cut into IGCs as little as possible, instead preferring seamless transitions, we are OK with doing it on a player-initiated action. Here we’re jumping down into the bushes and we know the player has to do that to trigger this scene so we can cut in.
Other examples we use are vaulting over objects or mantling over a wall, (like the example I used to illustrate recording from the game). As long as the player chose to do the action it’s ok to cut the camera into a scene. It avoids the unwelcome surprise of wrenching control away from the player unexpectedly.
E3 Demo
So last of all I’m gonna give an in-depth look at the creation of the City Chase sequence, which was the closing act of Sony’s 2015 E3 press conference. This was the largest contiguous action sequence in Uncharted 4. We chose to tackle this complex sequence first to help figure out how much work was gonna be left on the project.
I’ll start with the initial concept, then onto the exploratory previs like the example I showed earlier. Next up are the many prototypes we built for this sequence because it was so technically demanding.
Then I’ll focus on the polish of the finale bike-chase as that’s the section I took to final quality, covering some specific simulation examples that were used to finish this up. The whole section from start to finish comprised about 3 months work, with the final polish phase being just the last two weeks before the deadline.
Concept
So when I arrived on the project we only had a high-level direction for this sequence, and it was simply described as “The best chase sequence we’ve ever had in an Uncharted game”. We only had these two images and basically everything else was up for grabs.
Let’s start by looking at some highlights of how this sequence ended up playing out.
Previz
At Naughty Dog we don’t have producers so instead we have to get out of our seats and speak to other team-members and make our own work. I took it upon myself to start pre-vizing some ideas for this sequence after getting excited when speaking with the designers initially tasked with its creation.
So starting with just coming up with ideas, we knew we wanted to be hanging from a rope at some point so asking “what are the fun things that can happen with that?”
This is all very early so months before we even started making the demo. We found it isn’t that much fun to have to dodge cars but it is a lot of fun to be smashing through things.
Here again I’m recording actions from the game so that allows me to go into the animation and insert IGC ideas I might have. This grapple IGC could have been awesome but we didn’t go for it because we didn’t want to lose the player’s grapple hook.
Just like we didn’t want to give you the impression you could take over the trucks. While that would be awesome, we always need to balance it with what makes best sense for the story.
Here’s something that did make it in pretty much as-is, except we replaced the jeep sinking into the ground with the jeep being on fire – this gives the player that added time-pressure.
Finishing up with a chase on a bike. We knew we wanted a Terminator 2-style nemesis truck that was just really hard to escape.
Looking further into what things you can do on a bike but can’t do on a jeep or truck, we kept coming back to the idea of going through small spaces and having the truck smash through behind you.
Here I am spending more time and digging deeper into that idea, asking what could the cameras look like and still work for gameplay, how can we get that feeling of aggression from the truck – like a bull in a china shop it’s just destroying everything to get at you.
And last of all how do we eventually defeat it? We arrived at sliding under something – again, something a bike can fit under but the truck can’t. We dialed this one back a little because stylistically Uncharted doesn’t do slow-motion, but you can see where we’re going with this.
I ended up with over 10 minutes of pre-vis to look at when we did sit down to discuss what was going to go into the E3 demo so it was really helpful to nail some of those story and action beats for the sequence.
Prototypes
The next stage is to begin making prototypes in the engine. Here’s a prototype of crowds, that are not real game characters but instead simple props I made like the door I made earlier, that look like people. That keeps them cheap so we can have many.
Here’s a test with physics simulation on the bike – ended up not being usable because it just doesn’t look that good up close. But we DID end up putting it on top of the entire climb system.
Here’s an early test of what the path might be for the bike chase at the end. Both the bike and truck are on splines here – the same system as we drive the trucks that we climb over during the rest of the chase sequence. But again when the camera is that close it just doesn’t hold up.
So a couple of weeks before our deadline we decided to record the path back into maya and just keyframe the entire sequence, splitting the work between myself and another animator. He finished up the bike and rider while I retained the truck and everything it smashed into, as well as gameplay elements like the camera and weapon aiming and reloading etc.
Final
So here is what that eventually ended up as once it’s had all the polish. We’ve essentially got 1 minute of streaming animation that plays for this sequence, with two tracks on the player character, one aiming and one not, that we blend between whenever you’re shooting, and an additive pose so the player can aim his arm with the camera.
There’s a tonne of animated destruction. The small items are calculated in real-time while the larger objects are animated.
We have a 30 degree cone in which the player can aim the camera throughout except for points where we script that angle down to zero to push you through narrow objects before opening it up again.
In this final sprint we’re pulling the camera further out and zooming it in to flatten the perspective to make the truck appear closer and right behind you, and finishing with a seamless transition into a full cinematic at the end as the camera pans round.
Simulation
So a little on simulation now that you’re seen the final result. I used a maya simulation plugin throughout the entire project and I would use it for something like this as it’s actually quite hard to have an object land convincingly after flipping in the air.
I treat it just like mocap in that I’m not trying to get it exactly the way I want it – I just need it to get as close as it can then keyframe it on top to add more weight and have it land in the desired pose for the camera.
So here is my final keyframed version of the flip and land. When running the simulation I’m not actually having it hit the objects in the scene there – it’s very difficult to art-direct simulation as it’s so unpredictable.
Instead, I’m just using cubes to throw the truck up in the air, and you can see one there knocking the cargo containers over, with an invisible one pushing them out from below – taking the place of the flatbed you slide under.
Fail-states
Ultimately, we really want to keep scenes like this very simple gameplay-wise. We don’t want the player to die and ruin the flow and excitement of the beat, and have to start over again. As such that sequence has a simple gameplay of shoot the truck enough and you’ll succeed.
But it’s very important to include fail-states for your action scenes. We don’t want the player to see them but it does cover the instance of that one player who’s actively trying to break the scene – usually another game developer.
Contact | Further Reading
So that’s how we make interactive cinematics at Naughty Dog. If you’re interested in game animation I keep a website gameanim.com where I share anything and everything I find online about the subject, and it usually passes through my twitter as well.
If you want to know more about my personal philosophies on interactivity and cinematography in video games I gave a Montreal International Game Summit talk in 2009 entitled Cinematics Sans Cutscenes that you can search for on the site. Essentially it’s about implementing cinematography in games while still keeping things in gameplay without cutscenes.
And last of all if you’re interested more in the design side of the philosophies on interactive cinematics at Naughty Dog, the directors of Uncharted 4 and The Last of Us Neil and Bruce gave a great talk at GDC in 2010 called Creating The Active Cinematic Experience of Uncharted 2.
We still use all of those ideas and the things they discovered during that project on all our games going ahead. I hope this presentation gave you an insight into how and why we create interactive cinematics at Naughty Dog, and wish to see more of this from story-based games in the future.