This is an ‘HD Remaster’ of a talk I gave at the Twitch Animation Exchange in February 2017.
Find a complete transcript of the talk below.
Interactive Cinematics (IGCs)
So what are Interactive Cinematics?
Well basically theyāre any time weāre in a unique moment in the game and itās neither full gameplay nor is it a full cutscene, but thereās something for the player to do.
We refer to these internally as In-Game Cinematics (IGCs for short) although thatās something of a misnomer because, while PS3-era Naughty Dog cinematics were all pre-rendered, all cinematics from Uncharted 4 onwards are real-time in-game so the distinction is now whether they are interactive or not.
So today weāre talking about interactive cinematics, but to keep with habit Iām still going to refer to them as IGCs as thatās how theyāre referred to in the studio.
Introduction
A little bit about my background, Iāve worked on several big games typically heavy on story and Iāve worked in different roles; cinematics or in-game, and usually theyāre quite separate.
Most games have either distinct gameplay or cutscenes, but Uncharted 4 was the first time I had the opportunity to blend the two as thatās something that Naughty Dog really pushes. It was a lot of fun – essentially today Iām just gonna talk to you about how fun my job is.
Currently Iām creating these kinds of interactive scenes for The Last of Us: Part 2 so just bear in mind that Uncharted 4, Uncharted The Lost Legacy and The Last of Us: Part 2 all share a similar process and philosophies.
Overview
So the meat of this presentation is split into 3 parts. First Iām gonna go over my process. The way I typically set up a scene and take it all the way through to final.
Next Iām going to show a selection of examples that illustrate the various technical tricks and techniques that are used commonly throughout the games.
And finally Iām gonna finish up with a deep-dive on Uncharted 4ās Madagascar chase sequence that also doubled as our E3 2015 demo. That was one of the most complex scenes to work on and it really used every trick we have.
Process
So first up, Process. Every scene has to be set up in Maya correctly, then weāre going to bring in props.
Iām going to take you through the previz stage – thatās a lengthy process where weāre figuring out the issues with design via lots of iteration.
Once weāre done there we go to motion-capture, and things can still change after that because the very nature of game development is non-linear, but once weāre confident in a scene then weāll just hammer on the polish.
Setup
So, starting with a proverbial blank-slate, the first thing Iām gonna do is bring in a character, in this case the player Nathan Drake, and the next thing we always require is what we call an AP reference.
Thatās short for action-pack and that allows every required animation in the āpackā to be played relative to this position. This is how we perfectly align our animations in the game world, but importantly allows us to play animations off of moving objects by aligning the AP ref with the position of things like collapsing buildings or moving vehicles.
There Iāve slapped on a pose from our library on the character and have brought in an NDI camera, (short for Naughty Dog Inc), thatās automatically exported with the player-characterās animation.
The way I like to set up my scenes is I create a pivot on the character so I can rotate the camera around and it behaves a lot like it would if you were rotating with gameplay camera via the PS4ās dualshock thumbstick.
Now I donāt start every scene from scratch, I have a bunch of prefabs with various combinations of characters and sometimes vehicles like the jeep, so hereās Nateās wife Elena in this scene with him.
Iām gonna drop in what we call the Designer Blockmesh. Now this was still early on in the game so the designers were still building out the levels. There I created a Master Mover that can move the entire scene over to the correct location. This allows us to easily move everything in a scene because this happens a lot during production as weāre iterating on the level layouts.
Props
So the scene Iām gonna use as an example is simply going through a doorway, and in order to do that I need a door! Now we have a full team that will build props for us but for something simple like this Iāll use my fantastic animator modelling skills to build a door. As long as I give it a texture so it shows in-game and move the pivots to the correct place we have a tool that will automatically build it along with its collision and put it in the game with animatable bones etc.
Previz
Now the fun part – actually pre-vizing the actions. Now we spent a long time at the start of the project figuring out how much animation is too much, because we donāt want to waste time on it here – thereās still a high chance the scene will be cut. But we also donāt want to just have T-posed characters passing through a door as we need to playtest this and that requires players to understand whatās going on in the game.
Here Iām mostly just pasting poses, using the time-slider there at the bottom of the maya scene to quickly adjust timing. I even mirror Drakeās animation across to Elena – just any trick that we can use to save time and get this playable in the game as quickly as possible.
Finishing off there with a camera pass there. And now Iāve set it up for export into the game. Again at the time-slider in the bottom you see Iāve chopped it up into an entrance into an idle, door bashes each with an idle in case the player stops mid-way through the action by not hitting the action button, and then a success at the end.
Mocap
For mocap, weāre gonna start prepping before we even arrive at the stage, so here we are now further on, weāve got a more finished environment and Iām bringing in a replica of one of our two mocap stages we have down the road at the Sony Santa Monica studio.
This allows me to align our stage in the world and I can bring up our full inventory of building props we have down at the stage. Having this lets me to pre-plan the prop build at the stage for each individual scene.
Now here I donāt care too much about the actual doors themselves, itās more about where the doors are placed. Iāll be able to export this scene and bring it back up again when weāre at the mocap stage for us to measure everything out.
Here we are shooting that scene and we ended up using human doors – thatās because the team at the stage know the props best, and always have the best idea of how to actually build them out. I like to keep it quite loose when planning – I just care about the measurements.
What youāll notice here is once they go through the door weāve added on an extra story beat to the door bash whereby Nate & Elena will fall onto a sofa and find a note that delivers some important story information to the player.
That happens often because we donāt want to keep adding IGCs, so weāll take IGCs that already exist and rework them to get more story info to the player. That way weāre becoming as efficient as possible with the storytelling.
So weāll get our mocap back on a hard-drive the same day. The next day at work weāll then run a batch process to retarget it and add it to our animation library. Then Iāll make my selects of the takes I want and will just drop it into the scene.
Iāll use Mayaās Trax editor to sequence it. Iāll make my loops and blends at this point plus any re-adjustments to have it aligned in the world and an updated camera pass to match. This is just another quick pass that allows us to get this into the game because this has to be playable all the time.
This is now a more finished version, still with blocky props, and this will stay in until later in the project when we are confident enough itās not gonna change at all, and we can begin the polish process.
Polish
At that stage when weāre good to go, thereās going to be a tonne of iteration. This is a snapshot of my version-control here. A reminder for animators to always add notes when checking in changes because often weāre jumping back and forth based on changes.
At this point Iām doing things like polishing up the face after adding facial, polishing the camera motion, adding Depth-of-Field, bringing in final props and animating them. Making sure all the contact points work with the environment – just everything to bring that Naughty Dog level of polish.
So here it is with the final environment. The doors are much better than the prop I made earlier. Weāve added extra props in there to explain why the door needs to be bashed open in the first place. Weāve added the pile of books, (with a physics simulation on it), for visual polish.
I donāt know if youāve noticed but now Elena ends on the other side of Nate. That was done after the mocap stage because we found that the scene just works better as she walks towards a door on the right.
Weāve added to the scene further. Nate has an upper-body gesture so heās intently listening. Elena stands and stops in front of the door to prevent the player going through before Elena finishes. Importantly this is all under the player control so we donāt know where the player is throughout this.
If the player chooses not to follow immediately then Elena will wait on him. We have a directional gesture on Elena so sheāll always hand the note to him regardless of where he stands.
So thatās an example of a relatively simple scene that gets more complex over time – and illustrates our process for building these scenes. I had 120 scenes of various complexity on Uncharted 4 and there were around 1000 in the game overall.
Exploration
One part of the job thatās even more enjoyable than this is the coming up with ideas for each sequence long before we even figure out how weāre gonna do it. This is using the same animation style of quick pose-to-pose. Itās all exploratory previs of what COULD be awesome.
How CAN we put the player in these situations. Here I was trying a riff on the classic Raiders of The Lost Ark moving under a truck during the chase sequence. What are the elements that might push the player into that situation, and once there, what are some of the extra awesome moves that you could perform.
We ended up not going for this because we didnāt want to give the player the impression they could drive the truck so the drivers remained inaccessible. But some things do make it like jumping into the jeep so thereās always something you can take out of these exploratory exercises. Iāll show quite a bit more of that later.
Examples
So that was the full start-to-finish process of a complete IGC. Next Iāll go through some of the examples of tricks that we use in order to get them working in game.
But why we even bother making all these scenes unique? Youāll notice that if you play any Naughty Dog game there are lots of one-off moments..
How we align animations in the world and play actions on moving objects such as vehicles, as well as creating the gestures I mentioned earlier that were playing on Nate & Elena.
And some common techniques that we apply to animations such as playing them back and forwards as well as blending between them based on the playerās input.
And last of all, the process for Naughty Dogās trademark seamless transitions in and out of IGCs and cinematics.
Unique Scenes
So first off, why do we keep keep creating unique scenes where other games copy-and-paste and often take a more systemic approach. Essentially why do we make it so difficult for ourselves?
Well I learned this the hard way. When I first came to Naughty Dog, as I mentioned I mostly came from an in-game/gameplay animation background and was given the task of creating the buddy-boosts.
This is any time you boost a friend or they boost you up to a previously unreachable ledge. We use this to gate the player in combat arenas, bring them together or separate them sometimes if the boost ledge breaks beneath them.
Youāll see here that my initial pass was to create all the gameplay animations I expected it to require. Entering from different directions, different speeds, handling varying ledge heights to build a system that we could use anywhere.
However, I was warned that thatās not the way Naughty Dog works. They had tried it before and I learned why. The reason being, and we ended up with over 20 unique scenes for buddy boosts, is because you can see already weāre using a variety of character combinations.
We also have completely unique backgrounds and we have a level of polish where we want all the hands and fingers to perfectly touch everything they grab so we canāt simply re-use the same animations.
We also have different props we jump up to. We standardised the language of boosting up to ladders, but there are different styles of ladder that have to make sense in each level.
Sometimes the characters are even carrying props, and theyāre often talking to one another. As I said earlier, every time we have an IGC itās an opportunity to deliver story to the player. Weāll even use boosts as transitions into cutscenes.
All these mean we canāt just use one animation repeatedly, and why itās actually easier to maintain on our end if we just have 20 unique scenes to work on rather than trying to standardise backgrounds, props and characters across the game.
Alignment
Alignment in the world is something required for every scene, and I showed earlier as one of the first things we include in each scene setup, is our AP ref – again short for an Action Pack of animations. In this complex example Iāve got three in the scene (Uncharted 4 was the first Naughty Dog game to allow multiple in each scene – enabling us to up the complexity).
These locators essentially allow each animation we play in the game to play relative to one of these positions. In this scene there are three – a static one at the top of the hill, one on the jeep, and one attached to the player character.
Both the falling rock and the jeep are playing relative to the AP on top of the hill there because that part of the IGC always plays out exactly the same. However when we go back into gameplay we donāt know where the jeep is because itās under player control.
As such all the characterās animations play relative to the AP on the jeep. That means they get exported as if that position in Maya is the origin, and are then played back relative to wherever that point on the jeep is.
Lastly we have one that moves with the player character because we have an animated camera here thatās still playing as we start climbing out. Weāre giving control back to the player and we donāt know where theyāre gonna climb to as weāre blending out of that animation. Therefore we play the final camera move relative to the player character wherever they are.
This is the technique we use all the time on order to have our characters placed in the world correctly, and attached to moving objects.
Gestures
For gestures, hereās a relatively simple scene where weāre revealing a volcano. Weāre taking control of the camera but not the jeep, (although we are limiting the speed). So the jeep can still be driven by the player and rotated and turned around.
For the action on Nateās brother Sam as he stands up in the back we set it up as a directional gesture by animating the same action three times to cover three directions, and blending between them based on the jeepās orientation relative to the volcano.
We have other simpler kinds of gestures on the other two, so their buddy Sully sitting in the front passenger seat has a full-body gesture that will take over whatever the gameplay has him doing at that time.
Nate however, has a point, but we still want him to be driving while pointing so thatās what we call an additive partial animation. He has additive animation on his spine so he leans forward, retaining the underlying spine movement, and a partial on his head and left arm only so the point and head gesture take over 100%.
This leaves his right arm to continue playing the underlying driving anims such as steering and gearshifts. We break the body up into multiple partial sets so we can get really granular with how we break up and combine our animations, and thee are set at export.
So those were three different examples covering the most common types of gestures we use.
Scrubbing & Blending
Something weāll incorporate a lot is scrubbing animations forward and backward, most often when the player must use effort to lift up an obstacle to get under, and the obstacle can lower back down if the player stops hitting the action button.
Here weāre using this approach for what we call a squeeze-through. As the player pushes forward on a stick weāre playing an animation forwards with a speed up then slow down to stop when they let go of the stick.
But itās not as simple as that. Weāll also have an IK track on the hands that run along the wall that weāll turn on if the player stops so the hands rest on the wall. When stopped weāll also layer an additive breathing animation on the player so while the squeeze-forward animation has come to a stop, the characters still appear alive.
And last of all, via script here we offset the animations of the player and Sully following behind so Sully appears to be reacting to the playerās movements. It helps to make him that little bit more convincing as they both move through the gap.
Using that technique, hereās an example of a complex prop scene. We donāt always take control of the camera. Sometimes IGCs are just environmental such as collapsing buildings where weāre primarily animating a prop.
Hereās a basic bridge prop I made with a spline rig inside it, and what weāre doing here is scrubbing the animation forward as we drive across. If the player turns around and drives back across weāll scrub it backwards. Essentially the frame of the animation is dependant on the position of the jeep on the bridge.
But like Samās jeep-standing gesture before, we set it up so that we have we have three animations, (a centre, left and right version), and weāll play all three in sync but blend the animation displayed on the bridge depending on whether the player drives to the left or the right edge of the bridge so it appears to tilt.
I said earlier we have a full team that works exclusively on the props, and theyāll take something like that from me and make it much prettier. Here weāve got physics on every single plank that bend under the weight of the jeep. Some loose planks will fall for effect – itās all about conveying to the player that itās a very rickety old bridge.
Transitions
Now one thing weāre big on at Naughty Dog is smooth transitions in and out of these scenes. One of the key tricks we use to achieve this is to record animation playing in the game and bring them back into maya to get our in-and-out points.
Here you can see in my edited version I can push the camera in and create a scene from climbing over the wall, knowing the exact motion and camera position that action creates. This can be used for revealing whatās beyond the wall in a dramatic fashion.
I can just export that section as an IGC, and Iāll have the before and after poses for a seamless transition into and out of gameplay, or more often just keep the start and end poses and insert something entirely different in the scene – as long as it starts from that motion itāll be seamless.
Hereās an example of how we keep smooth transitions while offering player choice. In interacting with this tree here we donāt want the player to have to move to the one entry point weāve designated with a single AP to start our scene.
We found in playtesting that players would often run past the tree then double back and approach from the other side, so we simply placed multiple AP-Refs around the tree so the same scene could be played from several angles, with the scene just picking the closest one to the playerās current position. As long as we made sure that the hand always touched it worked.
This was also our first scene to prototype a new technique we used for smoother camera transitions. Notice that as the player gets closer to the interact point weāre pushing the camera in closer into a pre-set camera that matches the start of the IGC.
Then we play the scene with a fully-animated camera and come back out to another pre-set gameplay camera that as we walk away from the interact point we pull out to the regular gameplay camera. This avoids unsightly fast camera movements in and out as we interact with objects in the world.
It overcomes the conundrum of cameras needing to be far and wide for gameplay purposes but requiring to be close when we wish to show specific detail in our IGCs.
We ended up with around 30 different gameplay camera pre-sets that allowed us to better transition in and out of scenes. We would also use these during regular gameplay to affect certain moods so we had some loose cinematography complementing the gameplay.
Because for the first time in the Uncharted series we had real-time cinematics we could also do these kinds of transitions directly into the full cinematic system (which essentially turns off many gameplay systems so we can increase the quality of lighting as we go really close on the characters).
Hereās an example where Iād go back to mocap after the cinematic has been shot and recapture the start with a stuntman to get the hard land on the ground from the rope, and in maya blend it into the full cinematic weāre seeing here.
We also had a cool trick here where the player could aim for the left or right of the ledge and weād slide the AP-Ref of the cinematic to match the landing position. Weād then play just the first shot of the cinematic from that variable position, going back to the pre-authored position for the rest of the cinematic on the first camera cut.
So while we try to cut into IGCs as little as possible, instead preferring seamless transitions, we are OK with doing it on a player-initiated action. Here weāre jumping down into the bushes and we know the player has to do that to trigger this scene so we can cut in.
Other examples we use are vaulting over objects or mantling over a wall, (like the example I used to illustrate recording from the game). As long as the player chose to do the action itās ok to cut the camera into a scene. It avoids the unwelcome surprise of wrenching control away from the player unexpectedly.
E3 Demo
So last of all Iām gonna give an in-depth look at the creation of the City Chase sequence, which was the closing act of Sonyās 2015 E3 press conference. This was the largest contiguous action sequence in Uncharted 4. We chose to tackle this complex sequence first to help figure out how much work was gonna be left on the project.
Iāll start with the initial concept, then onto the exploratory previs like the example I showed earlier. Next up are the many prototypes we built for this sequence because it was so technically demanding.
Then Iāll focus on the polish of the finale bike-chase as thatās the section I took to final quality, covering some specific simulation examples that were used to finish this up. The whole section from start to finish comprised about 3 months work, with the final polish phase being just the last two weeks before the deadline.
Concept
So when I arrived on the project we only had a high-level direction for this sequence, and it was simply described as āThe best chase sequence weāve ever had in an Uncharted gameā. We only had these two images and basically everything else was up for grabs.
Letās start by looking at some highlights of how this sequence ended up playing out.
Previz
At Naughty Dog we donāt have producers so instead we have to get out of our seats and speak to other team-members and make our own work. I took it upon myself to start pre-vizing some ideas for this sequence after getting excited when speaking with the designers initially tasked with its creation.
So starting with just coming up with ideas, we knew we wanted to be hanging from a rope at some point so asking āwhat are the fun things that can happen with that?ā
This is all very early so months before we even started making the demo. We found it isnāt that much fun to have to dodge cars but it is a lot of fun to be smashing through things.
Here again Iām recording actions from the game so that allows me to go into the animation and insert IGC ideas I might have. This grapple IGC could have been awesome but we didnāt go for it because we didnāt want to lose the playerās grapple hook.
Just like we didnāt want to give you the impression you could take over the trucks. While that would be awesome, we always need to balance it with what makes best sense for the story.
Hereās something that did make it in pretty much as-is, except we replaced the jeep sinking into the ground with the jeep being on fire – this gives the player that added time-pressure.
Finishing up with a chase on a bike. We knew we wanted a Terminator 2-style nemesis truck that was just really hard to escape.
Looking further into what things you can do on a bike but canāt do on a jeep or truck, we kept coming back to the idea of going through small spaces and having the truck smash through behind you.
Here I am spending more time and digging deeper into that idea, asking what could the cameras look like and still work for gameplay, how can we get that feeling of aggression from the truck – like a bull in a china shop itās just destroying everything to get at you.
And last of all how do we eventually defeat it? We arrived at sliding under something – again, something a bike can fit under but the truck canāt. We dialed this one back a little because stylistically Uncharted doesnāt do slow-motion, but you can see where weāre going with this.
I ended up with over 10 minutes of pre-vis to look at when we did sit down to discuss what was going to go into the E3 demo so it was really helpful to nail some of those story and action beats for the sequence.
Prototypes
The next stage is to begin making prototypes in the engine. Hereās a prototype of crowds, that are not real game characters but instead simple props I made like the door I made earlier, that look like people. That keeps them cheap so we can have many.
Hereās a test with physics simulation on the bike – ended up not being usable because it just doesnāt look that good up close. But we DID end up putting it on top of the entire climb system.
Hereās an early test of what the path might be for the bike chase at the end. Both the bike and truck are on splines here – the same system as we drive the trucks that we climb over during the rest of the chase sequence. But again when the camera is that close it just doesnāt hold up.
So a couple of weeks before our deadline we decided to record the path back into maya and just keyframe the entire sequence, splitting the work between myself and another animator. He finished up the bike and rider while I retained the truck and everything it smashed into, as well as gameplay elements like the camera and weapon aiming and reloading etc.
Final
So here is what that eventually ended up as once itās had all the polish. Weāve essentially got 1 minute of streaming animation that plays for this sequence, with two tracks on the player character, one aiming and one not, that we blend between whenever youāre shooting, and an additive pose so the player can aim his arm with the camera.
Thereās a tonne of animated destruction. The small items are calculated in real-time while the larger objects are animated.
We have a 30 degree cone in which the player can aim the camera throughout except for points where we script that angle down to zero to push you through narrow objects before opening it up again.
In this final sprint weāre pulling the camera further out and zooming it in to flatten the perspective to make the truck appear closer and right behind you, and finishing with a seamless transition into a full cinematic at the end as the camera pans round.
Simulation
So a little on simulation now that youāre seen the final result. I used a maya simulation plugin throughout the entire project and I would use it for something like this as itās actually quite hard to have an object land convincingly after flipping in the air.
I treat it just like mocap in that Iām not trying to get it exactly the way I want it – I just need it to get as close as it can then keyframe it on top to add more weight and have it land in the desired pose for the camera.
So here is my final keyframed version of the flip and land. When running the simulation Iām not actually having it hit the objects in the scene there – itās very difficult to art-direct simulation as itās so unpredictable.
Instead, Iām just using cubes to throw the truck up in the air, and you can see one there knocking the cargo containers over, with an invisible one pushing them out from below – taking the place of the flatbed you slide under.
Fail-states
Ultimately, we really want to keep scenes like this very simple gameplay-wise. We donāt want the player to die and ruin the flow and excitement of the beat, and have to start over again. As such that sequence has a simple gameplay of shoot the truck enough and youāll succeed.
But itās very important to include fail-states for your action scenes. We donāt want the player to see them but it does cover the instance of that one player whoās actively trying to break the scene – usually another game developer.
Contact | Further Reading
So thatās how we make interactive cinematics at Naughty Dog. If youāre interested in game animation I keep a website gameanim.com where I share anything and everything I find online about the subject, and it usually passes through my twitter as well.
If you want to know more about my personal philosophies on interactivity and cinematography in video games I gave a Montreal International Game Summit talk in 2009 entitled Cinematics Sans Cutscenes that you can search for on the site. Essentially itās about implementing cinematography in games while still keeping things in gameplay without cutscenes.
And last of all if youāre interested more in the design side of the philosophies on interactive cinematics at Naughty Dog, the directors of Uncharted 4 and The Last of Us Neil and Bruce gave a great talk at GDC in 2010 called Creating The Active Cinematic Experience of Uncharted 2.
We still use all of those ideas and the things they discovered during that project on all our games going ahead. I hope this presentation gave you an insight into how and why we create interactive cinematics at Naughty Dog, and wish to see more of this from story-based games in the future.