When we initially showed Mass Effect at E3 2006 I recall a handful of Square developers attending to evaluate the facial animation. While it looks like their production methods are somewhat dated due to the long development cycle, playing FFXIII shows the eventual result to be outstanding – presumably due to their dedicated engine for facial closeups and meticulous planning.

The google translation for the piece is especially bad, but so far I can gather:

• Like Square Enix studio-offshoot Feelplus, cutscenes are divided into 4 categories depending on importance. A & B level cutscenes employ full keyframed facial animation (with lip-sync done individually for both Japanese and English versions). C & D level cutscenes just use procedurally-generated facial and lip-sync animation.
• Similarly, the cutscene team is divided into four distinct teams that handle each progressive stage of creating a full cutscene. They are:
  • Motion Capture Group: Shooting and cleaning up motion-capture data.
  • Body Motion Team: Creature keyframe animation, and human motions that cannot be captured.
  • Facial Group: Facial acting and lip-sync.
  • Simulation Team: Hair and cloth simulations.
• Cutscene shoots are meticulously planned ahead of time. Beyond storyboards, clean layout boards that contain descriptions of the actors, props and set layouts required for each scene are created.

• Temporary voice-over was used on the set with full ADR done later. Interestingly, it appears that animatics were projected on the walls during the shoots to give the actors a better sense of their virtual counterparts and surroundings.
• Within Motionbuilder, additional tools were created to easily allow editing and exporting of complex scenes via a check-box matrix of assets vs shots, seen at the lower-right of the image below – something I’ve had in mind for some time as the best solution for working on scenes that require a lengthy full export only very occasionally.

• Character-wise, the highest resolution characters (Lightning and Snow) consist of up to 223 bones, covering the basic skeleton, auxiliary (corrective and simulation), facial and hair bones.
• The keyframed facial animation was done via a traditional slider setup. The image below shows the numerous attributes, which must have become quite unwieldy and therefore given rise to the need for a dedicated team to specialise in this area.

• For simulations, the wind level is initially set for each scene to provide the requisite amount of movement in the simulated assets.
• Around 20 bones are used for hair, employing a spring system that dampens as a character’s inertia increases to prevent crashing with the head.

Some technical insights towards the end, but most interesting of all is the emphasis placed on the human side of the shoot, something that comes through in the finished work. Regarding casting:

“Well, we start with the obvious things – e.g. talent, distinctive voice, whether an actor is appropriate for the role, etcetera. But since we’re looking for people who will be doing both the mocap acting and the voice, we also watch how the candidates physicalize their performance. It’s also important to see how they deal with adjustments and to see if they take direction well. For the top candidates, we’ll actually bring them back for a second audition and have them perform a scene with Nolan (North, who plays Drake) to make sure they have good chemistry.”

First, a little bit of background about where I’m coming from. I’ve been working on games now for nearly a decade in a variety of both in-game and cinematic roles, with the in-game side mostly focussing on player control, cameras and animation-system design relating to gameplay, whereas the cinematic part is about pure art and storytelling, getting information across to the player in as efficient a manner as possible.

At the time of the presentation, our team was wrapping up the final stages of production on cutscenes for Mass Effect 2 and here’s an image of the wall behind my desk.

This was a deliberately blurred picture (so as to not give away any spoilers) of our workload as it stood, with only the last few bugs remaining and each of those lines there representing anywhere between 1 and 7 individual cutscenes, many of which we now know post-release were well-received cinematic experiences. However, with such a heavy history of in-game animation design, I feel that with games like ours and the recently released Uncharted 2 among others, we’re coming to the limits of where we can go with pure cutscenes in our games.

Uncharted succeeded with its seamless transitions into and out of cutscenes, as well as the sheer high quality of the charactes and acting, and ours with the logic behind them that decides which sequences (and how) to play depending on choices. As animators, we didn’t know whether your player-character was male or female, what their face looked like, or who you’d brought along with you on your team, not to mention the myriad choices that you can make throughout the game that will determine the outcome of particularly climactic scenes. After this point I can see cutscenes only limiting our progress in terms of developing storytelling in games, so I’m interested in exploring alternative methods that will allow us to truly develop stories for our medium rather than continually within the pre-established constraints of film – which are essentially where cutscenes lie.

The Problems With Cutscenes

When researching this topic, I began by initially trying to define exactly what a “cutscene” is. I’m sure when we’re all watching one we can tell that it’s a cutscene, but we have had so many variations of them over the years that it’s difficult to define them in a pure sense. The first and most natural idea that springs to mind tends to be “whenever the camera cuts”, but this is not the case as we often have no cuts, particularly during “designer cutscenes” whereby the camera simply transitions to a location to show the relationship between the player and the subject.

Next is to assume that player control is removed entirely, but as we’ll see later there have been many attempts to include some level of interactivity during cutscenes that doesn’t detract from them being cutscenes. The creation method is irrelevant – we have different terms in the industry such as Cinematic Cutscenes, Designer Cutscenes, Ingame and Pre-rendered, but in the end what really matters is how the player perceives the result, of which length is a sizeable factor.

Below is a great example of a cinematic sequence from Resident Evil 4 that follows all the “rules” of what we typically consider to be a cutscene, but because of its short duration the player doesn’t consider this a cutscene, and instead just another cinematic moment that contributes to the overall feeling of the game being a cinematic experience.

Click here to view the embedded video.

Defining The Problem

As such, a better approach is rather than to define what a cutscene is, instead to define the problem, so hopefully we can concentrate on simply attempting to overcome the issue rather than arbitrarily abolishing cutscenes themselves which has been done in the past to greater and lesser degrees of success.

Control and choice is restricted, if not removed entirely.

One second the player is running around, making choices and personally involved in many aspects of how the story or adventure is progressing, then the next they are force-fed exactly what we as developers want them to see and experience. This is fundamentally against the nature of our medium, and greatly restricts their experience for the duration of the cutscene back to the levels of the older medium of film.

Cutscenes make a clear division between the gameplay and the story.

The visual difference between cameras (and often character fidelity) in gameplay and those for cinematic cutscenes negatively reinforces the fact that most of gameplay involves little to no story progression at all.

Cutscenes are opt-out.

Once a cutscene is triggered, often without the player’s consent or choosing, his or her only option is to exit the cutscene. This brings with it issues of missing important information after the point of exit, and more importantly highlights the fact that we have a major problem here. If we have a design element in our medium that consistently requires we add the option to skip it entirely, then surely there is something fundamentally flawed with some aspect of that element?

This brings me to the definition that cutscenes are essentially “Imposed Cinematography”, whereby we as developers wish to impose narrative or progression information required to continue, or to ensure that the player experiences a part of the game exactly as we wish them to. This is the heart of the conflict between developer and player where, unlike other older mediums, we perform best when we play to our strengths and require participation from the player rather than purely imposing our creative will upon them.

Beyond examples of animated gifs like the one included here, you can download Photoshop PSDs of all the character animations in the game from this handy location. Certainly worth a look to get an idea of the animation process behind this wonderfully infuriating little game.

They certainly give the outward impression at least that they’ve attained the developer holy grail of sharing technology across projects, recently announcing early Bots for Team Fortress 2 using the same decision-making as described in this paper. Find a list of all of Valve’s speaker presentations here.

Looks like a simple rig in Maya to compliment the game nicely, and they use morph shapes for the facial emotes which seems a natural fit given the squashiness of the character.

Via the Google translation I see that the game has 25 characters of around 16,000 polygons each, comprising some 5000 animations. The rigging videos are of most interest however, highlighting both their facial & finger sliders and the unique controls for Dhalsim’s squash and stretch limbs. In a break from what I’m used to , the team take a less modular approach to facial expressions, with broad sliders for various facial expressions as opposed to sliders for each area of the face which can afford greater control for the animator but proves more time consuming and being prone to going off-model. This might be a viable approach with such stylised characters however, and they control the following variables:

• DEFAULT_UPPER_PARTS
  • DEFAULT
• DEFAULT_LOWER_PARTS
  • DEFAULT
• SPK_LOWER_PARTS
  • SPK_A_ACCENT
  • SPK_A
  • SPK_E_ACCENT
  • SPK_E
  • SPK_I_ACCENT
  • SPK_I
  • SPK_N_ACCENT
  • SPK_N
  • SPK_O_ACCENT
  • SPK_O
  • SPK_U_ACCENT
  • SPK_U
• BHV_UPPER_PARTS
  • BHV_BLINK
• EMO_UPPER_PARTS
  • EMO_ANGER
  • EMO_ATTACK_ACCENT
  • EMO_ATTACK
  • EMO_AWAKENING
  • EMO_CRY
  • EMO_FEAR
  • EMO_HATE
  • EMO_JOY
  • EMO_SHOUT
  • EMO_SMILE
  • EMO_SORROW
  • EMO_SURPRISE
  • EMO_TIREDNESS
• EMO_LOWER_PARTS
  • EMO_ANGER
  • EMO_ATTACK_ACCENT
  • EMO_ATTACK
  • EMO_AWAKENING
  • EMO_CRY
  • EMO_FEAR
  • EMO_HATE
  • EMO_JOY
  • EMO_SHOUT
  • EMO_SMILE
  • EMO_SORROW
  • EMO_SURPRISE
  • EMO_TIREDNESS

Click here to view the embedded video.

All body animation appears to be rotation-only, given that the limb lengths do not warp and stretch to assume the positions of the bones in the animations. IK does however appear to be solved for each limb as the arms and legs of the shorter characters hyper-extend (stretch out) to meet the required feet and hand positions of the original animations. The only visual artifacts like this occur in the face, showing that position keys must be involved in creating the facial animations.

The camera animations for intros and special moves are bundled with the animation sets, highlighting that both the sound and visual effects of the chosen character remain intact on the character rather than the animation sets and play out on corresponding actions quite well. This illustrates that there must be identically organised sets of each for every character.

[via GiantBomb]

There are instructions as to the various stages of the art creation but unfortunately the text is image-based so I can’t babel it. While there are 5 characters at the time of writing, it looks like it’s going to grow over time so certainly something to check back on.

UPDATE: Thanks to Jason Porath for swiftly providing a translation for the stages of production:

1. Get the design of the character, usually from the art director, or sometimes Rough Design. One character usually takes about 3 days.
2. Make a 3d model of the character. This usually takes 2 weeks per character. You also make the ranges of motion, which takes around 2 months/character.
3. Render out the 3d character. To bake out all the animation for one character usually takes around 2 weeks.
4. Touch up the render, according to art director’s wishes. This usually takes 1 week per character.
5. Add in additional stuff like wrinkles, muscle creases, and the like, while maintaining the form. This takes each character around 6 and a half months (!). This is where all the character’s consistency in form is checked.
6. Adding in gradients. This takes 2 and a half months per character.

Each character usually has around 500 frames of animation, but some are up to 4x that.

Click here to view the embedded video.

Though it looks a little slow in relation to real gameplay balance that might have come later in production, the sheer fuidity of the actions and camera, and the multitude of states (poses) the character can move between is something of a marvel. One can only hope that we’ll see some variation of it in an upcoming game.