Click here to view the embedded video.

While mostly a lighting demo of Infinite Realities‘ scanned head we’ve encountered before, seeing it running in realtime on a PC at work both terrifies and exhilirates me. Once he opens his eyes and begins to move around, it will be our task to breathe life into characters that look like this in immersive worlds that are consistent and do not break the suspension of disbelief.

Since arriving in Canada, I’ve worked exclusively on games with character casts are in the hundreds, requiring procedural and systemic solutions to create their animation. Perhaps this leap in character realisation will instead force us to concentrate on stories with fewer actors that have an unprecented depth of character.

Click here to view the embedded video.

While the visual benefits are obvious, I can’t help but wonder just how much it cost the production for so many actor contracts when not only voices but likenesses are required. Additionally, the production task must be one of the hardest parts of the game, considering that at the time of writing they’re still hiring senior roles for the cinematics team around 6 months before the planned release date.

That said, I’ve really been keeping an eye on this since E3 so can’t wait for spring to roll around, plus I’m a Noir nut since a Humphrey Bogart stint last summer.

Read more about the MotionScan tech and process here.

UPDATE: They’re not still hiring as the front-page job posting on their site is dated 2008. Makes me feel better about the frequency of my own posts.

“This has been the Holy Grail in a lot of respects for video games – the ability to have characters on the screen that you can have an emotional connection with. The medium for the last 25 years has been very visceral, interactive, immersive medium – but it was very hard to have characters to actually have empathy towards or an emotional connection with… or that might make you laugh or make you cry; be some catalyst for an emotional reaction… Call Of Duty: Black Ops is the first game where we’ve been able to perfect the facial animation, mouth movement technology so that the lines that are being delivered are believable. The facial animation looks like a real person”

As to my position on the still theoretical Uncanny Valley, I’m convinced it more than certainly exists in Masahiro Mori’s pure sense as I’ve seen (or to put it correctly, felt) it in films like Beowulf and Midnight Express, though in games we often mistakenly attribute it to a combination of bad rigging and animation – failure not even near the Uncanny Valley.

However, to say that we have still to cross it is a misconception as I strongly believe it has been done many times already in CG films and games, though importantly only for a flash,when everything comes together for a few fleeting frames in a performance to convince viewers they are witnessing a “real” person acting. The real challenge is for a consistent performance that never dips back into the unappealing, which cannot come to games before film simply due to the controlled nature of the latter medium.

A more realistic target for us is to aim for believable, rather than photo-realistic, performances which is a far more worthy goal that still only a few developers have managed to attain – with quality voice-acting and writing only now becoming more common and expected by players. I expect the Black-Ops performances are good due to the high caliber of acting talent of Gary Oldman and Ed Harris, perhaps even believable, but not realistic and not fooling us into believing we see a real person at the end of our gun.

[via CVG]

When we initially showed Mass Effect at E3 2006 I recall a handful of Square developers attending to evaluate the facial animation. While it looks like their production methods are somewhat dated due to the long development cycle, playing FFXIII shows the eventual result to be outstanding – presumably due to their dedicated engine for facial closeups and meticulous planning.

The google translation for the piece is especially bad, but so far I can gather:

Like Square Enix studio-offshoot Feelplus, cutscenes are divided into 4 categories depending on importance. A & B level cutscenes employ full keyframed facial animation (with lip-sync done individually for both Japanese and English versions). C & D level cutscenes just use procedurally-generated facial and lip-sync animation.

Similarly, the cutscene team is divided into four distinct teams that handle each progressive stage of creating a full cutscene. They are:

Motion Capture Group: Shooting and cleaning up motion-capture data.

Body Motion Team: Creature keyframe animation, and human motions that cannot be captured.

Facial Group: Facial acting and lip-sync.

Simulation Team: Hair and cloth simulations.

Cutscene shoots are meticulously planned ahead of time. Beyond storyboards, clean layout boards that contain descriptions of the actors, props and set layouts required for each scene are created.

Temporary voice-over was used on the set with full ADR done later. Interestingly, it appears that animatics were projected on the walls during the shoots to give the actors a better sense of their virtual counterparts and surroundings.

Within Motionbuilder, additional tools were created to easily allow editing and exporting of complex scenes via a check-box matrix of assets vs shots, seen at the lower-right of the image below – something I’ve had in mind for some time as the best solution for working on scenes that require a lengthy full export only very occasionally.

Character-wise, the highest resolution characters (Lightning and Snow) consist of up to 223 bones, covering the basic skeleton, auxiliary (corrective and simulation), facial and hair bones.

The keyframed facial animation was done via a traditional slider setup. The image below shows the numerous attributes, which must have become quite unwieldy and therefore given rise to the need for a dedicated team to specialise in this area.

For simulations, the wind level is initially set for each scene to provide the requisite amount of movement in the simulated assets.

Around 20 bones are used for hair, employing a spring system that dampens as a character’s inertia increases to prevent crashing with the head.

Via the Google translation I see that the game has 25 characters of around 16,000 polygons each, comprising some 5000 animations. The rigging videos are of most interest however, highlighting both their facial & finger sliders and the unique controls for Dhalsim’s squash and stretch limbs. In a break from what I’m used to , the team take a less modular approach to facial expressions, with broad sliders for various facial expressions as opposed to sliders for each area of the face which can afford greater control for the animator but proves more time consuming and being prone to going off-model. This might be a viable approach with such stylised characters however, and they control the following variables:

• DEFAULT_UPPER_PARTS
  • DEFAULT
• DEFAULT_LOWER_PARTS
  • DEFAULT
• SPK_LOWER_PARTS
  • SPK_A_ACCENT
  • SPK_A
  • SPK_E_ACCENT
  • SPK_E
  • SPK_I_ACCENT
  • SPK_I
  • SPK_N_ACCENT
  • SPK_N
  • SPK_O_ACCENT
  • SPK_O
  • SPK_U_ACCENT
  • SPK_U
• BHV_UPPER_PARTS
  • BHV_BLINK
• EMO_UPPER_PARTS
  • EMO_ANGER
  • EMO_ATTACK_ACCENT
  • EMO_ATTACK
  • EMO_AWAKENING
  • EMO_CRY
  • EMO_FEAR
  • EMO_HATE
  • EMO_JOY
  • EMO_SHOUT
  • EMO_SMILE
  • EMO_SORROW
  • EMO_SURPRISE
  • EMO_TIREDNESS
• EMO_LOWER_PARTS
  • EMO_ANGER
  • EMO_ATTACK_ACCENT
  • EMO_ATTACK
  • EMO_AWAKENING
  • EMO_CRY
  • EMO_FEAR
  • EMO_HATE
  • EMO_JOY
  • EMO_SHOUT
  • EMO_SMILE
  • EMO_SORROW
  • EMO_SURPRISE
  • EMO_TIREDNESS

The image to the side shows the skeleton used for main protagonist Snake, revealing the inclusion of deformation bones to maintain volume on the elbows, knees and wrists on top of the now-standard twist bones for the shoulders, hips and wrists. Unidentifiable, however, are the curious bones at the neck – perhaps to aid shoulder deformation or simply to attach weapons to?

Some stats from the article:

• 115 bones in total, comprising:
• 36 in the face.
• 47 in the body.
• 32 in the hands, (3 for each finger, with an additional bone on each hand between the thumb and index finger – presumably to maintain volume).
• 1700 animations, over MGS3′s 1200.
• 1400 polygons, up from MGS3′s 4400.
• 5MB of textures, with a 512×512 for the face and 1024×1024 for the body.

Additionally, a higher-res screenshot of the FaceManager facial animation sliders allow us to peer deeper into the variables used to bring their fantastic characters to life. Here’s the modest list of facial expressions to accompany their similarly conservative facial bone-count:

• Nose_Up
• Open_Jaw L/R
• Smile L/R
• Anger L/R
• Kiss L/R
• Frown L/R
• Extra_A L/R
• Extra_B L/R

One imagines the last two to be unique to each character, and there are clearly additional tabs for Phonemes, Eyes and Wrinkles.

A few weeks back, details of the facial animation rig and other workflow info had been posted on the Japanese XSI website and I was planning to extract information via the google tranlation and observation alone, but someone beat me to it, (and managed to do a much better job than I would ever have). Head on over to Chris Evans’ (Tech-Art Lead at Crytek) blog for full translations of the following sections:

• MGS4 Facial Animation
• MGS4 Character Pipeline
• MGS4 Cluster Constraint Setup

Regarding the  facial setup, it looks very reminiscent of the same method I saw presented at ADAPT 2007 by Aaron Holly of Disney. This involved a similar setup of a bone rig driven by a mesh giving the two following important advantages.

1. It was highly flexible and able to be moved between multiple similar faces as the animation is stored on a nurbs mesh that drives the bones rather than the bones directly, therefore allowing for varying bone positions.
2. If using a pose-based facial animation solution such as FaceFX, the bones travel along the curve of a nurbs surface rather than a simple linear translation, therefore better mimicking the movement of skin across the skull.

This is certainly something I’d be keen to try in the near future given that it now appears to have successfully been put through a full videogame production.

Square Enix: The Technology of FINAL FANTASY

Taku Murata – General Manager, Technical Research Division

Traditionally, a new platform was created for each title, with the game first made in Japanese and translations following much later. This looks set to change with the latest upcoming releases which will be very exciting to many western fans, and the target platforms (for the engine) are PS3, PC and XBOX360.

Murata’s history reads like something of a chronology of technological breakthroughs in Japanese game development, with much of his work driven by animation – in particular facial animation. Of interest most of all was the admission that several of the driving forces for this new engine centred on displaying characters’ faces to a very high fidelity in close-up.

1997 – Final Fantasy Tactics: First time using real-time previewing on the console, convincing Murata of the power of this approach. The game was edited on PS1 in realtime, driven by the artists’ requests.

2000 – Vagrant Story: Used a unified tool to create cutscenes, preview textures and visual effects. Apart from the opening FMV, every cinematic was in-game and featured skeleton/bone animation for the first time. One requirement was that the team had to preview facial texture animations to ensure they looked good without anti-aliasing. Murata spoke at length about what he called “peak-points”, which are presumably normals. Apparently they had to change the peak-points relative to the camera to maintain the facial integrity. Lots of post-effect and field-of-view corrections, such as a fish-eye lens filter, were employed in order to promote a wide range of facial expressions.

2004 – Data Standardisation: Established a common 3D data format. Within the company there was a big debate over whether to use COLLADA, FBX or a proprietary file format, with each team previously using a different format. They eventually decided on their own proprietary data format, but the integration process wasn’t easy, especially when convincing certain entrenched teams.

2005 – Tech Division Established: Murata’s team was formally created with the objective of establishing a company-wide technology.

2006 – Final Fantasy XII: Team sizes by this time were incomparable to previous projects and as such brought with them large volumes of assets required to be created by staff with diverse skill levels. To aid this, Murata’s team created separate tool sets for different needs and skill levels.

2007 – Crystal Tools: The company-wide technology was finally rolled out. Previously named the “White Engine” , the Crystal Tools platform is the result of Murata’s team’s work. The essence of the Crystal Tools was described as thus:

• Must support extensive use of character close-ups.
• Focus on stylised facial expressions designed to promote the anime style.
• Must allow detailed control of characters.
• Specialises in physics, visual effects, post-effects and Graphical User Interface.
• Must support a large team, with a detailed division of work.
• Contains seperate tools for different functions.

With an extensive use of GUI to accommodate veterans and novices alike and focusing on ease of use, the Crystal Tools are currently being used for the production of Final Fantasy XIII, Final Fantasy XIII Versus and unannounced next-gen MMORPG. Separate teams for separate tools allowed v1.0 to be created in a year, though as is often the case in engine development they did not employ a technical writer so the documentation suffered. This is an area that game developers are slowly coming around to, where complex tools and system are created on an almost daily basis but oftentimes their power is not realised due to bad or lacking documentation.

Apparently, after one post-session questioner inquired, Murata admitted that Square Enix cannot currently license tools due to documentation, but possibly in the future – something very interesting indeed should the chance to work with their tools arise. Of all the tools mentioned, (Character Viewer, Effects Editor, Cutscene Editor, Layout Tool and Sound Maker), I fortunately managed to find screens online of the two most interesting to videogame animators.

Character Viewer: The Character Viewer is only for previewing of textures and animation and exists on PC only, with modeling and animation still created in Maya or XSI. It clearly displays a hierarchy view though, hinting towards additional character set-up that could be performed and maintained from within this external viewer.

Cutscene Editor: Bearing a resemblance to Unreal Engine 3′s “Matinee”, the Cutscene Editor offers timeline control over cinematography, visual effects and audio all in a single editor. The multi-screen view shows that this is an editor and not simply a tool in which to re-construct pre-exported cameras and cutscene assets, with an asset browser and animation curves clearly visible towards the lower right.

Sony Pictures Imageworks: A Believable Character Postmortem: Motion Capture on the Virtual Set of BEOWULF

Parag Halvadar – Lead R&D Engineer

Hailing from the same studio that created Monster House, Halvadar’s talk concentrated on facial motion as that’s a recent topic for games industry. As is often the case with movie industry approaches they couldn’t directly be recreated for use in a game development situation, but nonetheless provided an interesting insight into some of the lengths that must be gone to in search of the (some say, false) holy grail of truly photo-real virtual characters.

The first portion of the talk involved simply tallying the vast amounts of data, equipment and effort used in the production:

• 260 Vicon MX40 cameras were used synchronously to record motion.
• Body, facial and hand motion were captured simultaneously.
• An Electro Oculograph (EOG) was used to record eye-tracking.
• 20 actors could be captured simultaneously.
• Actions were captured in a 55x55x25ft volume.
• 81 actors were tracked over the course of the movie.
• 4 horses.
• 1 pony. (Only one?!)
• 46 days of shooting.
• 250 props made and captured.

The second portion detailed the methods required to bring the faces to “life”. It must be said that, despite often firmly entrenched in the Uncanny Valley as is always the case with attempts to simulate realistic facial motion, Beowulf has done the best job yet at providing real glimpses of coming up the other side. The tallies continue:

• 4 layers of face rigs.
• 3D facial models did not match actors faces in a 1:1 ratio, (Ray Winstone in particular), causing lots of marker-swapping.
• Adhered to FACS (Facial Action Coding System), to recreate all the muscles of a human face, totalling around 60 facial expressions (including head motions), with 16 different phoneme shapes.
• Face poses were created from combinations of weights of a smaller set of basic poses.
• Motion-capture values were run through a script to find the closest match with the facial expressions and were replaced with blendshapes.
• The EOG recorded horizontal and vertical eye movement, saccades and blinks via and eyepack on back with electrodes by the eyes to detect eye-muscle movements.

It should be noted that, while in my opinion the face of Angelina Jolie was the most successful and consistent in quality throughout all shots involved, Halvadar explained that hers was the scan that was deliberately adjusted the most to form an exaggerated impression of how we picture her, backing up my belief that realism is simply just not realistic enough when it comes to artistic endeavours such as this. This was also apparent in the scene of her naked, gold-dripping body emerging from the water – something which he felt the need to show several times over and was also manipulated drastically due to her pregnancy at the time of shooting.

In closing, it was most interesting of all that Halvadar’s decision to show each scene step-by-step revealed that every shot only achieved the final visual quality after a final pass was made by an animator working with video reference of the original scene, begging the questions as to why go to the bother of all the technicality when that process could be done from scratch with presumably similar results.

If absolute realism in games still is your thing, then you may wish to investigate the work of George Borshukov at EA and his Universal Capture (UCap) method. Proven in The Matrix trilogy and Tiger Woods tech demos this really is something to watch, especially since its optimisation for real-time implementation.

As noted on the previous movies, these recent screens of SFIV point towards some pretty animation-intensive details, sporting bone scaling, cloth and an impressive detailed facial setup, as well as what must be some form of muscle deformation to maintain volume in the limbs. On top of all this is a look-at system between not only the two main protagonists but also the crowd of onlookers – something that will become standard in the near future to overcome the lifeless gaze of previous-generation videogame characters.

Similarly, this Team Fortress-esque video displaying the cartoony facial setup for EA’s Facebreaker looks worth keeping an eye on too. Most impressive of all though, is that the game is made by the “developers of the critically acclaimed Fight Night Round 3″ despite them all being fired a few months ago…

Click here to view the embedded video.