Industrial Light & Magic: Building Iron Man

Marc Chu – Animation Supervisor

Beginnng with his history, Marc joined ILM in 1994 and has since then served on 20 films, perhaps most notably as Animation Lead on “Pirates of The Caribbean” character Davy Jones. As Ironman was Marvel’s own first fully self-financed feature, they had 6 different companies competing for VFX work. It was interesting to see that despite its reputation, ILM must still compete for bang-for-buck value as film studios are keen to shop around. It comes as no surprise though, that the work was won in part on the back of the impressive Transformers work.

To this end, he showed a rough animation test of Iron Man taking off done over the course of two weeks. ILM has plentiful archive footage from which it can draw resources, and for this piece air footage repurposed from Ang Lee’s Hulk was used to create a high-quality flight sequence.

The Suit Model

The talk continued with Marc taking us through the various versions of the Iron Man suit, which were farmed out to seperate VFX houses. The Mk I was handled by Vancouver studio The Embassy, with the Mk II (above) by The Orphanage. Stan-Winston’s Studio was drafter to create the full non-CG suit, with the traditional approach of working on small maquettes to reduce costs.

From this base, ILM built the fully working Mk III suit in Maya, showing tests to demonstrate difficult rigging areas such as around the pelvic area. 44000 photos and 1100 quicktime movies were gathered for reference, and interestingly Marc displayed Photos as frames in .mov files for ease of access. These photos contained everything possibly having to be referenced or recreated in CG.

As part of the CG suit creation process, the team would make daily turntable renders to compare to the real Winston models and compared shaders to real iron balls. Regarding this, around 6 different damage materials needed to be created for the suit at various stages throughout the movie.

Chu admitted that Iron Man’s flight was based on Star Wars Snowspeeder movement, though initial air-brake test concepts were weak – based on an idea of the hero being some kind of “Samurai In The Sky”. Of interest, many of the initial movement tests were done by the suit modeller, whom one assumes must have been rather multi-talented, especially after we were shown the depth to which Iron Man’s endoskeleton was fully detailed and modelled.

Some Mk III suit modelling stats:

• 1,900 Individual Objects
• 872,000 Polygons
• 3756 Texture Maps
• 483 Rigging Nodes

The Animation

The film featured lots of “Digital-Double” shots, same for cars & jets etc. ILM made continued flight tests on their choices of air shots, with Chu admitting that the relationship between the VFX team and the Director had never been as highly collaborative as this before, with the Director asking animators to provide ideas for shots. Sometimes entire shots were created to match ILM test renders.

For the final battle with Ironmonger, ILM had to redesign the back-piston rig to avoid crashing when he moved, with Marc demonstrating some excellent keyframed walk-cycle tests. On the set, actors interacting with Ironmonger traked a low-tech “head-on-a-stick” for eyeline reference. At this point he mentioned it was more difficult working on live-action plates due to the numerous limitations such as predefned locations where the character’s head must be placed.

Perhaps the technological highlight of the talk was the section on ILM’s proprietary iMOCAP technology that allows the motion-retargetting of CG assets using just the filmed plates, avoiding the requirement for motion-capture suits. This method was used primarily to allow actor Robert Downey Jr. to have the bulky MKIII suit added back on in CG in some instances where the real-life suit would have proven too resticting.

Surprising for a humanoid super-hero, only 4 shots in entire movie used real mocap. With the tally of CG shots as thus:

• 409 CG shots, of which…
• 289 were 3D animation, comprising…
  • 254 Keyframe Animation
  • 29 iMOCAP
  • 4 Mocap

Director Jon Favreau’s summation of the hero’s movement was, “Superman takes off fast and lands slow. Ironman takes off slow and lands fast”. With this in mind, ILM had to show a visible improvement to character Tony Starks attempts at taking off throughout the movie. We were shown jet-blast tests done over a still photo, (to create an easy looping sequence), with Chu remarking that this area proved a challenge compositing all the flare VFX.

As I’d been previously informed at Giant Studios, the team mocapped an actor in a wind tunnel with his feet tied together to get an idea of how Iron Man might perform during flight, but in the end this proved useful as good reference only. Of note, the actions during air flight used FK animation only – a change from the typically IK-based rigs ILM use in Maya.

The last part of the talk focused on the machine Stark used to build and don the suit, the “Suit Machine”. This effect required a tremendous amount of work to create, and was notable mostly for the technique of recreating the background set digitally. By capturing many photos, taken in multiple exposures, ILM created a single 360 degree shot of the set – placed on sphere like a videogame’s skybox. All shots during this sequence used this virtual background, with the Suit Machine (and Iron Man) rigged by another multi-talented indivdual, a Transformers animator that did some of the most complicated shots of that movie.

Suit Machine modelling/rigging numbers:

• 15461 Objects
• 3,980,000 Polygons
• 1825 Texture maps
• 2284 Rigging nodes

In summary, the project totalled 400 VFX shots over the course of 3 or 4 months, though when pressed at the end on a shot quota at ILM, Chu explained that there is no quota in the traditional sense, and the studio never rushes animators, taking time to get each shot right. As such, one presumes they have banks of animators waiting to pick up the slack when schedules slip, as he described the practice of bringing on extra people before deadlines etc. Overall, Iron Man had 15-20 animators over the course of production.

Walt Disney Animation Studios: Building A Hamster Named Rhino

Clay Kaytis, Philippe Brochu & Hidetaka Yosumi – Lead Animator, Lead Modeller and Technical Director for the character Rhino.

This presentation could easily be split into two parts, with the first concerning the solution achieved to maintain model fidelity in a character that could easily move between biped and quadruped movement, and the second on their general deformation solution for all areas of body/limb movement.

Beginning with an exclusive new trailer, the speakers began by describing the requirements of the rig which required the dual functionality of quadruped rodent-like movement and bipedal anthropomorphic acting. The character Rhino was described as essentially a ball of fat covered in fur that exists primarily inside a hamster-ball. While the ball-rig setup may have proven an interesting topic enough, with the TD writing special software for this alone, it proved enough of a challenge to overcome the transition between 4 and 2-leg stances.

Initially, tens of steps of deformers were employed to handle the transition which required the character to undergo a variety of transformations such as rotating the head 90 degrees to face front once standing, however all these corrective tools slowed down the rig somewhat. As is something of a standard now, the team began putting the model through its paces by forcing it into many poses and drawing on top what should be the corrections made to fully realise them. However, difficult posing not ideally suited to the rig cause further corrective work on the modellers’ side downstream, and more importantly caused the animators to treat the rig somewhat conservatively with less-adventurous animation.

At some point towards the end of pre-production, a decision had to be made whether to change the rig to better support the animators. This required a new way of approaching the modelling/animation workflow for Disney studios, eschewing the previous linear progression of modelling->rigging->animation-> in favour of a much more versatile circular progression under the banner of the “character team” where each area has buy-in at every stage, much like agile development. The name given to this month-long process was affectionaly termed “Rhino-Palooza”, and it’s success spawned similar “palooza’s” for every other major character.

As the aim was to share only one rig for both pedal stances, the solution was to to simply morph the character model between ideal quad and biped topologies, allowing for a smooth transition between running and standing even during motions. This is essentially the same method employed in High Voltage’s Xenomorph XNA demo in 2004, and was overshadowed by the clever solution to the second problem posed by Rhino’s body.

With such a short neck, arms and legs, in order to fully realise Rhino’s mandate of “a ball of fat covered in fur” the team had to come up with a new system for using corrective chapes to intelligently deform the character. To this end, they improved on Disney’s own PSD (Pose Space Deformer) system that applied morphs for every major movement to maintain character model integrity. PSD is an in-house tool at Disney that directly talks with Maya’s skin-clusters.

Unlike many game-engine limitations, Disney’s animators are free to fully animate bones using all the areas of translation, rotation and scale, and most importantly are free to break the rigs as they please. As such, animators are able to get the most out of any character without fear of affecting the final result downstream, but this also means that traditional methods for employing corrective shapes do not always come into effect.

The old method of creating a shape for each an every rotation value (eg, x,y & z for the neck) did not take into account actions that were animated with scaling of the bone or even breaking it via translation, and creating shapes for each value would have resulted in far too many shapes than would both be realistic to maintain and would likely slow the rig to a halt. Additionally, simple actions like leaning the neck into the shoulder might cause the fat to bunch up there, but if the shoulder was already down the fat shouldn’t squash up at all.

The elegant solution they finally arrived at was a sytem of locators positioned on the low-res character mesh whereby, rather than using simply bone rotation values to drive keys on morphs, instead measured the relative distance between two or more relevant locators to decide if a morph should be blended in. Interestingly, the team actually found that results were increased as they reduced the locator-count and fine-tuned the driven-key inputs.

This system is a much more robust approach to the familiar driven-key morph-target and, (as were are moving closer to employing driven-keys in realtime game engines), sounds like a good method to investigate – should one ever have the chance to work on a game invloving a small number of high-resolution characters.

Square Enix: The Technology of FINAL FANTASY

Taku Murata – General Manager, Technical Research Division

Traditionally, a new platform was created for each title, with the game first made in Japanese and translations following much later. This looks set to change with the latest upcoming releases which will be very exciting to many western fans, and the target platforms (for the engine) are PS3, PC and XBOX360.

Murata’s history reads like something of a chronology of technological breakthroughs in Japanese game development, with much of his work driven by animation – in particular facial animation. Of interest most of all was the admission that several of the driving forces for this new engine centred on displaying characters’ faces to a very high fidelity in close-up.

1997 – Final Fantasy Tactics: First time using real-time previewing on the console, convincing Murata of the power of this approach. The game was edited on PS1 in realtime, driven by the artists’ requests.

2000 – Vagrant Story: Used a unified tool to create cutscenes, preview textures and visual effects. Apart from the opening FMV, every cinematic was in-game and featured skeleton/bone animation for the first time. One requirement was that the team had to preview facial texture animations to ensure they looked good without anti-aliasing. Murata spoke at length about what he called “peak-points”, which are presumably normals. Apparently they had to change the peak-points relative to the camera to maintain the facial integrity. Lots of post-effect and field-of-view corrections, such as a fish-eye lens filter, were employed in order to promote a wide range of facial expressions.

2004 – Data Standardisation: Established a common 3D data format. Within the company there was a big debate over whether to use COLLADA, FBX or a proprietary file format, with each team previously using a different format. They eventually decided on their own proprietary data format, but the integration process wasn’t easy, especially when convincing certain entrenched teams.

2005 – Tech Division Established: Murata’s team was formally created with the objective of establishing a company-wide technology.

2006 – Final Fantasy XII: Team sizes by this time were incomparable to previous projects and as such brought with them large volumes of assets required to be created by staff with diverse skill levels. To aid this, Murata’s team created separate tool sets for different needs and skill levels.

2007 – Crystal Tools: The company-wide technology was finally rolled out. Previously named the “White Engine” , the Crystal Tools platform is the result of Murata’s team’s work. The essence of the Crystal Tools was described as thus:

• Must support extensive use of character close-ups.
• Focus on stylised facial expressions designed to promote the anime style.
• Must allow detailed control of characters.
• Specialises in physics, visual effects, post-effects and Graphical User Interface.
• Must support a large team, with a detailed division of work.
• Contains seperate tools for different functions.

With an extensive use of GUI to accommodate veterans and novices alike and focusing on ease of use, the Crystal Tools are currently being used for the production of Final Fantasy XIII, Final Fantasy XIII Versus and unannounced next-gen MMORPG. Separate teams for separate tools allowed v1.0 to be created in a year, though as is often the case in engine development they did not employ a technical writer so the documentation suffered. This is an area that game developers are slowly coming around to, where complex tools and system are created on an almost daily basis but oftentimes their power is not realised due to bad or lacking documentation.

Apparently, after one post-session questioner inquired, Murata admitted that Square Enix cannot currently license tools due to documentation, but possibly in the future – something very interesting indeed should the chance to work with their tools arise. Of all the tools mentioned, (Character Viewer, Effects Editor, Cutscene Editor, Layout Tool and Sound Maker), I fortunately managed to find screens online of the two most interesting to videogame animators.

Character Viewer: The Character Viewer is only for previewing of textures and animation and exists on PC only, with modeling and animation still created in Maya or XSI. It clearly displays a hierarchy view though, hinting towards additional character set-up that could be performed and maintained from within this external viewer.

Cutscene Editor: Bearing a resemblance to Unreal Engine 3′s “Matinee”, the Cutscene Editor offers timeline control over cinematography, visual effects and audio all in a single editor. The multi-screen view shows that this is an editor and not simply a tool in which to re-construct pre-exported cameras and cutscene assets, with an asset browser and animation curves clearly visible towards the lower right.

Sora Ltd: Development – SUPER SMASH BROS. BRAWL

Masahiro Sakurai -Director

Despite directing the original SSB games under the Nintendo/HAL Laboratory collaboration, Sakurai has been working as a freelance game designer since 2004 under his own company Sora Ltd. – (the company comprises just him and his assistant). After sub-contracing creative direction to Sora, Nintendo rented offices in Tokyo and employed the bulk of staff from long-time development house Game Arts. In addition, they temporarily contracted many of the original Smash Bros team for this project as HAL were not officially involved.

This is the way I’d like to see the game development process go in the future so we can move away from the restrictive full-time studio model towards a more talent based one where individual creatives and full development teams can be married, before disbanding once the pipelines and initial creative visions are established.

Sakurai continued with the challenges faced in developing such a diverse character roster,admitting that they were “blessed” with having characters from many famous games. By July 7th 2005, all the planning docs were finalised, as was the roster. The character line-up was only reduced from there on, except Sonic who was added in 2007.

CharacterDesign:

• Listing his requirements for a successful character design in SSBB, Sakurai pointed to individuality, ease of implementation and qualities that would bring balance to the roster.
• Some rights issues prevented even Nintendo characters from being included.
• He wanted characters with unique visuals to stand out.
• The team had to unify the art style somewhat, drawing a comparision between Mario vs Link throwing up inconcistencies like Bugs Bunny cartoons vs Photo-real.

• Changes that were made included a washed-out colour pallette and detailed textures for some of the more cartoony characters. Notably, Nintendo allowed them to add more details to Mario despite usually enforcing strict style guidelines.
• Thankfully body proportions were not unified, allowing for an interesting variety in deliberate mis-matchs of character sizes.
• Sakurai pointed towards an improvement in the team’s ability rather than Wii’s power facilitating an increase in visual fidelity.
• All of the team’s “interpretations” of characters were supervised and approved by the original character creators.
• Kid Icarus main protagonist Pit, who had not been updated in 20 years, had to be completely redesigned in order to fit with other characters that had improved and moved with the times – requiring the original creator’s consent. At this point the talk spent an uncomfortably long time examining the finer points of Pit’s jewelry and sarong, betraying the Japanese fascination with effeminate male characters.

Character Animation:

This latter section was by far the most interesting part of the talk, mostly due to the clearly determined approach in which Sakurai dictated animation timing and posing to his team. To this end, he began by thinking of moves with the approach that one must maintain a “preciseness and steadfast confidence that the move will work.”

• Every attack in the game can be broken down into 4 component parts or phases:
  • StandBy – either the idle pose or the fall for mid-air assaults – (essentially the first frame).
  • WindUp – the initial buildup as the character draws back in anticipation of the attack. This should never take longer than 0.75 seconds.
  • Strike – the strongest pose, seen often by players as the action pauses here when a hit lands.
  • FollowThrough – after the attack, the period of vulnerability following a missed move. It is important that this pose must not look like the StandBy, and typically is the longest part of the move.
• The animations needed to be exaggerated in order to ensure the player had enough visual feedback following a key-press as well as warn the opponent to dodge.
• Translating each of these 4 stages for each action into numbers greatly aided the designers.
• Every action was hand-done in key-frame, involving no motion capture.

• Sakurai’s method for illustrating moves to his animation and design team relied on the use of side-on photographs of “Microman” poseable action figures (above) to convey each of the four attack stages visually.
• He dictated every pose, but said the final quality was down to the animator. Nevertheless, I believe this would be a hard sell to Western developers even with a track record such as his.
• Stringently enforcing the posing in this manner, however, insured the moves would work within the game and the number of changes to animation for balancing purposes was reduced.

When balancing the game, he closed with the advice that parameters, motions and “character essence” must all be consistent and are the most important elements of all. (eg. weight of Mario and Samus makes them fall differently). Furthermore, he feels it is imperative that a designer sniffs out why design decisions are made in other games and learn from them. Game designers must only take a “try out and see” iterative approach when they have enough lee-way and time to do so.

Feelplus: Looking Back at LOST ODYSSEY – The Challenge of Cross Cultural Development

Ray Nakazato – President, Feelplus Inc.

As with each of the Japanese presentations, Nakazato began by detailing the hierarchy of the companies involved in the project. Feelplus Inc. was established in 2005, with the team quickly growing in size to the final headcount of around 100 developers, many of which came from Microsoft and SEGA. Feelplus is 1 of 3 companies under the AQ Interactive Group, (including Artoon and Cavia), and the project was a collaborative effort with Sakaguchi’s team at Mistwalker who formed the core desgin team.

Of interest, Feelplus found difficulty in hiring staff under the Microsoft banner. One imagines that much of the staff was populated with juniors as a result, as can happen here in the west when attempting to ramp up into production quickly. The total timeline for the project was given as:

• 8 months to prototype
• 11 months to achieve First Playable stage
• 8 months to reach Alpha

This is an incredibly short time for an RPG of Lost Odyssey’s scale despite the 100-man team size and is testament to the efficiency of their working practices. Their engine of choice was Unreal3, which may have expedited some of the risk associated with building a brand new studio but I know only too well how difficult it is to shoehorn an RPG and all the unique systems it requires into an engine built primarily for shooting games. As such, their experience was all-too-familiar, with integrations of new engine builds often holding up production, taking 3 people 6 weeks for each integration.

Nevertheless, the in-game animation production actually went quite well:

In-game Animation:

• At peak production, there were 20 ingame animators, with Animation falling under the charge of the Art Directors.
• A “Setup Group” was employed to integrate animations into the game.
• Characters and creatures were done on time.

However, the cutscene production provided several challenges due to the sheer quantity and variable quality as a direct result:

Cutscenes – What went right:

• The cutscene Visual Director came from the movie industry, bringing with him a wealth of experience.
• Facial expressions were keyframe-animated as the automated method (presumably via the Unreal-integrated FaceFX) proved too low quality. Nakazato was especially proud of the pupil movement.
• Events were of high story-written quality.

Cutscenes – What went wrong:

• Battle, Adventure and Cutscene systems and scenarios were developed seperately.
• Too much time was spent creating locations that were passed through swiftly, therefore negating the effort put into them.
• Over 300 cutscenes were required, with 4 different quality levels:
  1. FMV (Full Motion Video) Event – Pre-rendered cinematics.
  2. A Event – In-game, with keyframed facial animation and specifically recorder Motion-Capture.
  3. B Event – Off-the-shelf mocap.
  4. Scripted Events – Purely in-game.

• Looking back, FMV and A Events used same assets, so there was an unusual inconsistency despite using all the same models.
• Users were confused over inconsistent quality as they were not aware of what was an A or B in-game cutscene.

I’m sure many players are aware of the visual disparity when mixing up pre-rendered and in-game cinematic cutscenes. Even if they use the exact same models the difference in lighting, texture resolution or even full-screen effects like motion-blur become immediately apparent. It cannot be denied though that pre-rendered cinematics, by their very pre-made nature, are much more stable and easy to integrate into a game. Not much can go wrong with simply playing a video, plus you can render out at a much lower framerate and therefore throw in more characters and visual effects into a single scene that would otherwise slow down the engine, but something like visually customisable characters can prohibit their use entirely.

Nakazato said of this incosistency, “So we did two different methods, but we ended up that the players didn’t know which one is A event and which one is B event, so they just felt that the quality of the entire movies are inconsistent. That’s what we are regretting at this point.” However it’s not all bad, with the game’s opening sporting one of the most seamless transistions from pre-rendered cutscene to in-game action since Final Fantasy VII and the original Abe’s Odyssey, which you can see below – (skip to around the 3-minute mark if you’re the impatient type).

Click here to view the embedded video.

Sony Pictures Imageworks: A Believable Character Postmortem: Motion Capture on the Virtual Set of BEOWULF

Parag Halvadar – Lead R&D Engineer

Hailing from the same studio that created Monster House, Halvadar’s talk concentrated on facial motion as that’s a recent topic for games industry. As is often the case with movie industry approaches they couldn’t directly be recreated for use in a game development situation, but nonetheless provided an interesting insight into some of the lengths that must be gone to in search of the (some say, false) holy grail of truly photo-real virtual characters.

The first portion of the talk involved simply tallying the vast amounts of data, equipment and effort used in the production:

• 260 Vicon MX40 cameras were used synchronously to record motion.
• Body, facial and hand motion were captured simultaneously.
• An Electro Oculograph (EOG) was used to record eye-tracking.
• 20 actors could be captured simultaneously.
• Actions were captured in a 55x55x25ft volume.
• 81 actors were tracked over the course of the movie.
• 4 horses.
• 1 pony. (Only one?!)
• 46 days of shooting.
• 250 props made and captured.

The second portion detailed the methods required to bring the faces to “life”. It must be said that, despite often firmly entrenched in the Uncanny Valley as is always the case with attempts to simulate realistic facial motion, Beowulf has done the best job yet at providing real glimpses of coming up the other side. The tallies continue:

• 4 layers of face rigs.
• 3D facial models did not match actors faces in a 1:1 ratio, (Ray Winstone in particular), causing lots of marker-swapping.
• Adhered to FACS (Facial Action Coding System), to recreate all the muscles of a human face, totalling around 60 facial expressions (including head motions), with 16 different phoneme shapes.
• Face poses were created from combinations of weights of a smaller set of basic poses.
• Motion-capture values were run through a script to find the closest match with the facial expressions and were replaced with blendshapes.
• The EOG recorded horizontal and vertical eye movement, saccades and blinks via and eyepack on back with electrodes by the eyes to detect eye-muscle movements.

It should be noted that, while in my opinion the face of Angelina Jolie was the most successful and consistent in quality throughout all shots involved, Halvadar explained that hers was the scan that was deliberately adjusted the most to form an exaggerated impression of how we picture her, backing up my belief that realism is simply just not realistic enough when it comes to artistic endeavours such as this. This was also apparent in the scene of her naked, gold-dripping body emerging from the water – something which he felt the need to show several times over and was also manipulated drastically due to her pregnancy at the time of shooting.

In closing, it was most interesting of all that Halvadar’s decision to show each scene step-by-step revealed that every shot only achieved the final visual quality after a final pass was made by an animator working with video reference of the original scene, begging the questions as to why go to the bother of all the technicality when that process could be done from scratch with presumably similar results.

If absolute realism in games still is your thing, then you may wish to investigate the work of George Borshukov at EA and his Universal Capture (UCap) method. Proven in The Matrix trilogy and Tiger Woods tech demos this really is something to watch, especially since its optimisation for real-time implementation.

As an extra little teaser, there will soon be something new coming to Game Anim of interest to videogame animators everywhere over the next few weeks. So on with the notes…

Naughty Dog: Uncharted Animation – An In-depth Look at the Character Animation Workflow and Pipeline

Jeremy Lai-Yates & Judd Simantov – In-Game Animation Lead & Lead Character TD

After a fantastic opening to GDC with Ken Levine’s inciteful speech on Storytelling in Bioshock, this, my second lecture, turned out to be not quite all that I’d hoped for. I was really expecting to gain insight into their facial animation setup and workflow as my time spent with Drake’s Fortune have proven the cinematics to be something quite special and well-produced. However, the talk focused squarely on their mocap workflow which was a fairly standard 3-skeleton setup. 1 animation, 1 game, and 1 mocap – snapping poses and animations between them.

What confused those present most was, on deciding against employing Motionbuilder as part of their process due to their exporter being Maya-based, they proceeded to manually recreate many mocap-related features Motionbuilder provides inside Maya, (though with the notable absence of layers), rather than simply recreate their exporter inside Motionbuilder. This was reflected upon at the end with the closing statement – “We had a tendency to over-think things”.

My notes:

Early Considerations

• Naughty Dog are traditionally known for cartoony games. A move from a stylised look to realism would necessitate a review of their approach. As such, they decided on integrating mocap from the start, employing the service of House of Moves.
• The initial goals were to create a stable rig and pipeline. Previous games left cinematic and game skeletons incompatible, but for Drake’s Fortune the aim was to have a single skeleton shared across both disciplines.
• Early on, they decided against Motionbuilder as their exporter was Maya only and they didn’t have the skillset to allow them to recreate their exporter inside Motionbuilder. As a result the entire workflow was created inside Maya.

Tools

• They created a custom tool to reference characters. Maya already supports referencing but they improved upon this to allow character selection and easy replacement. They needed consistent namespaces for references, and so the editor was employed to better handle this.
• Their importer/exporter ran at around 300 frames per second.
• The mocap and animation rig existed in the same file. The default setting for copying mocap onto the animation rig was 1 key every three frames. One assumes this approach was taken in the absence of animation layers.
• They created an animation library, (affectionately named iToons), for easy pasting of poses, (body or hand), and pre-existing animations as well as a time-warp curve for easy adjustment of an entire animation’s timing.
• A layer manager was devised to allow for subdivisions in Maya’s pre-existing visual layers.
• One interesting addition was a visual arc tool, much like Max’s (or Motionbuilder’s) trajectory display, that would show the trajectory of a point over the course of the animation, only the keyframed positions were colour-coded and could be manually edited to affect the actual trajectory of the animated element it represented.
• Of note, their FK rig was set up in a manner that allowed it to be be manipulated as if it were IK.

The Road To Realism

• Their initial treatment of the character movement style was very agile and versatile, involving flips etc, but this was scaled back in favour of a more “everyman” approach to the main protagonist. They naturally found this difficult to mocap due to the limitations of real-world physics on an actor. To overcome this the keyframed over the mocap to afford much more exaggeration.
• The entire project ended up being around 40% mocap vs 60% keyframe. Motion was captured at 120fps, a high fidelity, but they noticed a difference between the mocap and keyframes.
• In order to smooth the discrepancy, they willfully lowered the mocap quality by reducing keyframes, sometimes to only the key poses from an action. In addition, they employed what they called “Poor Man’s mocap” – video reference, (does anyone still work without video reference?), accentuating poses, adding “punch” to timing, snapping extreme poses and removing unnecessary pauses.

The closing section of the speech was the most interesting, focusing not on the process, but on what they did with the animations after they’d been implemented in the game engine.

First up were what they called “Partials”, essentially sets of bones that could be animated independently of the rest of the body. We’ve been seeing this for years, but when combined with the rest of the multiple animations played on the character the weapon-reloads really looked natural and flowed as Drake moved from one state to the next.

The second take-away idea was their imaginative use of additive animation, where they changed not the additive loop, but the underlying pose. With around 150 different poses for the cover state, they would change the player character’s posture based on various influences such as health, state-of-duress, as well as randomness.

Pixar: How Pixar Animation Studios Brings Characters To Life

Andy Schmidt – Animator on Ratatouille

This was an incredibly valuable lesson in the workflow for polishing an animated feature, which has some lessons we can directly employ for our own peer-review processes. The initially self-deprecating yet entertaining Andy Schmidt took us through the challenges of creating the characters for Ratatouille, (namely, how to turn vermin into an appealing character) before moving on to Pixar’s general approach to taking a scene through various levels of polish.

The biggest element of the talk that struck me was the difference between an animated film and videogame cutscene schedule – two supposedly similar projects in concept, with the key being when voice-over is recorded. Below is a comparison between Pixar and what is my experience of the norm for large-scale videogame project storytelling, taking a direct comparison with only the elements shared across mediums.

Pixar Animated Features – Videogame Cutscenes

Story Story
Script Lighting (Often unchanged from the level’s default)
Voice-Acting Storyboard
Storyboard Animatic
Pre-viz (Character placement, Cameras) Script
Animation Blocking Voice-Acting (Temporary in-house VO)
Animation Polishing Animation Blocking
Add Simulations (Skin, Cloth, Hair etc.) Voice-Acting
Add VFX Animation Polishing
Lighting Add VFX

Additionally, the four VO and Animation segments in the videogame timeline often contain a loop as the VO is reworked due to the script often being in flux – this unnatural workflow is every bit a by-product of not locking down the script early in the project, which is something our industry really needs to take strive towards.

Notes on Pixar character development:

• Each character was given a signature movement style to differentiate themselves from one another, this is something that would be great to explore with a game containing only a few diverse characters, as game animation is often diluted to facilitate sharing among multiple characters.
• Much research was undertaken on both rats and the cooking world, with the team joining a local cooking course and live rats brought in to the studio and hooked up to a web-cam. This unfortunately offered nothing more than lots of footage of sleeping rats, so old favourite the BBC Motion Library proved invaluable for rat movement.
• Among the many character-development sketches on show were what Andy called “Mechanical Sketches”, where the artist would envision how the skeleton rig might be placed inside the characters – something that is clearly considered at every stage of the process.
• Extremely finished paintings were created over the top of in-progress models to provide early lighting/material tests.
• Pose character sheets were devised to illustrate various correct-vs-incorrect methods for animating the characters – essentially style sheets setting a brand bible for each character.
• A decision made late in the character-development process to begin the movie with the rat characters walking on all fours necessitated a complete rebuilding of the character rig to allow optimum animation for both biped and quadruped motion.

Notes on Pixar rig development:

• As soon as the first rigs are created, vast amounts of animation tests are performed to try and break them when pushing characters to extremes. Similarly, facial rigs are run through calisthenics animations to push the expressions as far as they can go. At this stage, many character-defining walkcycles are also prototyped. Of interest, the walkcycles were completely symmetrical, pointing towards Pixar having a tool or method to quickly facilitate this.
• The animators work closely with the riggers and modeling department during the creation process. At the self-declared risk of sounding arrogant, Andy described the animators at Pixar as being the “vehicle” for telling the story, so modelers etc. bend over backwards to support them.
• One example of communicating change requests to modelers involved notes drawn over screenshots to illustrate the exact changes and improvements required.
• Animation tests would be created to test the reach of each character, therefore defining how some scenes might play out. The example shown highlighted a rather large chef’s inability to move in close to the kitchen-top, therefore necessitating additional collision deformation on his body.
• Similar tests are performed to find how far cloth and other dynamic elements of a character’s person could be pushed. It was interesting to note that by this stage the models were already fully complete in design.

Notes on Pixar animation style:

• Squash & Stretch is an integral part of their animation style.
• Many Pixar animators come from a 2D only background, including the speaker, as animation is seen as interchangeable between mediums. In my own experience, hiring good animators without any 3D experience has always proved fruitful, undermining perceived difficulties within own industry of hiring animators that have experience in Maya over Max and vice versa.
• Andy strongly recommended Ed Hooks book, “Acting for Animators”. Attendeding one of his lectures I also found it to be quite insightful, basing much of his observation on a character’s centre of mass defining his movements, as well as useful ideas concerning what goes on outside of a scene you are animating determining what plays out inside it.
• Andy went back to basics and displayed some inspiring examples of bouncing balls he’d found on the web. Noting that John Travlota begins every new character by creating a unique walk for him, Andy moved on to various walkcycles of his own creation, moving from realistic to highly stylized to illustrate how to promote caricature.
• Next, he showed a selection of admittedly rough though long character tests created to explore how characters might hold kitchen objects based on video research. These long tests would later prove useable for background characters in the final movie.
• Facial tests were also performed by matching frame-for-frame against movie clips of actors used as inspiration for the creation of the Pixar characters.
• Finally, “Lineups”, where all the major characters were lined up side by side performing actions in a brief scene, were created to explore consistency.

Notes on Pixar review process:

• Shots are divided out among animators almost like a casting process depending upon their particular strengths, such as humour, timing, caricature etc.
• Peer review is paramount, where showing in-progress shots for advice from the rest of the team is consistent in “Dailies” meetings, though it was not clear if this was mandatory. This is something essential to maintain, (and increase), quality among any group of animators, though a weekly assembly should suffice.
• The Director, in this case Brad Bird, uses a laser-pointer to highlight areas for work on the projector screen, though a video showed him to be incredibly animated when illustrating his comments – despite appearing to micro-manage the acting.
• A lot of, (but not all), Pixar animators shoot their own animation reference – this is sometimes sped up to fit better with the cartoony style.
• Their test renders for review, just like the Halon ones, were done with black bars at the top and bottom containing information such as the filename, version number, date, frame numbers, camera info and state, eg. blocking, polishing etc.
• In summary, it appeared the motto was to rework, rework and rework again to achieve the levels of quality Pixar is known for.

Finally, while most videogames unfortunately still rely on text and exposition, Andy finished his talk with an educational section stating that “Theatrics” should be used to tell a story without the viewer noticing, breaking this into the following insights.

• “You can say a lot with key-posing”: Mentioning storytelling poses, Andy informed us that you can really nail a scene in one pose. This is best illustrated in paintings, but he also showed some extreme poses from sports press that really showed more motion than many animated sequences. He advised to “sneak extreme poses” into animations to really sell a movement, using a scene from The Jungle Book as an example where Mowgli’s head is literally buried in his ass when climbing a tree. I myself have noticed rig-destroying examples like this being used to great effect in the Dead or Alive series of games to really accentuate attack motions.
• “Playing an action until something happens”: A simple observation to never have characters idle without at least something appearing to be going on in their heads – something not easily avoidable in videogames unfortunately.
• “What’s happening vs what’s really happening”: Basically, what is the scene really about? This idea centres around layered character motivation affording a vehicle for more interesting dynamic between characters where secret motivations allow acting on two levels – that of what the other characters know and of what the audience knows.
• “What does your character want? What do they do to get it?”: Another motivational observation, though this time a more general rule about character actions defined by their needs and what they do to achieve them.
• “What happens to your character isn’t as important as what you feel about it”: Essentially a separation of the events playing out around your character and how they choose to react, allowing the story to focus more on the characters themselves rather than what events take place for a deeper more human storytelling experience.

Di-O-Matic: Efficient methods for creating lip-sync blend shapes

Laurent Abucassis – Founder: Di-O-Matic

A considerably lower-key affair than Halon’s, this talk revolved around a demonstration on how to make phonetic mouth-shapes for lip-sync via blend-shapes (or morphing), something that, while offering more control over mesh deformation than simple bone positions, can be quite a pain to actually create and maintain the multitude of models required to create a blend-shape list.

While it did turn into something of a product pitch towards the end, the educational portion of the talk began with pointing out the first mistake most animators make when creating lip-sync for the first time, whereby they try to for shapes for every letter. However as Laurent said, “A letter is not a sound”.

He went on to advise that you should lip-sync what you hear, basing his observations on the following research:

• Phoneme shapes are what most lip-sync animation is taken from, and are generally considered to be the smallest building block on which lip-sync should be generated. Phonemes represent the sound that we pronounce. This is in keeping with current beliefs regarding lip-sync that we should concentrate on the simplest sounds that each word makes. However, his research has brought him to the conclusion that it is in fact Visemes, those being the basic mouth positions required to voice these sounds and therefore an even smaller building block from which to take.
• Quite how his procedural generation finds these based on an audio file (which offer phonemes only) is intriguing, but Voice-O-Matic did not appear to give superior results to a Phoneme-based extraction method, so the Viseme observation may be for animator reference only.
• He also referred to what he calls the “Thunderstorm Factor”, where the speed of light vs sound requires that you place your lip-sync poses 1-3 frames ahead of where you’d expect them to be based on the audio – a questionable proposition based on experience but something that is quite harmless to try when working with animation curves.
• Richard Williams has a handy technique for deciding when and when not to animate the jaw opening – put your hand under your chin when speaking the line to feel the absolute necessary moments required to sell the action.
• Finally, he touched on the idea of “Sticky Lips” – the method of adding a delay while opening the mouth to give a realistic feeling of soft lips uncompressing. This would presumably work nicely when the mouth closes also.

When giving his demo, he listed the following shapes required for the most basic lip-sync:

• Vowels: A | E | O | U/W
• Consonants: B/M/D | S | Ch
• Tongue: L | Th/D | N
• Teeth (optional): F/V

The meat of the presentation though was the demonstration for quickly creating multiple morph-target models. Beginning with just 4 base models, (mouth neutral, mouth open wide, mouth “O” shape, and finally a wide toothy smile), he used combinations of these 4 to create all of the shapes above within just a few short minutes. Additionally, he used the “mouth open wide” morph slider to hand-tweak adjustments inside the mouth, easier facilitating access to the tongue that can often be quite tricky.

What was especially interesting about this workflow was that changing one of the 4 base models had a recurring effect on each of the created faces, due to them retaining the modifiers of the originals. This makes management of the large number of models for each face plausible for any multi-character project, that’s assuming morph-targets are a viable method for facial animation on any project displaying more than just a few characters at a time.

Industrial Light & Magic: VFX Used On Transformers

Todd Vaziri – VFX Sequence Supervisor for Transformers

One of the most entertaining presentations of the week, due to both Todd’s upbeat yet humble attitude and the sheer multitude of videos displayed during the presentations, ranging from multiple render-passes highlighting the various explorations of lighting and materials on the robotic protagonists to behind-the-scene shots of the film plates throughout the various layers of post-production layering.

Incredibly heartening were the animation renders illustrating the sheer amount of cheating going on when characters went off-screen. With the original brief requiring 14 robots in total, they scoped for only 14 transformation animations, but ended up creating over 140 due to each transformation being created specifically to sell the particular shot. Some examples shown had Transformers’ legs going through the ground, various parts scaling into the body to be hidden away, even bits flying off only to return just at the moment they were required on camera – just like our cutscenes!

Some animation-related notes:

• Despite having a very humanoid rig underneath all the additional moving parts, Mocap was used for subtle idling movements for the Transformers – everything else was keyframed.
• Similarly, procedurally-generated secondary (i.e. physics-based) animation was used rather liberally on the characters. The demo illustrating this (showing Optimus Prime) had only a few flaps around the shoulder and arm areas, leaving much of this type of work to the animators themselves.
• Character animation was done in two passes, with one team providing the finished body animation of the Transformers and a second group later going in to animate the vast (see: tens of thousands) number of smaller moving parts required to raise the characters to big-screen fidelity.
• Todd’s main inspiration came not from other movies based on their action and visual effects, but rather on the actual type of film used when shooting the live-action plates which results in certain lighting artifacts due to the stretching of the film image to fit the screen. In his words, “You don’t want to use VFX films as your VFX reference”.
• He had a little advice for camera-shake, being that it should always be implemented slightly after the event causing it, presumably varying by distance from the event. This is something we could easily experiment with in future games.
• Surprisingly, several of the VFX shots were created almost entirely using 2D in After Effects, with one example shown involving explosions made entirely of video footage being translated in 3D space – good old sprites are still useful when it comes to creating smoke and explosions even for the big guys – it is fair to say that Mr Vaziri is an advocate of low-fi techniques.

I’ve been to a few ILM lectures now, and at each one I’m staggered by the amount of time it can sometimes take to render just one frame for a CGI sequence. For those who still wonder how long it will take for videogames to achieve photorealism, (a fool’s quest at best), consider this gulf:

• Time currently allocated to render one frame in most XBOX360/PS3 games – 1/30th or 1/60th of a second.
• Time currently required to render one ILM VFX-heavy frame for film – anywhere from 10 to 30 hours.