To replace the faces of Yuen's men with that of Agent Smith - while retaining the level of photorealism that the Wachowskis demanded - Gaeta and his team built a system for sampling the real at a higher resolution than had ever before been attempted, dubbing this process universal capture.
Gaeta began by making lo-res laser scans of Reeves' and Weaving's heads in relaxed, neutral poses. These scans furnished the basic geometry upon which succeeding layers of real-world data would be applied.
Then Reeves and Weaving each sat down on a stage in front of five Sony HDW-900 video cameras. The massive datastreams from these cameras - one gigabyte a second - were treated like holy water; even the cameras' color-correction software was disabled to prevent any loss of data. Instead of recording to tape, which requires compression, the cameras were modified to send uncompressed data to a bank of high-end PCs that stored it on a huge disk array. "The scene in that room was surreal," Gaeta recalls. "There's this guy onstage, and his face is surrounded with this f*cking Cape Canaa-averal backup system."
As Reeves and Weaving acted out a range of facial expressions for their rumble in the courtyard, the cameras captured each twitch of muscle and every change in the blood flow to the skin. This data was then analyzed with algorithms written by Borshukov that tracked each individual pixel as it moved from frame to frame. The tiny irregularities in the actors' faces actually made this job easier, giving Borshukov's algorithms distinctive points in space to grab on to as he reconstructed the actors' features moving through time.
