The tessellator allows for the use of very high polygon surfaces with a minimal memory footprint by using a form of compression. This hardware takes two inputs?a low-poly model and a mathematical description of a curved surface?and outputs a very detailed, high-poly model. AMD's Natalya Tatarchuk showed a jaw-dropping demo of the tessellator in action, during which I kept thinking to myself, "Man, I wish she'd switch to wireframe mode so I could see what's going on." Until I realized the thing was in wireframe mode, and the almost-solid object I was seeing was comprised of millions of polygons nearly the size of a pixel.
This tessellator may live in a bit of an odd place for this generation of hardware. It's not a part of the DirectX 10 spec, but AMD will expose it via vertex shader calls for developers who wish to use it. We've seen such features go largely unused in the past, but AMD thinks we might see games ported from the Xbox 360 using this hardware since the Xbox 360 GPU has a similar tessellator unit. Also, tessellation capabilities are a part of Microsoft's direction for future incarnations of DirectX, and AMD says it's committed to this feature for the long term (unlike the ill-fated Truform feature that it built into the original Radeon hardware, only to abandon it in the subsequent generation). We'll have to see whether game developers use it.