It supports MP3, MP4/AAC, ATRAC, WMA, WAV, and audio CDs. Older (pre-fat 40gb) support SACD as well. Sometimes it can be a bit fickle with bit rates for AAC files.
For movies, it supports up to DTS-MA and DolbyDigital TrueHD. The fat models can't bitstream high def audio though, not that it makes any real difference where it's decoded.