It depends. For most modern processors, the differences are fairly slight. Opterons are essentially the same as corresponding Athlon 64s and Athlon 64 X2s in most regards, but have done better in testing and may have SMP support enabled.
Xeons are all over the map. A number of recent-ish Xeons have had some beefy L3 cache to distinguish them from regular Pentium 4s. The newer Woodcrest Xeons are available with a faster Front Side Bus than corresponding Core 2 Duos. Some Netburst Xeons have slower a FSB than corresponding Pentium 4s. And again, Xeons generally support some sort of SMP configuration.
In addition to the above, note that memory requirements may be different for systems using these processors. Server chips also often use different sockets (J, F) from desktop parts.
Short answer: the exact differences vary, but are generally not huge. If you would be happy with the performance of an equivalent (-ish) desktop part, you'll probably be happy with the server part.