Actually, running a firewall on a classic server is pretty well known - Checkpoint made it's name as an app that was installed on NT and Solaris. Only in the last year or so have the non-server-based appliances come out and gotten popular.
The advantages of a dedicated appliance are many. First, they don't have an OS, so they don't have that level of vulnerability. Granted, a well-patched Linux box should be pretty safe, but not everyone aggressively keeps up with their patches.
Next, the boxes perform much better. The are custom-made to do ONE thing, and that's it. No legacy OS to get in the way, just fast performance.
Lastlly, I don't know about you, but I don't really like the idea of an open source firewall. If the guys in black can get the source to your firewall, they can pick through it and find any possible holes or ways to get around it. You're much better off with a commercial product that nobody sees the source to and that's rigorously tested by a company whose future depends on the box running fast, securely and stably.
- G