Are some of you people seriously fingering the GTX 780, 4850, 2900XT as some of the worst cards in history? I guess most of you got into PC gaming fairly recently, otherwise I just can't see how you would ever consider such cards as some of the worst GPUs of all time.
Lots of candidates for this one. i740 obviously, S3 Virge, Matrox Parhelia, etc.
Kind of obscure now, but the single worst GPU I've ever come across was the 3Dlabs Permedia 2. Creative Labs decided it would be a great idea to sell a gaming graphics card, the modestly named Graphics Blaster Exxtreme, based around a buggy, slow chip that was designed for OpenGL CAD work and wasn't fully compatible with DirectX.
...
People criticise today's GPUs, but at least they work. Back in the 90s that wasn't always true.
I mostly agree with your last sentence, however the Permedia 2 was a pretty decent card, it was never marketed as a gaming card per se. More so, it was a serious OpenGL card, that could actually run some games on the side and was actually decent at that! I have heard of a lot of fun stories of people gaming on their systems at work with such cards or having one at home for work and gaming on the side.
Quirky and somewhat obscure, but certainly not lame. Same goes for the i740. Massively overhyped, but decent card overall. More of a massive disappointment than a bad card honestly.
Focusing on 3D and not 2D because that would take ages, I have a few cards of my own:
-Nvidia's NV1
Triangles just happened to catch on way more than quads. I feel sorry for anyone that bought this card, it had pretty mediocre 2D and audio as well. Terrible support, but at least you could play some Saturn ports on it

.
-Alliance AT3D
If this card had come out in 1995 I could maybe excuse it somewhat, but it came out 1997. Performance was terrible and the card lacked a ton of features that were required by games at the time. End result is that IQ is usually horrible, with insane rendering errors and effects missing.
Have a look for yourselves:
http://vintage3d.org/at3d.php#sthash.AgE1Ui6K.dpbs
https://www.youtube.com/watch?v=ysfAAX3zXmM
Be sure to take a look at the galleries from Vintage3D, they are quite a treat!
-Cirrus Logic Laguna3D
Another POS, this time by Cirrus Logic. Lackluster performance and many rendering errors as above. On top of that, the card "featured" texture warping ala Playstation 1

.
http://vintage3d.org/cirrus.php#sthash.8OJVPygK.dpbs
https://www.youtube.com/watch?v=mbvEhJBzi84
https://www.youtube.com/watch?v=9wMkDHuT4WQ
-Number Nine Imagine 128
Again, terrible performance, late to the market, awful drivers and lack of features meaning games looked like crap or did not run at all.
http://vintage3d.org/n9.php#sthash.AYhol5Bk.dpbs
-Trident 3DImage 9850 and 9750
Performance was actually okay with these two, but IQ was terrible.
http://vintage3d.org/trident2.php#sthash.GrmF5eDR.dpbs
https://www.youtube.com/watch?v=oFF-P4vQ0to
https://www.youtube.com/watch?v=WYGlfYuKz60
All 4 of the cards I mentioned above came out at a time when 3D accelerators were maturing (especially in 1997). In the early years it was somewhat acceptable to experiment and come out with products with questionable longscale support (which is why I'm a bit reluctant to include the NV1). Early adopters were supposed to bite the bullet with these. But, by 1997, with so many great cards out on the market that supported standards such as Direct3D and OpenGL (and Glide for the lucky 3Dfx owners

) it was simply unacceptable to release video cards that pretended to be Direct3D compliant but lacked important features.
-Matrox Parhelia
Total mess of a card. Very expensive and underwhelming, despite some huge hype before it was released. Supposedly DX9 compatible, but it was missing features and never really got proper drivers because of that. Even with latest drivers it can be very buggy and glitchy with certain games.
Honorable mention: S3 Virge
I actually dig the 2D portion of it, but damn this was such a lousy 3D accelerator. Instantly one of the all time worsts, but there are far worse contenders. And hey, at least it displayed everything properly and was supported extensively because it achieved such worldwide spread.
Honorable mention 2: ATi Rage Fury MAXX
Really oddball dual chip card utilizing a couple of Rage 128 Pro chips. The performance was there actually and it came out before the Radeon did and was competitive with the original GeForce. The issue is that driver support only existed for Win9x (the way the chips worked just could not work with Windows 2000 or XP which came out later) and far more importantly, the frametimes were horrible and there was noticable stuttering.
Again, just an oddball card with terrible support.
Honorable mention 3: Nvidia's entire FX 5xxx series
That was pretty bad. I was a sad owner of an FX 5600 XT, a card that performed slower than the FX 5200. The 5700 and 5900 were somewhat decent and would have been better spins, but still weren't enough.
Certainly Nvidia's worst line of cards, but certainly not the worst of all time.