Android performance is strange. It's very fast in the sense that applications open quickly enough (most nearly instantly). Switching between windows, loading websites etc. is also extremely quick. However some animations and scrolling are choppy for no particular reason. Sometimes when bringing up the virtual keyboard, it appears in the wrong size first, then it's repositioned and re-scaled correctly. This all happens within a fraction of a second, but it's still noticeable. That's just one of many examples of this behavior. For example, why is scrolling between home screens super-smooth, unless I use the track pad? Why is scrolling through SMS-messages choppy, while Email and the Gallery scroll completely fluidly and smoothly? This makes the whole OS feel slower than it really is.
It's kind of like XP vs Vista/Win7. Moving and resizing windows, opening and closing windows etc. is actually faster in XP, but because it's not hardware accelerated or double-buffered, you get lots of visual strangeness like clipping, tearing, partially updated windows, hall of mirrors effects etc.
Android has however almost never crashed for me. Maybe all the constant tweaking, those unofficial experimental ROMs etc. explain your frequent crashes? I'm on the official 2.2 ROM from HTC, with just a few system applications installed. I do have a task killer, but I've only used it a few times to check CPU and memory usage. Never had to use it to actually kill tasks.