I don't understand the fascination, but I don't understand the hatred towards it either.
It's not all tentacle porn, Yu-Gi-Oh/Pokemon style battling, or Dragonball style senseless violence and redundancy. (Seriously, DragonballZ = Power Rangers as far as complexity and redundancy. [I particularly hate DBZ. I've never seen so many people so rabid over such a crappy cartoon before. Every episode I've ever seen was a five minute battle, padded to last the full half hour with mindless and nonsensical dialogue exchanged between good guy(s) and bad guy(s), then ended with one guy summoning a solar system sized energy ball of some sorts in an attempt to kill the other guy but ends up decimating a 500 mile radius piece of the earth instead. Ending with a dramatic "to be continued" that leaves fan boys ejaculating on their television sets.])
Part of my understanding is that you really need to see the original Japanese versions, in Japanese (with subtitles), to really get the full effect. Most of the people I've met who try to get me onto anime try to sell me on this specific point. Almost all anime shown in the states, translated into English, are considerably watered down to a kiddy level, or missing a lot of worthwhile content that's lost in translation. This turns into a separation of classes so to speak. People who buy the ridiculously expensive anime DVD's for original/full content, people who catch a few pieces of hentai porn and the occasional episode of Yu-Gi-Oh/DBZ/Pokemon/Digimon and just don't "get it", and the people who watch Yu-Gi-Oh/DBZ/Pokemon/Digimon and THINK that they know 'good' anime, so they act like rabid fan boys make everyone hate them and by proxy, anime itself.
(All of this is either my opinion, or my understanding. Nobody assume that I'm trying to lay down law here.)
Personally, I like some anime because I really like the animation style and more 'adult' themes. A lot of american cartoons are simply pathetic. Just watch the Kids WB for an afternoon or Cartoon Network almost any time of day outside of Adult Swim. Catch some 'Ed, Edd, and Eddie', 'Moocha Loocha' (sp?), or if anyone remembers the abomination of 'Molly-O' (the misadventures of a 5 year old girl rock star with a kangaroo drummer.) Just to name a few of the ones I hate the most. I love cartoons. I always have. But the golden cartoons like the old Merry Melodies, Bugs 'n Daffy, Tom and Jerry, etc, are all gone. Even the good (IMO) WB stuff is gone. I thought Animaniacs had a few great moments. Freakazoid and Pinky & the Brain were even brilliant at times with subtle humor and gags. (Many of which were aimed at adults, but the kids were cracking up at the more obvious humor.) While most anime isn't funny, (or at least not much of what I've seen), it's still a neat little area to explore. (I like Inuyashaa and Cowboy Beebop myself.)
Just keep in mind, most of any anime you've seen in the US isn't really 'good' anime, with a few exceptions. It's either kiddy crap, or designed just to market toys, (any Kids WB anime).
(Sorry for the long post, I didn't mean for it to go on and on.)