Wow, some pretty long winded posts... guess I'll add mine
I am an audiophile but more importantly a music lover. Before burning out on it, I spent 20 years in the high-end audio industry, most of it as a trainer/teacher. My system includes components that most people would consider to be way "over the top," but to some audiophiles it's low-end. (Audio Research tube preamp, New York Audio Labs Moscode tube amp, Linn Sondek turntable, Wadia D/A converter, etc.) My two cents (equivalent to 50 dollars if sold by a high-end audio store)...
1) The difference between an audiophile and a music lover, in my opinion, is that an music lover listens to music and an audiophile to equipment. Many audiophiles spend more time listening to the "air around the cymbals" (how well they are reproduced) than to the way the drummer is playing them. They're so busy tweaking their equipment and comparing tiny nuances that the music becomes secondary. They're missing the whole point. If the music moves you on the equipment that you have, then enjoy the music, even if it's playing on a boombox. On the other hand, it's worth the experience of listening to a state-of-the-art system once to see what you're missing. It's like driving a Porsche once in your life even if you own a Chevy. You may or may not decide that the experience is worth the extra money (or even have the extra money), but at least you've had the experience.
2) High-end audio reaches a point of diminishing returns. Again, using the car analogy, a particular $40k car may go twice as fast as a $20k car, but a $200k car will not go 10 times as fast. (Okay, I'm simplifying, but you get the point.) There are some people who will spend boatloads of money to eek out that extra 2%. More power to them, if they can afford it and that's how they want to spend their money. If you hear he difference, and feel that it's worth the investment, then enjoy it. If not, spend your money on something else.
3) Specs can't accurately measure musicality. Again, a car analogy. Both a Rolls Royce and a Ferrari are exceptional cars, but they couldn't be more different. Each does things far better than a Ford Escort, but in different ways. Many of these cannot be measured. How do you translate the "feel" of these two cars into specs? Even if you could measure every possible spec how would you determine which particular combination was best for everybody? High-end audio is the same way. Also, there are ways of cheating on specs. The way the most common audio specs are measured have little bearing on how the component performs in the "real world" playing "real music." Mass-market audio manufacturers often design components to look good on a spec sheet, whereas some high end products "measure" worse but sound better because the engineers worried more about how they sounded playing music on speakers than how they measured playing pink noise into a resistor.
4) A lot of it comes down to how well the particular components in your system, and the room acoustics, work together. I've heard systems that were comprised of exceptional components that each sounded great in their own right, but sounded really bad together. On the same note, a double blind test proves only that a certain combination of equipment sounded better to a certain group of people playing certain music in a certain room.
5) Even then, if trying to judge what sounds better by listening (either by double-blind test or informally), what is your point of reference? Unless you know how that recording sounded in the recording studio when the engineer mixed it down (using electronics and speakers that themselves imprinted their own sound), then how can you judge what component sounds closer to "the real thing?" Unless you happen to have the actual artist in the room playing live next to you, it's all a "best guess" anyway.
6) So you can't go by specs. You can't go by double-blind testing. And you may or may not notice a difference from listening. So how do you know what's best for you? It's easy. Remember my firs point. It's all about enjoying the music (or movie), not the equipment. If what you have brings you "into" the music - involves you - then it's right for you.