POLL: Blind MP3 encoder test (RESULTS POSTED)

mithrandir2001

Diamond Member
May 1, 2001
6,545
1
0
As stated in this thread, we're going to have another MP3 encoder blind test.

This is how it works. I am hosting 5 WAV files, each a little over 1MB. Each of them is named after a color for anonymity. Download each one and rank them in order of preference. One of them is the original WAV (a segment of "Sun" by Live's The Distance to Here) while the other 4 were encoded to a lossy format and then decoded back to WAV. To make things more "scientific", I won't tell you what formats I used...I may have used MP3 for all 4 or I could have thrown in some WMA or I could have used all Ogg Vorbis...basically, you don't know.

For the poll above, select which of the 5 is the original WAV. Then reply to the thread, listing your preference, adding comments if desired.

e.g.

Orange
Brown
White
Purple
Grey

(Yes, I used different names above.)

blue.wav - MusePack 0.90s standard (170kbps VBR MPC)
red.wav - LAME 3.91 -b 128 -q2 (128kbps CBR MP3)
yellow.wav - LAME 3.91 --alt-preset insane (320kbps CBR MP3)
green.wav - LAME 3.91 --alt-preset standard (265kbps VBR MP3)
black.wav - Original WAV

I will reveal the true identities in a few days, so don't rush.

---

The correct answer is black.
Those who picked yellow were very close since 320kbps MP3 is almost always transparent.
Those who picked red need some listening lessons because that file sounds the worst by far (I picked it out 16/16 on ABX). You can hear the swishy artifacts throughout the clip.

I'm surprised most people did not like blue, which uses my new preferred encoder, MusePack (MPC/MP+). I think it sounds remarkably transparent given the bitrate and it has the advantage of being a time-domain encoder, not a frequency-domain encoder like LAME MP3. This means when you decode the MPC back to WAV, the filesize is the same. Notice how blue and black (the original) both are 1135KB while the other three are 1139KB? MP3 adds frames to the beginning of each file and this throws off the track-by-track timing if you reassemble them into an album, for instance. MPC is the best when it comes to preserving transients and other time-sensitive acoustical events and I suppose my ears prefer the artifact masking of MPC to MP3. MusePack has been dubbed "the audiophile format".

Anyway, I did not really expect anybody to accurately pick which one of the 5 files was the original and I personally would have picked "unable to determine". Unless you have special hearing, it is very difficult to hear any differences between the files, except for red or perhaps green.
 

squirrelman

Senior member
Jan 1, 2001
869
0
76
Wow they all do sound pretty close to me. Now I dont claim to have the best ears but I do have great headphones ;) I did think that yellow.wav sounded a little better to me. Other than that one I couldnt tell much difference between any of them really.
 

DaLeroy

Golden Member
Dec 4, 2000
1,406
0
0


<< As stated in this thread, we're going to have another MP3 encoder blind test.

This is how it works. I am hosting 5 WAV files, each a little over 1MB. Each of them is named after a color for anonymity. Download each one and rank them in order of preference. One of them is the original WAV (a segment of "Sun" by Live's The Distance to Here) while the other 4 were encoded to a lossy format and then decoded back to WAV. To make things more "scientific", I won't tell you what formats I used...I may have used MP3 for all 4 or I could have thrown in some WMA or I could have used all Ogg Vorbis...basically, you don't know.

For the poll above, select which of the 5 is the original WAV. Then reply to the thread, listing your preference, adding comments if desired.

e.g.

Orange
Brown
White
Purple
Grey

(Yes, I used different names above.)

blue.wav
red.wav
yellow.wav
green.wav
black.wav

I will reveal the true identities in a few days, so don't rush.
>>



Not gonna do this at work....got sh*tty speakers. I'll download at home and burn to a cd, and play on my home stereo :)
 

OS

Lifer
Oct 11, 1999
15,581
1
76

huh, awfully silent in this thread, I wonder where the "MP3s sound like crap" bandwagon went to. Must have made a wrong turn somewhere. :D
 

MikeO

Diamond Member
Jan 17, 2001
3,026
0
0

I'll do it when I get home. I have some damn crappy 10W Juster "speakers" at work.. 64kbit mp3 sounds exactly like audio CD so... :D
 

notfred

Lifer
Feb 12, 2001
38,241
4
0
downlaoding now. I think everyone who claims to be able to tell the difference between CD and 256kbit MP3, and 256 and 192, and 192 and 128, etc, should be rounded up, and publicly ridiculed if they get the answer wrong.
 

Ameesh

Lifer
Apr 3, 2001
23,686
1
0
my headphones are at work, i'll try it out tommorow.






btw wm8 ownz jo0! wm9 will h4x0r jo0!
 

chiwawa626

Lifer
Aug 15, 2000
12,013
0
0
grr i voted yellow..but im no audiophile nor do i know crap about stuff....but i wanna know!! can u post the answer or email me..i wont tell anyone..
 

silverpig

Lifer
Jul 29, 2001
27,703
12
81
I think it's black, although yellow sounds pretty good too. It'd be easier to tell with a different selection though. There was a lot of distortion and pulsing bass in that clip that it makes it harder.

If I'm right, it's cause of my Sennheisers :)

If I'm wrong, it's cause their wireless Sennheisers :)
 

notfred

Lifer
Feb 12, 2001
38,241
4
0
no idea, couldn't tell a difference. I did, however, notice that my radeon card interferes with my SB live card, makes noises hen you scroll, minmize windows, etc. Maybe I should move the SB Live to the bottom PCI slot? Any ideas on this?
 

mithrandir2001

Diamond Member
May 1, 2001
6,545
1
0


<< It'd be easier to tell with a different selection though. There was a lot of distortion and pulsing bass in that clip that it makes it harder. >>


Yea, it's a busy clip but it seems to be rife with opportunities for an encoder to screw up since the treble oscillates actively.

If I do another one of these tests, I may pick a simple clip with acoustic guitar or something with transients so that we can test for pre-echo artifacts.
 

kami

Lifer
Oct 9, 1999
17,627
5
81
Can't tell from that clip...i think i need more than 6 seconds and more range of sounds to be able to do any serious comparisons.

this does attest to the fact that compression done properly is very good and people who shun MP3 are full of sh*t...and btw, my audio setup is around $2k total
 

RedFox1

Senior member
Aug 22, 2000
587
0
76
Just want to bump this to the top -- I'm interested in seeing the (tainted by Yakko ;)) results.

-Russ
 

worth

Platinum Member
Feb 4, 2001
2,369
0
0
I listened to them, and yellow seeemed to sound a little better, followed by black. I couldn't really tell the difference between the rest, but blue sounded like it had some noise in it, so it was the worst.

Then again, I claimed I can't tell the difference between 128kbps and 192kbps.
 

pulse8

Lifer
May 3, 2000
20,860
1
81
Why do I have the feeling that they are all the same wav file and he's just playing with our minds? :p
 

mithrandir2001

Diamond Member
May 1, 2001
6,545
1
0
I don't want to encourage "cheating", but if you brought up each file under a spectro analyzer (like Nero or CoolEdit), you'd see that they are each different...sometimes, rather different. Of course, just because they look different under the "scope" doesn't mean our ears will be able to hear the difference. That's whole premise of psychoacoustics.

One hint, though. If you compare these over ordinary computer speakers, I doubt you can hear much difference at all between the five. Instead, plug a good pair of headphones into your sound card, or better yet, burn the WAVs to a CD and listen to them on a CD player in a quiet environment...that's a better way to compare.

If you think they all sound the same, then you need a better playback environment. On one particular file, I was able to ABX it against the original WAV 16 out of 16 times. There's a tool at PCABX that can do ABX tests on your PC...can be quite handy.
 

silverpig

Lifer
Jul 29, 2001
27,703
12
81
<gloat>Sweet. I was right. And I pegged yellow to be a close second too. All bow down to the master. :)</gloat>
 

fitzhue

Golden Member
Sep 24, 2000
1,242
0
71
do u have any more info on that musepak encoder? i'd like to try out making some files from that. will the files work in winamp or will i have to use some plugin to make it work right? sounds kinda interesting.