MP3 vs AAC vs OGG: A Simple Quantitative Comparison

[drteming]

I recently got a new car with a radio that accepts a SD card. The system supports mp3, aac, ogg and wma formats. In trying to figure out what format I should use when ripping my CD's and transcoding FLAC's to put on the SD card, I did a bit of research. I read that aac is the better format, achieving audio transparency at a lower bit rate, but all the tests that I've seen have been subjective. My ears (and audio equipment) are not up to the task of differentiating between the various formats and bitrates, so this is what I came up with:

I ripped a song from a CD's to an uncompressed wav file. I then transcoded it to lossy formats:

MP3 VBR q0 (266kbps-6,667 KB)
MP3 VBR q2 (227kbps-5,691 KB)
MP3 VBR q4 (178kbps-4,478 KB)

AAC VBR q6.9 (267kbps-6,723 KB)
AAC VBR q6.1 (228kbps-5,758 KB)
AAC VBR q5.0 (178kbps-4,511 KB)

OGG VBR q7.5 (268kbps-6,708 KB)
OGG VBR q6.9 (228kbps-5,712 KB)
OGG VBR q5.2 (179kbps-4,498 KB)

I used fre:ac to transcode the mp3 and ogg (LAME encoder v3.98.4 and Ogg Vorbis encoder v1.3.2) and used NeroAacEnc for the aac files. I tried to keep the aac and ogg files the same size and bitrate as the mp3 by adjusting the quality settings. I didn't include wma because the encoder won't spit out files with comparable size and bitrate as the others.

After I had all my files, I opened up Audacity (1.3.11 beta) and imported the original uncompressed wav file. I then imported the compressed files individually, and using the time shift tool, lined up the wave forms (only needed in mp3 and aac).

I then inverted the compressed waveforms and mixed it with the uncompressed waveform, subtracting one from the other, resulting in a wave form representing the audio that was lost during compression.

The picture below shows the 3 residual wave forms from the medium bit rate files, mp3 on top, aac in the middle and ogg on the bottom.

Just by visual inspection, aac has less audio that was lost.

I then measured the RMS volume of each of the residual audio waves as a quick and dirty way of quantifying the amount of audio that was lost. The lower the volume (higher on the graph), the less audio was lost.

It looks like aac is the clear winner. In fact, aac at 178 kbps beat out ogg at 228 kbps and came awfully close to mp3 at 227 kbps.

I don't know how valid my conclusion is as I am a total amateur, and I only tried this on one song and one style of music. The results may change with other songs and analytic techniques.

Now, the 64 dollar question is: can I tell the difference subjectively? The answer is no. But this little project kept me out of my wife's hair for the afternoon, and to her, it was well worth it.

 

[0roo0roo]

I dunno, lossy audio compression is based on perceptual reproduction, so whether the "lost" bit makes a difference I don't know. Its the result of a subtraction from what was never meant to be an identical reproduction in the first place.

 

[JAG87]

Nice comparison.

You should add WMA Pro, I think it's more efficient than Vorbis (at least from my experience at low bitrates) and probably on par with AAC.

Also, it seems very odd that MP3 beats Vorbis at high bitrates. I would never expect that.

 

[drteming]

Nice comparison.

You should add WMA Pro, I think it's more efficient than Vorbis (at least from my experience at low bitrates) and probably on par with AAC.

Also, it seems very odd that MP3 beats Vorbis at high bitrates. I would never expect that.

WMA 9.2 q95 (345kbps-8,574 KB): -41.3 dB
WMA 9.2 q90 (195kbps-4,769 KB): -33.7 dB

WMA 10 pro q95 (305kbps-6,950 KB): -41.3 dB
WMA 10 pro q90 (195kbps-4,290 KB): -33.8 dB

At the high bitrates, the WMA's beat out AAC, but at larger file size. At the low bitrates, AAC won out. Of note, Audacity does not recognize WMA 10 pro, so I had to transcode the WMA 10 pro files back to WAV before importing into Audacity. Also, none of my portable media players support WMA 10 pro.

I redid the high bitrate MP3 vs OGG with a different song of a different style:

MP3 VBR q0 (247kbps-3,149) KB -39.5 dB
OGG VBR q7.1 (247kbps-3,141) KB -39.0 dB

The latest LAME is really good.

 

[JAG87]

Is there any reason why you didn't use CBR? It would make it a lot easier to compare. Also like someone else said, your quantitative comparison is good up to a certain point. It really depends on what is discarded by each codec, whether you can hear it and how much difference it makes.

 

[ChAoTiCpInOy]

Very good work sir.

 

[drteming]

Is there any reason why you didn't use CBR? It would make it a lot easier to compare.

I've always thought CBR was inefficient(?), endcoding the quiet parts at the same bit rate as more complex parts. I figured efficiency is a direct reflection of the quality of a codec and the encoder.

Also like someone else said, your quantitative comparison is good up to a certain point. It really depends on what is discarded by each codec, whether you can hear it and how much difference it makes.

I totally agree. Like I said, I can't tell the difference between any of the encoded songs by my ears. It was a fun afternoon playing on the computer and drinking beer.

 

[JAG87]

I've always thought CBR was inefficient(?), endcoding the quite parts at the same bit rate as more complex parts. I figured efficiency is a direct reflection of the quality of a codec and the encoder.

Yes it is inefficient, but it makes it a lot easier to compare codecs.

VBR puts in so many variables. First, it's impossible to obtain the same bitrate between two codecs (nor the file size), second, the exact same quality setting (even on the same codec) will have huge variations depending on the content of what is being encoded.

With CBR, you take bitrate and content out of the equation, and you analyze the codec purely on it's output and how transparent it is compared to the original.

If one codec is better than another at 256K CBR for example, one would figure that a VBR encode at roughly ~256K would also exhibit the same properties while being more efficient on the file size.

 

[Mr. Pedantic]

So how valuable is a quantitative analysis? As some people have pointed out before, different codecs have different ways to discard what they determine as unnecessary for a certain compression level, and while it may be that two codecs may discard the same amount, one codec's discarded material could be a lot more important to the actual sound than the other's.

If one codec is better than another at 256K CBR for example, one would figure that a VBR encode at roughly ~256K would also exhibit the same properties while being more efficient on the file size.

From what you said previously in the exact same post, this cannot be necessarily true.

 

[drteming]

I did some CBR comparison using the original song. Same results, but OGG does come out ahead of MP3. The WMA encoder does not have settings for CBR bitrate above 192kbps. Interestingly, WMA 10 pro, like OGG, does not have frame shift like AAC, MP3 and WMA 9.2.


AAC 192kbps-4.850KB: -36.9 dB
MP3 192kbps-4,812KB: -32.7 dB
OGG 192kbps-4,834KB: -35.1 dB
WMA 9.2 192kbps-4,838KB: -34.3 dB
WMA 10 pro 192kbps-4,838KB: -36.6 dB

AAC 256kbps-6,453KB: -40.2 dB
MP3 256kbps-6,415KB: -37.3 dB
OGG 256kbps-6,446KB: -38.5 dB

 

[JAG87]

Ah, your 192K results pretty much mirror what my experience has been.

MP3 is the worst, OGG is better, WMA pro and AAC are the best. And the lower the bitrates the more pronounced is the difference.

At 256K even though they are all good enough, the pattern is the same.

 

[lamedude]

I've always thought CBR was inefficient(?), endcoding the quiet parts at the same bit rate as more complex parts.

MP3's bit reservoir can free up some bits for later more complex parts.

 

[fire400]

wma if done right, can achieve dvd sound quality.

mp3 is fine if it's 128k on a good system.

 

[Phynaz]

So how valuable is a quantitative analysis?

Worthless. Doesn't say a thing about how the file sounds.

You can learn more from the pros at http://www.hydrogenaudio.org.

 

[vvneagleone]

... to read, but most people don't realize that sample heights aren't DIRECTLY related to sound. For example (just an example!) the mp3 may differ more from the wav but MAY still sound more accurate when PLAYED... a comparison like this needs detailed audio experiments for real results. Thanks anyway. AAC will almost certainly be better for most songs...

 

[Midwayman]

Check out
http://www.stereophile.com/features/308mp3cd

A much better quantitative analysis.

 

[Anarchist420]

I really, really wish lossy compression would die out. IMO, it's not necessary, never was (considering the fact that CDDA was once used and there were never any problems with that), and the quality is pisspoor compared to lossless especially if it's music you like. Same thing for textures, lossy textures can be blotchy as hell.

I've always found it odd how we went straight from raw uncompressed CDDA to 128kbps mp3s.

 

[Modelworks]

AAC is what I prefer if I can't do FLAC . AAC is one of the codecs that you really need to have the proper software to get the best results. AAC has so many options and configurations that if not set correctly can produce poor results. One of the biggest features of AAC is in built error correction for artifacts that you would normally get by using mp3.

 

[Midwayman]

I've always found it odd how we went straight from raw uncompressed CDDA to 128kbps mp3s.

Not a surprise if you remember the storage capacities back in the day. Today there isn't any real good reason not to use flac. You can still get fit a very useful amount of audio on most players. Hell even uncompressed CDDA could fit quite a bit on most players.

 

[0roo0roo]

First mp3 players couldn't even really hold an album, 32MB? Lol

 

[mmntech]

First mp3 players couldn't even really hold an album, 32MB? Lol

Roughly 30 minutes worth at 128kbps. It's why all early successful players were hard drive based. Kind of makes me want to buy an iPod Classic for nostalgia's sake. Those things were king back in the day.

 

[Anarchist420]

Not a surprise if you remember the storage capacities back in the day. Today there isn't any real good reason not to use flac. You can still get fit a very useful amount of audio on most players. Hell even uncompressed CDDA could fit quite a bit on most players.

Not really sure what you're talking about. Storage capacity back then was less than it is now. We technically have unlimited strorage cap today because of digital downloads.

 

[sm625]

I then inverted the compressed waveforms and mixed it with the uncompressed waveform, subtracting one from the other, resulting in a wave form representing the audio that was lost during compression.

But if you are subtracting one from the other then aren't you also gaining unwanted audio at certain points? When the lossy format adds amplitude that should not be there, then you get a positive number when you subtract. That positive number could add to your score and make one codec look better than it actually is. Are you taking the absolute value after your initial subtraction?

 

[drteming]

... Are you taking the absolute value after your initial subtraction?

I then measured the RMS volume of each of the residual audio waves as a quick and dirty way of quantifying the amount of audio that was lost.

But as I said before, I don't know how valid my conclusion is. There was a lot of beer involved.

 

[0roo0roo]

Not a surprise if you remember the storage capacities back in the day. Today there isn't any real good reason not to use flac. You can still get fit a very useful amount of audio on most players. Hell even uncompressed CDDA could fit quite a bit on most players.

Not really, on the pc you can keep double copies sure, but on a portable? There are no 1TB iphones. Furthermore no one is passing a blind test with earbuds between lossy and lossless. and of course big files rape your battery life..