MP3 vs AAC vs OGG: A Simple Quantitative Comparison

Discussion in 'Audio/Video & Home Theater' started by drteming, May 30, 2011.

  1. drteming

    drteming Senior member

    Joined:
    May 9, 2005
    Messages:
    694
    Likes Received:
    0
    I recently got a new car with a radio that accepts a SD card. The system supports mp3, aac, ogg and wma formats. In trying to figure out what format I should use when ripping my CD's and transcoding FLAC's to put on the SD card, I did a bit of research. I read that aac is the better format, achieving audio transparency at a lower bit rate, but all the tests that I've seen have been subjective. My ears (and audio equipment) are not up to the task of differentiating between the various formats and bitrates, so this is what I came up with:

    I ripped a song from a CD's to an uncompressed wav file. I then transcoded it to lossy formats:

    MP3 VBR q0 (266kbps-6,667 KB)
    MP3 VBR q2 (227kbps-5,691 KB)
    MP3 VBR q4 (178kbps-4,478 KB)

    AAC VBR q6.9 (267kbps-6,723 KB)
    AAC VBR q6.1 (228kbps-5,758 KB)
    AAC VBR q5.0 (178kbps-4,511 KB)

    OGG VBR q7.5 (268kbps-6,708 KB)
    OGG VBR q6.9 (228kbps-5,712 KB)
    OGG VBR q5.2 (179kbps-4,498 KB)

    I used fre:ac to transcode the mp3 and ogg (LAME encoder v3.98.4 and Ogg Vorbis encoder v1.3.2) and used NeroAacEnc for the aac files. I tried to keep the aac and ogg files the same size and bitrate as the mp3 by adjusting the quality settings. I didn't include wma because the encoder won't spit out files with comparable size and bitrate as the others.

    After I had all my files, I opened up Audacity (1.3.11 beta) and imported the original uncompressed wav file. I then imported the compressed files individually, and using the time shift tool, lined up the wave forms (only needed in mp3 and aac).

    [​IMG]

    I then inverted the compressed waveforms and mixed it with the uncompressed waveform, subtracting one from the other, resulting in a wave form representing the audio that was lost during compression.

    The picture below shows the 3 residual wave forms from the medium bit rate files, mp3 on top, aac in the middle and ogg on the bottom.

    [​IMG]

    Just by visual inspection, aac has less audio that was lost.

    I then measured the RMS volume of each of the residual audio waves as a quick and dirty way of quantifying the amount of audio that was lost. The lower the volume (higher on the graph), the less audio was lost.

    [​IMG]

    It looks like aac is the clear winner. In fact, aac at 178 kbps beat out ogg at 228 kbps and came awfully close to mp3 at 227 kbps.

    I don't know how valid my conclusion is as I am a total amateur, and I only tried this on one song and one style of music. The results may change with other songs and analytic techniques.

    Now, the 64 dollar question is: can I tell the difference subjectively? The answer is no. But this little project kept me out of my wife's hair for the afternoon, and to her, it was well worth it.
     
  2. Loading...

    Similar Threads - Simple Quantitative Comparison Forum Date
    Hopefully an easy answer to a simple question! Audio/Video & Home Theater Aug 24, 2015
    What's a super simple TV system nowadays? Audio/Video & Home Theater Jul 24, 2013
    Projector and In Ceiling Speaker Comparison Audio/Video & Home Theater Jun 20, 2013
    Help: Easy, simple setup for streaming to TV? Audio/Video & Home Theater Sep 22, 2012
    SHOULD BE A SIMPLE SOLUTION: 3-prong grounded to 2-prong AC Audio/Video & Home Theater Apr 8, 2012

  3. 0roo0roo

    0roo0roo No Lifer

    Joined:
    Sep 21, 2002
    Messages:
    64,806
    Likes Received:
    47
    I dunno, lossy audio compression is based on perceptual reproduction, so whether the "lost" bit makes a difference I don't know. Its the result of a subtraction from what was never meant to be an identical reproduction in the first place.
     
  4. JAG87

    JAG87 Diamond Member

    Joined:
    Jan 3, 2006
    Messages:
    3,921
    Likes Received:
    3
    Nice comparison.

    You should add WMA Pro, I think it's more efficient than Vorbis (at least from my experience at low bitrates) and probably on par with AAC.

    Also, it seems very odd that MP3 beats Vorbis at high bitrates. I would never expect that.
     
  5. drteming

    drteming Senior member

    Joined:
    May 9, 2005
    Messages:
    694
    Likes Received:
    0
    WMA 9.2 q95 (345kbps-8,574 KB): -41.3 dB
    WMA 9.2 q90 (195kbps-4,769 KB): -33.7 dB

    WMA 10 pro q95 (305kbps-6,950 KB): -41.3 dB
    WMA 10 pro q90 (195kbps-4,290 KB): -33.8 dB

    At the high bitrates, the WMA's beat out AAC, but at larger file size. At the low bitrates, AAC won out. Of note, Audacity does not recognize WMA 10 pro, so I had to transcode the WMA 10 pro files back to WAV before importing into Audacity. Also, none of my portable media players support WMA 10 pro.

    I redid the high bitrate MP3 vs OGG with a different song of a different style:

    MP3 VBR q0 (247kbps-3,149) KB -39.5 dB
    OGG VBR q7.1 (247kbps-3,141) KB -39.0 dB

    The latest LAME is really good.
     
  6. JAG87

    JAG87 Diamond Member

    Joined:
    Jan 3, 2006
    Messages:
    3,921
    Likes Received:
    3
    Is there any reason why you didn't use CBR? It would make it a lot easier to compare. Also like someone else said, your quantitative comparison is good up to a certain point. It really depends on what is discarded by each codec, whether you can hear it and how much difference it makes.
     
  7. ChAoTiCpInOy

    ChAoTiCpInOy Diamond Member

    Joined:
    Jun 24, 2006
    Messages:
    6,442
    Likes Received:
    0
    Very good work sir.
     
  8. drteming

    drteming Senior member

    Joined:
    May 9, 2005
    Messages:
    694
    Likes Received:
    0
    I've always thought CBR was inefficient(?), endcoding the quiet parts at the same bit rate as more complex parts. I figured efficiency is a direct reflection of the quality of a codec and the encoder.

    I totally agree. Like I said, I can't tell the difference between any of the encoded songs by my ears. It was a fun afternoon playing on the computer and drinking beer.
     
    #7 drteming, Jun 1, 2011
    Last edited: Jun 1, 2011
  9. JAG87

    JAG87 Diamond Member

    Joined:
    Jan 3, 2006
    Messages:
    3,921
    Likes Received:
    3
    Yes it is inefficient, but it makes it a lot easier to compare codecs.

    VBR puts in so many variables. First, it's impossible to obtain the same bitrate between two codecs (nor the file size), second, the exact same quality setting (even on the same codec) will have huge variations depending on the content of what is being encoded.

    With CBR, you take bitrate and content out of the equation, and you analyze the codec purely on it's output and how transparent it is compared to the original.

    If one codec is better than another at 256K CBR for example, one would figure that a VBR encode at roughly ~256K would also exhibit the same properties while being more efficient on the file size.
     
  10. Mr. Pedantic

    Mr. Pedantic Diamond Member

    Joined:
    Feb 14, 2010
    Messages:
    5,040
    Likes Received:
    0
    So how valuable is a quantitative analysis? As some people have pointed out before, different codecs have different ways to discard what they determine as unnecessary for a certain compression level, and while it may be that two codecs may discard the same amount, one codec's discarded material could be a lot more important to the actual sound than the other's.

    From what you said previously in the exact same post, this cannot be necessarily true.
     
  11. drteming

    drteming Senior member

    Joined:
    May 9, 2005
    Messages:
    694
    Likes Received:
    0
    I did some CBR comparison using the original song. Same results, but OGG does come out ahead of MP3. The WMA encoder does not have settings for CBR bitrate above 192kbps. Interestingly, WMA 10 pro, like OGG, does not have frame shift like AAC, MP3 and WMA 9.2.


    AAC 192kbps-4.850KB: -36.9 dB
    MP3 192kbps-4,812KB: -32.7 dB
    OGG 192kbps-4,834KB: -35.1 dB
    WMA 9.2 192kbps-4,838KB: -34.3 dB
    WMA 10 pro 192kbps-4,838KB: -36.6 dB

    AAC 256kbps-6,453KB: -40.2 dB
    MP3 256kbps-6,415KB: -37.3 dB
    OGG 256kbps-6,446KB: -38.5 dB
     
    #10 drteming, Jun 1, 2011
    Last edited: Jun 1, 2011
  12. JAG87

    JAG87 Diamond Member

    Joined:
    Jan 3, 2006
    Messages:
    3,921
    Likes Received:
    3
    Ah, your 192K results pretty much mirror what my experience has been.

    MP3 is the worst, OGG is better, WMA pro and AAC are the best. And the lower the bitrates the more pronounced is the difference.

    At 256K even though they are all good enough, the pattern is the same.
     
  13. lamedude

    lamedude Golden Member

    Joined:
    Jan 14, 2011
    Messages:
    1,193
    Likes Received:
    3
    MP3's bit reservoir can free up some bits for later more complex parts.
     
  14. fire400

    fire400 Diamond Member

    Joined:
    Nov 21, 2005
    Messages:
    5,072
    Likes Received:
    2
    wma if done right, can achieve dvd sound quality.

    mp3 is fine if it's 128k on a good system.
     
  15. Phynaz

    Phynaz Diamond Member

    Joined:
    Mar 13, 2006
    Messages:
    9,829
    Likes Received:
    623
    Worthless. Doesn't say a thing about how the file sounds.

    You can learn more from the pros at http://www.hydrogenaudio.org.
     
  16. vvneagleone

    vvneagleone Junior Member

    Joined:
    Jul 9, 2011
    Messages:
    1
    Likes Received:
    0
    ... to read, but most people don't realize that sample heights aren't DIRECTLY related to sound. For example (just an example!) the mp3 may differ more from the wav but MAY still sound more accurate when PLAYED... a comparison like this needs detailed audio experiments for real results. Thanks anyway. AAC will almost certainly be better for most songs...
     
  17. Midwayman

    Midwayman Diamond Member

    Joined:
    Jan 28, 2000
    Messages:
    4,682
    Likes Received:
    24
  18. Anarchist420

    Anarchist420 Diamond Member

    Joined:
    Feb 13, 2010
    Messages:
    8,649
    Likes Received:
    0
    I really, really wish lossy compression would die out. IMO, it's not necessary, never was (considering the fact that CDDA was once used and there were never any problems with that), and the quality is pisspoor compared to lossless especially if it's music you like. Same thing for textures, lossy textures can be blotchy as hell.

    I've always found it odd how we went straight from raw uncompressed CDDA to 128kbps mp3s.
     
  19. Modelworks

    Modelworks Lifer

    Joined:
    Feb 22, 2007
    Messages:
    16,243
    Likes Received:
    2
    AAC is what I prefer if I can't do FLAC . AAC is one of the codecs that you really need to have the proper software to get the best results. AAC has so many options and configurations that if not set correctly can produce poor results. One of the biggest features of AAC is in built error correction for artifacts that you would normally get by using mp3.
     
  20. Midwayman

    Midwayman Diamond Member

    Joined:
    Jan 28, 2000
    Messages:
    4,682
    Likes Received:
    24
    Not a surprise if you remember the storage capacities back in the day. Today there isn't any real good reason not to use flac. You can still get fit a very useful amount of audio on most players. Hell even uncompressed CDDA could fit quite a bit on most players.
     
  21. 0roo0roo

    0roo0roo No Lifer

    Joined:
    Sep 21, 2002
    Messages:
    64,806
    Likes Received:
    47
    First mp3 players couldn't even really hold an album, 32MB? Lol
     
  22. mmntech

    mmntech Lifer

    Joined:
    Sep 20, 2007
    Messages:
    17,512
    Likes Received:
    8
    Roughly 30 minutes worth at 128kbps. It's why all early successful players were hard drive based. Kind of makes me want to buy an iPod Classic for nostalgia's sake. Those things were king back in the day.
     
  23. Anarchist420

    Anarchist420 Diamond Member

    Joined:
    Feb 13, 2010
    Messages:
    8,649
    Likes Received:
    0
    Not really sure what you're talking about. Storage capacity back then was less than it is now. We technically have unlimited strorage cap today because of digital downloads.
     
  24. sm625

    sm625 Diamond Member

    Joined:
    May 6, 2011
    Messages:
    8,175
    Likes Received:
    134
    But if you are subtracting one from the other then aren't you also gaining unwanted audio at certain points? When the lossy format adds amplitude that should not be there, then you get a positive number when you subtract. That positive number could add to your score and make one codec look better than it actually is. Are you taking the absolute value after your initial subtraction?
     
  25. drteming

    drteming Senior member

    Joined:
    May 9, 2005
    Messages:
    694
    Likes Received:
    0
    But as I said before, I don't know how valid my conclusion is. There was a lot of beer involved.
     
  26. 0roo0roo

    0roo0roo No Lifer

    Joined:
    Sep 21, 2002
    Messages:
    64,806
    Likes Received:
    47
    Not really, on the pc you can keep double copies sure, but on a portable? There are no 1TB iphones. Furthermore no one is passing a blind test with earbuds between lossy and lossless. and of course big files rape your battery life..