Codec shoot-out 2004
Table of content:
2: Test setup
3: Test 1: Matrix Revolutions (normal bandwidth) -- ultra high bandwidth
4: Test 2: Saving Private Ryan (normal bandwidth) -- ultra high bandwidth
5: Test 3: Futurama (normal bandwidth) -- ultra high bandwidth
7: Besides the codecs
8: Future outlook
Welcome ladies and gentleman, to yet another installment of my (in)famous codec comparisons.
It has been a full year since I last looked at a handful of video codecs and how they are suited for typical DVD backup scenarios. We've seen a lot of activity in the codec market in 2004: For one, there are now several H.264 / MPEG-4 AVC (referred to as AVC henceforth) encoders available. With the adoption of AVC as codec for the upcoming high definition DVD formats (be it Blu-ray or HD DVD), as well as its adoption into digital TV standards, I expect to see a lot of movement in that market in the years to come. Microsoft's WMV9 has also been approved for the upcoming high definition DVD formats, and Microsoft is making some inroads into the television business as well, but so far those are proprietary systems and tied to specific geographical areas.
In the MPEG-4 Advanced Simple Profile (referred to as ASP henceforth) area, we have seen major consumer electronic manufacturers release DVD players that can play ASP content (all with the exception of 3 point GMC which rules out the use of XviD's and NeroDigital's GMC for standalone playback). And improvement to existing codecs has been ongoing throughout the year.
Now a quick overview of what has happened since the last comparison with respect to this year's contestants:
As I have announced last time, I will no longer include DivX3 SBC in codec comparisons. I feel by now we have solutions that are superior to SBC, plus then there's the fact that DivX3 is a hack of Microsoft's first MPEG-4 (though not quite specs compliant) codec and thus stands on legally shaky ground.
Unfortunately, I also had to exclude Moonlight's AVC codec, even though I've been doing some tests prior to the submission deadline. One of the reasons was that their decoding filters interfered with playback of other content (even AviSynth scripts), plus they currently only write their AVC streams into TS (transport streams), which are suited for streaming. TS is not a suitable container for DVD backups though, and adds a significant overhead that would unfairly disadvantage this codec, because due to the additional overhead, the video bitrate would be much smaller than the corresponding video bitrate of every other contestant. Moonlight is working on an MP4 multiplexer for their content, so they might be included in another comparison in the future (hopefully by then also with a 2 pass rate control suited for local playback).
As in previous comparisons, codec makers were contacted two weeks prior to the start of encoding, and asked to provide a codec build to be tested, along with the suggested settings for my three movie sources. I have gotten suggested settings for every contestant, so if you think other settings would've yielded a more favorable result, you are more than welcome to take it up with the codec makers (and not with me.. I only used what I was told to). I'd like to thank all the people involved in this process, it has been a pretty intense two weeks (and I'm still in contact with most companies as the test proceeds).
All codecs were tested in a 2 pass setup using the settings suggested by the developers.
All testing was done on the following hardware:
AMD Athlon 64 3500+
Shuttle FN95 mainboard (part of the Shuttle SN95F5 barebone system)
2x512MB PC3200 Kingston HyperX DDR RAM CL2
HIS Excalibur Radeon 9600 XT
Philips 230W5B2 Display connected via DVI
To encode I used the following software:
DGIndex 1.0.12 & DGDecode.dll
AviSynth 2.55 for frameserving
VirtualDub 1.5.10 for encoding (see exceptions below))
Nandub 1.0 RC2 lumafix to multiplex VBR MP3 audio
Helix DNA Producer 10.1 (build number 10.1.0.390)
To encode RV10 I used AutoRV10. To encode to NeroDigital I got a special commandline encoder from ateme, which supports AviSynth input (beta testers of the Main Profile codec will know the tool).
Movies I encoded:
Revolutions - Region1, NTSC, length: 2h09, 185917 frames (@23.976 fps).
I will call this movie Matrix3 from here on
Saving Private Ryan - Region1, NTSC, length: 2h49, 243770 frames (@23.976fps). I will call this movie SPR from here on
Futurama Season 2 Episode 1 - Region2, PAL, length: 0h21, 41271 frames (@25.00 fps)
I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-preset 128 setting) for all movies, then tried to put Matrix3 onto 1 700MB CD, SPR onto 2 700MB CDs and 4 Futurama Episodes onto one 700 MB CD (resulting in 175MB for an episode). The size of the audio track was 117'263 KB, 157'455 KB and 20'582 KB respectively. This resulted in an effective video bitrate of 621kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit). I used GordianKnot for the bitrate calculations (it can calculate the effective overhead of using VBR MP3 audio).
Since there's no VFW codec for RV10 and the ND codecs, I had to use different audio streams for those. In case of NeroDigital, I used Nero's own AAC encoder to create a 128kbit/s CBR audio track (122'481 KB, 160'441 KB and 20'489KB). Since NeroDigital uses the MP4 container, I had to perform special bitrate calculations for that codec. Ateme told me to expect a 10.4 byte overhead per frame which, assuming the audio was 128kbit/s (turns out later that the audio was a bit oversized which resulted in a slightly oversized video as well - the Matrix3 audio bitrate was 129.40 kbit/s and the SPR audio bitrate was 129.27 kbit/s), I got an effective bitrate of 627kbit/s for Matrix3, 1025kbit/s for SPR and 1000kbit/s for Futurama. In case of RV10, AutoRV10 took care of the bitrate calculations, and I just settled for a 128kbit/s AAC audio track, so I expect the RV10 bitrate to be in line with NeroDigital.
I also got different bitrates from Microsoft: 595 kbit/s for Matrix3, 1000 kbit/s for SPR and 900 kbit/s for Futurama. Obviously I asked why I shouldn't take my usual calculated bitrates and was told to use those bitrates to reach my target size. Could it be that Microsoft doesn't trust its own rate control?
As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got:
|3ivX||718'302 KB||1'433'300 KB||178'254 KB|
|DivX||717'618 KB||1'434'812 KB||179'188 KB|
|HDX4||717'090 KB||1'434'104 KB||179'050 KB|
|ND Main Profile||718'923 KB||1'436'367 KB||179'515 KB|
|ND High Profile||718'929 KB||1'436'368 KB||179'509 KB|
|RV10||713'441 KB||1'432'701 KB||178'012 KB|
|VP6||742'396 KB||1'482'878 KB||178'884 KB|
|VSS||728'834 KB||1'452'308 KB||165'370 KB|
|WMV9||702'928 KB||1'440'128 KB||168'370 KB|
|XviD||716'786 KB||1'433'996 KB||179'132 KB|
Note that 700MB equals 716'800KB, 1400MB equals 1'433'600KB and 175 MB equals 179'200 KB. Please note that I assumed that VP6 uses kbit = 1000 bit, so my bitrates for Matrix3 and SPR were too high (I had to re-encode Futurama so I could correct that mistake). The effective final size for the VP6 codec with proper bitrate should be: 728'289 KB for Matrix3 and 1'452'626 KB (but as you can see, both are still severely oversized). I've marked every size in red if it is more than 2.5 MB above target size (2.5 MB is about the oversize a CD can have and you can still burn it without having to activate overburning). The ones marked in bright yellow are undersized more than 2.5 MB (that points to a rate control issue as well but is less problematic than oversized files).
As you can see, 3ivX, DivX, HDX4, both ND codecs and XviD reach the target size. RV10 got a bit undersized, but it'll definitely fit onto CDs. VP6 got severely oversized (except for Futurama which interestingly was on target the 2nd time I encoded it - the first time it was off by more than 5 MB), and thus at that point is not very well suited for DVD backups until On2 revisits the rate control. WMV9 was undersized twice, in line with the lowered suggested bitrate, but still got oversized beyond the allowable limit once. So, for backup purposes I also have to put a question mark right there. VSS also has some serious rate control issues and delivered two oversized and one undersized file.
Codec settings (applied w.r.t. the default codec settings)
Here I took Matrix3 as an example. The resolution of SPR was bigger, hence the performance also dropped. Note that you should take these values with a grain of salt. Only the first 10'000 frames were used for speed measures so the overall average FPS might be a bit different. Furthermore, I did not reset the PC before each measurement and I did not perform several measurements per codec. For good absolute values, more testing would be required, but these values should nevertheless give you a good impression of the relative speed difference between codecs. Note that the values are averaged over the two passes where applicable.
|ND MP||19.36 fps (31.40 fps)|
|ND HP||15.79 fps (25.72 fps)|
I've taken the liberty to split the speed into three categories: above 2x real-time speed (green), in between 1x and 2x real-time speed (white) and below 1x real-time speed (red). Even though quality is more important than speed, some encoders really severely taxed my nerves. In the end I'll also talk about quality per FPS, and as it turns out, a high FPS is not necessarily an indicator of bad quality, and likewise, a low FPS is not necessarily an indicator of good quality.
A few noteworthy things: DivX's Insane mode during 2nd pass contributed to
the low FPS. I was told to use that mode, but perhaps a faster mode could give
a more balanced picture without considerable loss of quality. Likewise, for
VP6, the best quality mode could be replaced by the standard mode and this would
most likely not lead to a significant quality reduction. Also note that the
two values in brackets for the NeroDigital codecs: In fact, the values in brackets
are the actual framerates from the full movie encode (which took place during
the night without any manual intervention). It turns out, ND speeds up during
the first pass over time and the full firstpass speed can only be reached if
the full movie is encoded (and as you can see this has quite an effect on the
average framerate). I can't tell you why this is (but it's a neat trick and
they reach more than an average of 100 fps during a full movie first pass) because
it's obviously a well kept secret.
Other important stuff
Here's the AviSynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DGIndex in Matrix and SPR. Futurama was interlaced so I used Decomb's FieldDeinterlace with no blending to get a progressive image.
I've used the default playback filter for every codec, using automatic postprocessing strength selection where this was possible. In XviD I turned on both deblocking and deringing, in HDX4 I turned on chroma and luma deblocking. I used a special Ateme DS filter for the NeroDigital playback (since the Nero filter only decodes Main Profile content - beta testers might also know an earlier version of this filter). I disabled the film effect where applicable. A note on this: HDX4's optimize for film mode actually filters out film grain during encoding, stores a parametric representation of the grain and applies it again during decoding. Thus, we have grain in HDX4 content. In future comparisons, I expect to see more of this as this method has been added to the AVC High Profile.
All files were reviewed using Media Player Classic as it can play back pretty much everything you can throw at it. Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player!
A note about screenshots: Unlike what you might think hearing about the go-to function in MPC, it's not easy to take screenshots. The following codecs behaved properly (insofar they they either exhibited no delay and using frame X from the AviSynth script meant I could go to frame X in the encoded video and get to see the same frame, or they'd have a constant delay with respect to the AviSynth source): DivX, VP6, XviD. Both NeroDigital codecs exhibited an constant delay as well, but still have some problems in the filter (stepping back frame by frame is not possible). ateme has promised to work on that. Now for the hall of shame: 3ivX and RealVideo were usually ±1 frame off, but since that kept changing, I had to make a screenshot, compare it with the reference frame (switch very fast between the two frames in the picture viewer), and go back to make another one if it didn't match. It got even worse for HDX4, VSS and WMV9. First HDX4: according to the creators, the problem is created by a feature they've introduced to handle corrupt MPEG-4 streams and it should be possible to turn this off in a future version (there'll be a next codec comparison where they can prove that ;). And all three codecs had problem to jump to the proper frame. If you're playing a movie encoded with them, and you use the position slider to move to another place in the movie, you either have black screen until the next I-frame (HDX4), or the video fast forwards (WMV9, VSS) to the next I-frame. When it comes to taking screenshots, that means I have to find the I-frame previous to the frame I want to make my screenshot, then advance frame by frame until I find the proper frame (and if there's no visually very distinctive mark in a screen that's not visible or at a different place in frame X-1 or frame X+1 that's really hard to do). So, let it be said that 3ivX, RV10, HDX4, WMV9 and VSS require work in the filter department and have cost me hours upon hours that are basically wasted because the filters don't implement all the seeking functionality that DirectShow forsees (you'd expect screenshots to take you perhaps 40 minutes.. but the way things are all the screenshots took me 4h15m).
And as an addendum: I had to redo a lot of screenshots due to a color conversion issue. Turns out this time the VMR9 renderer, which I used last time to make screenshots, would lead to discoloration. In order to make screenshots from NeroDigital, I had to install the DirectX SDK to get a new default renderer that exposes a snapshot button in the filter properties. But, while this new default renderer would no longer lead to discoloration, it effectively made making screenshots even more complex: The codecs that don't support frame accurate seeking would now require that I use the snapshot function of the renderer, rather than MPC's built-in one (using the MPC one would return me the previous keyframe). The same also applies to RealVideo.
Now proceed to the first test (low bandwidth JPEG version. This version loads 533KB of images for the beginning, and the total image size for the matrix test is 3.85 MB). If you have a lot of bandwidth and / or don't mind waiting, there's a high bandwidth PNG version (loads 1.98 MB of images initially and if you want to see all the images, you have to download 16.6 MB). Also note that the high bandwidth version requires that you have enough browser cache left or the images will be reloaded each time you zoom in or out.
This document was last updated on December 26, 2005