Codec shoot-out 2005 - Main round

Table of content:

1 : Introduction
2 : Test setup
3 : Usability
4 : Sources, bitrates and sizes
5 : Settings
6 : Encoding speed
7 : Test 1: Matrix 3 (normal bandwidth) -- ultra high bandwidth
8 : Conclusion
9 : Outlook

Welcome ladies and gentleman, to the main round of the 2005 codec comparison.

Just like last year, we have a new first timer: XviD AVC. Yes, you read right, the XviD team has whipped up an MPEG-4 AVC encoder which makes its testing debut right here. Because the codec is yet unreleased and still very young, I allowed a late submission, the build I tested only arrived 4 days after the deadline, a courtesy which will of course only be extended once.

I'll dispense with a lot of information that usually goes on the first page, because of the new mode introduced this year. You can read all about it in the article about this year's qualification round.

Just to quickly recapitulate, codecs that I had not previously tested, and codecs that did not perform too well in previous comparisons had to go through a qualification round. In the main round we find all qualificants, as well as codecs that directly qualified due to past results and / or my personal experience.

Let's have a look at the history of the codecs that are entered in this comparison:

• Ateme's High Profile AVC encoder is already available in ateme's own encoder suite. Some time in 2006, the encoder will also be available as part of Nero's NeroDigital toolchain.
• DivX 6.1 was released just days before the submission deadline for the comparison. It fully support SMP setups, as well as multiple quality improvements (many from the 6.0 series which was released a couple months after last year's comparison).
• Elecard's AVC encoder didn't make the cut last year because of various issues. This year it offers two pass encoding, supports the AVC High Profile and can write raw streams as well as MP4 files. The codec can be found in Mainconcept's H.264 encoder, as well as Elecard's MobileConverter software.
• HDX4 has been formally released in 2005. Since my last test, the noise handling has been improved and encoder bugs have been taken care off. The software also comes with a whole tool suite to facilitate encoding and playback.
• LMP4 is my shortname for the MPEG-4 ASP codec contained in libavcodec, the multipurpose open source codec suite.
• Nero's ASP codec NeroDigital has been significantly improved with the addition of RDO.
• VC-1 is the official name Microsoft gave their WMV9 Advanced Profile when submitting it for standardization to the SMPTE. I'm not sure of VC-1 has been fully ratified and there seems to be some controversy about the licensing, but VC-1 is a mandatory codec for both upcoming high definition DVD formats.
• VP7 is On2's latest incarnation of their VPx encoder series. Initially released for a small amount of money, the codec is now available free of charge for personal use. On2's codec is now also used in the video capabilities of the popular Skype software, but looking at the speed table, they won't use the same complex algorithms ;)
• x264 is an open source AVC High Profile encoder that has captured much attention in 2005, and which has found its unofficial forum home right here on Doom9.
• XviD 1.1 is finally about to be released. A few bugs have been fixed since beta2, and recently there has been some SMP optimization, but the release of those optimizations was too close to the deadline to make the test.
• XviD AVC is, as the name says, the AVC High Profile encoder from the XviD team. It supports all AVC profiles, and by the looks of it, even interlaced encoding. The codec will be made available to the general public in January 2006, under the GPL license as usual.

Similar to the comparison in previous years, codec makers were given two weeks to provide suggested settings and binaries for the task at hand. Unlike the qualification phase, all participants in the main round replied to my request to provide parameters that provide optimal quality while not making you wait forever.

If you are wondering about 3ivX and RV10, I got in touch with the respective people but we concluded that the progress made in terms of visible quality would simply result in a re-test of essentialy what we saw last year. So I guess we'll wait for 3ivX's AVC encoder and RV11..

The contestants

• Ateme High Profile Encoder, core 1.4.0.3. I'll refer to this codec as simply Ateme from this point on
• DivX 6.1, as available on divx.com
• Elecard AVC commandline encoder dated December 15th
• HDX4 1.5.4
• LMP4 - mencoder CVS build dated 11/30/2005
• NeroDigital ASP - core version 4.1.6.7
• VC1 - I found no way to extract the actual build number but the installer is dated December 16th
• VP 7.0.9.0, as available on on2.com
• x264 revision 387 as available from x264.nl
• XviD 1.1 CVS build dated December 19th
• XviD AVC preview build dated December 21st

All codecs were tested in a 2 pass setup using the settings suggested by the developers.

In accordance with the rules for the main round, the codecs will be placed into the following three groups:

• MPEG-4 ASP: DivX 6.1, HDX4, LMP4, NeroDigital ASP, XviD
• MPEG-4 AVC: Ateme, Elecard, x264, XviD AVC:
• Non MPEG-4: VC-1, VP7

Test setup:

All testing was done on the following hardware:

AMD Athlon 64 X2 4600+
Shuttle SN25P barebone system
2x 512 MB PC3200 Kingston HyperX DDR RAM CL2
Gigabyte Geforce 7800GT GFX card
Philips 230W5B2 Display connected via DVI

To encode I used the following software:

DGIndex 1.4.5 & DGDecode
AviSynth 2.55 for frameserving
VirtualDub 1.6.11 for VfW encoding
Nandub 1.0 RC2 lumafix to multiplex AVI
Mp4box CVS dated December 16th to multiplex MP4 (except for XviD AVC, it requires a fix that was recently made to mp4box but was not readily available yet, so I used a mp4creator build dated December 20th to mux XviD AVC)

Usability and additional software

This year marks the introduction of another review category: usability. This is because the test field is very inhomogeneous and we have various means of encoding. Besides the traditional VfW codecs, more and more codecs offer a commandline utility or a DirectShow filter for encoding, or have a whole suite that does a lot more than just encoding. In this test, QuickTime is the most special tool because it's basically a player that can export a file it can play to another format.

Here is a quick table outlining which encoding methods are available

( 1): not publicly available
( 2): via NeroDigital in 2006, filters restricted to the use of Nero's own tools
( 3): via Recode in 2006
( 4): Elecard MobileConverter
( 5): Mainconcept H.264 encoder
( 6): ffdshow
( 7): ffmpeg or mencoder
( 8): only when the filters are used via Recode
( 9): Recode
(10): Windows Media Encoder
(11): xvid_encraw

Unlike the qualification round, encoding was problem free in the main round. All non VfW codecs were made available as a commandline tool supporting AviSynth input and I didn't have to tinker around with a lot of external tools to get things done.

The only note I have is regarding the VC-1 encoder. It uses the registry to store its settings, like VfW codecs do, but there is currently no GUI to make this configuration. I was given a separate HTML application that allowed to write export the proper registry keys so that they could be imported into the registry, but this made things a bit more complex as the encoder itself was available via the commandline.

Playback

(1): If the stream is muxed into an AVI and the fourCC is set to a value a pre-installed MPEG-4 AVC VfW codec supports
(2): Ateme's own filters or the NeroDigital filters
(3): mplayer or VLC
(4): ffdshow

When it comes to playback, DivX, HDX4, LMP4, x264 and XviD have the most out-of-the box functionality, but by using tools for other codecs, VfW and DirectShow is never a problem. Basically you can install ffdshow and anything goes except for the non MPEG-4 codecs. This is also a considerable improvement over what we've seen in the qualification.

Sources, Bitrates and Sizes

Movies I encoded:

Matrix Revolutions - Region1, NTSC, length: 2h09, 185917 frames (@23.976 fps). I will call this movie Matrix3 from here on

Encoding parameters:

I used a 128kbit/s CBR MP3 audio track I created using BeSweet 1.5 beta 31 for all codecs using the AVI container. I used the same BeSweet and the latest Nero 6 AAC encoder DLLs to create a 128 kbit/s CBR HE AAC audio track for all codecs using the MP4 container. Microsoft uses the ASF container (WMV files are ASF files that contain just video and audio), so I encoded my audio to a 128 kbit/s WMA file for VC-1. As in previous years, the goal is to put Matrix3 onto a 700 MB CD. The audio file sizes were as follows:

• MP3: 121'621 KB
• AAC: 121'875 KB
• WMA: 121'871 KB

The proper filesize would be 121'156 KB, and all encoders don't manage to hit the target size on the stop, but a 250 KB difference between the streams is nothing to worry about too much.

Because the makers of each ASP participant asked that their decoder be used, I switched the whole ASP category back to the AVI container, where using a different decoder is not so much of a problem. The only problem was HDX4 and LMP4, as ffdshow has both codecs in the same group, so either ffdshow decodes both, or none. Consequently I had to reconfigure ffdshow each time before reviewing either of the two codecs. I've asked the makers of HDX4 to offer a selection of which codecs the filter decodes, so that you an ensure it only decodes its own content, but it would be nice of ffdshow offered more flexibility in that area as well.

With MP3 being muxed into an AVI and the AAC audio stream into the MP4 container, this resulted in a video bitrate of 620 kbit/s for codecs using the MP4 container, and 626 kbit/s for codecs using the AVI container. Microsoft still owes me a formula for the ASF overhead, but for this test I used 620 kbit/s for the video as I was told the overhead was not much different from AVI. Please refer to the FAQ if you have any questions or doubts why this scenario was chosen.

As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got:

Note that 700MB equals 716'800KB. I've marked every size in orange if it is more than 2.5 MB above target size (2.5 MB is about the oversize a CD can have and you can still burn it without having to activate overburning), red if it's more than 10 MB off target (grounds for disqualification). The ones marked in bright yellow are undersized more than 2.5 MB (that points to a rate control issue as well but is less problematic than oversized files).

All codecs except VC-1 managed to be within the unproblematic size range. We can see that VP7 and XviD AVC could use a little more work to more accurately hit the target size, but I have no complaints about the rest of the participants. Note that when it comes to XviD AVC, when using mp4box, the output size is much closer to target, but so far I have no mp4box build that produces acceptable output. The encoder's output makes use of certain properties in raw AVC streams that make the raw file bigger.. the mp4box build dated December 16th cannot properly handle that and produces choppy output. A bugfix has already been submitted so as soon as I get a fixed build, I can update the size table.

Now about VC-1: Unfortunately, the pre-beta build I was given contained two bugs that were only discovered during the course of this comparison: It would not encode every frame, constantly missing 6 (but this may just be a reporting problem, I did not not verify that every frame is indeed there), and more importantly, somewhere in the 75'000 to 100'000 frame range, the encoder started missing the target bitrate. When encoding 100'000 frames, the bitrate was already off by 100kbit/s, when encoding the full movie, the bitrate was only half of what I asked for. I immediately informed Microsoft and performed various experiments (which lead me to figure out the the range of the breaking point), but I did not receive a fixed build or workaround in the time allotted for such a scenario (2 business days). As it makes no sense to compare two video streams at completely different bitrates, I chose to stop the evaluation of VC-1 at this point.

Codec settings (applied with respect to the default codec settings)

• Ateme: Extra quality mode, 8x8 transform, 2 reference frames, psy level 3, 2 threads, deblocking strength -3.
• DivX 6.1: Unconstrained profile, MPEG quantization and for the second pass only: bitrate modulation 0.2, better quality mode and QPel.
• Elecard: High profile, CABAC, adaptive DCT, 3 B-frames, pyramid ordering of B-frames and use of B-frames as references, IDR period set to 1 GOP length (300 frames max), CBP size 1000000 (initially filled by 50%), motion estimation settings turned up all the way, 4 reference frames, ME search range 256, deblocking enabled with strength -1 (both alpha and beta), RDO turned on (2nd pass only), fast intra decision and hamad transform. So if you've previously used x264, you can configure the Elecard encoder just fine ;)
• HDX4: Unconstrained profile, MPEG quantization, QPel, 2 b-frames, highest quality mode and optimize for film.
• LMP4: I'll just give you the mencoder -lavcopts for mencoder:
  vcodec=mpeg4:vbitrate=560:mbd=2:mv0:trell:cbp:precmp=2:cmp=2:subcmp=2:predia=2:dia=2:preme=2:v4mv:last_pred=2:vqcomp=0.6:vmax_b_frames=2:qpel:vpass=1:turbo
  The second pass uses the same settings except no turbo and of course vpass set to 2.
• ND ASP: Simple profile, unrestricted motion vector, motion vector range 64, spatial prediction, h.263 quantization (method 2), 1 B-frame
• VP7: 2 pass good mode, encode speed 1, sharpness 0, maximal keyframe interval 9999
• x264: 1st pass: 2 b-frames, subme 2, no MB partitions. 2nd pass: 2 b-frames, subme 6, 3 reference frames, 8x8 DCT, weighted biprediction, trellis level 1, bitrate variance 4, deblocking filter strength -1/-1 and 2 threads.
• XviD: MPEG quantization, QPel, VHQ for B-frames
• XviD AVC: 3 reference frames, inter RD method 17. Since you can't have the encoder yet, here's a quick outline of the defaults: B-frames are enabled (spatial prediction), CABAC, 8x8 DCT

Encoding speed:

Since there is only one source in the main round, the choice of a source for this test was easy ;) As usual, I'm performing a two pass over the first 10'000 frames, note the time spent from the start of the first pass till the end of the second pass, and divide this by the number of frames encoded. The usual disclaimer also applies: even though 10'000 frame should be rather representative, I did only perform each measurement once and did not reset my PC in between. Looking at how major hardware sites on the web measure speed, I think my methodology still is a considerable improvement, but it in no way lives up to a scientific standard.

(1): Not used because it trades visible quality for a rather low gain in speed
(2): Non optimized codec, but because VirtualDub separates input reading (and thus decoding and resizing of the source) from encoding, it can still make good use of an SMP system. The speed gain seems to be dependent of the encoding speed, the CPU use of a faster codec (i.e. XviD) is considerably higher than of a slower codec (i.e. VP7)
(3): Not used because it's not fully ready yet

Note that since Ateme uses a proprietary mechanism to speed up the first pass, which gains in speed the longer the movie is, the numbers for the Ateme and ND ASP codec are taken from a full movie encoding session.

In addition for disqualification due to usability issues, I put another killer requirement in the speed area. With 1 MB cache away from having the fastest workstation CPU there is for video encoding these days, I decided to disqualify all codecs that do not reach at least 10.0 FPS. This is not only cutting down the total encoding time for the comparison, it is also to be considered that if you have a slower CPU, encoding time will be much lower. And with HD content gaining more importance, imagine encoding HD content on a 1 GHz box with a codec that doesn't even do 10 fps on my high end machine.

I've used the following colorscheme for the speed table: red = disqualified due to a single digit FPS rate. Orange: below real-time speed (real-time means the codec encodes at least as many frames as the video has per second), and green for codecs that encode faster than 2x real-time. If you read the qualification, this table is much more to my liking. In fact, It took me only 2 days to encode all the content for the main round, not taking into account re-encoding (more about that in the reviewer notes in the conclusion). And AVC High Profile encoders going at almost 50 FPS shows that AVC doesn't necessarily indicate slow encoding speed, and coming from last year's winner, I expect that the codec did not simply trade quality for speed - otherwise it is unlikely to make the final round, considering what the best two participants showed (Elecard and XviD) in the qualification round.

Last but not least, if you've seen the speed table from the qualification, please note that XviD did not use Turbo in the main round, which explains the considerable speed difference.

Other important stuff

Here's the AviSynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DGIndex in Matrix.

Matrix script:

mpeg2source("D:\DVDs\THE_MATRIX_REVOLUTIONS_D1\VIDEO_TS\matrix3.d2v")
crop(8,60,-8,-64)
LanczosResize(640,272)

With the exclusion of VC-1, the non MPEG-4 group only had one member (the qualification eliminated every other potential codec for this group), and since comparing VP7 against itself makes not much sense, I decided to merge the AVC and non MPEG-4 category, and thus have two groups of 5 codecs each. Only having two groups and reviewing them close together also allows to draw some direct conclusions between the two groups.

Playback

Because there is no fourCC in MP4, there is effectively only one playback filter for all MPEG-4 AVC codecs this year: the ateme decoder. It supports all MPEG-4 AVC high profile features including interlaced content (for which I have zero use and which should be banned altogether but that's another issue entirely).

In the two other categories, each codec could use its own filter, but this may change again in the future.

I enabled deblocking for MPEG-4 ASP content. There is no postprocessing for AVC content because AVC has a built-in deblocker (and those who have activated postprocessing for AVC content in ffdshow will be in for a really bad surprise.. it works but completely ruins your picture).

All files were reviewed using Media Player Classic as it can play back pretty much everything you can throw at it. Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player!

Now proceed to the test (low bandwidth JPEG version. This version loads 432 KB of images for the beginning, and the total image size for the matrix test is 3.66 MB). If you have a lot of bandwidth and / or don't mind waiting, there's a high bandwidth PNG version (loads 1.90 MB of images initially and if you want to see all the images, you have to download 16.7 MB). Also note that the high bandwidth version requires that you have enough browser cache left or the images will be reloaded each time you zoom in or out.

This document was last updated on December 26, 2005