So here we are, asking the much dreaded question, which two codecs of each group will make it to the final round.
DivX 6.1 left a good impression. If their marketing department had their way, we could probably squeeze a movie in DVD quality on a 128MB memory stick, but this clearly isn't true. The technical department was more modest, claiming comparable quality to XviD, the reigning champion in the ASP area. In the end I have to agree with them, although not completely. I still found a small advantage of XviD, but the differences are small. DivX also closely follows XviD in terms of speed, thus making it a very viable choice.
I was not too pleased with HDX4. It's fast all right, but not the fastest anymore. In terms of quality, if you decode it with a decoder that doesn't restore the stored film grain, you see that the codec loses quite a bit of details. If decoded with its own decoder, it looks better, but I cannot help but notice that the grain restored in no way reflects the source.. there's grain almost everywhere and at places where there is none visible in the source.
LMP4 delivered a good fight with DivX and XviD. I found myself giving it similar rating with the other two aforementioned codecs, sometimes surpassing one of them, sometings being rated a bit lower. In the end, it was not enough to make the finals, even though getting close. And if we have a quick look at the speed table, the choice is clear, almost comparable quality but considerably slower, that's not enough for the final round.
NeroDigital ASP won the speed round again, but as in the qualification round, it shows that it trades quality for speed. Consequently, it did not make the final round.
XviD once again lives up to the expectations. In my review scenes, it either beat DivX by a small margin, or was rated the same. Combined with excellent speed despite not having any SMP optimization in time for the test, it's a worthy contender in the final round.
So here's the ASP group again, listed in order of the results of the quality review
Now let us proceed to the AVC group plus VP7.
The Ateme High Profile Encoder, last years winner in terms of quality, consistently delivers very good results, and while not always rated the best in every review scene, it was enough to proceed to the final round.
The Elecard encoder also delivered excellent results, but it was not enough to make the finals. While the deblocking settings seem comparable, I think the deblocker is still stronger, and while that can get rid of blocks, it can also get rid of details (although that is very hard to spot).
Even though VP7 delivers very good results, I always rated it in the lower half of my contestants. Looking at all non MPEG-4 codecs in the test, there's no doubt this is the best non MPEG-4 performer (especially since VC-1 didn't make it), but because it was put in what is definitely the hardest comparison ever, it just couldn't get any further than this.
x264 does live up to the hype. It fought a long and hard battle with the Ateme and Elecard encoder and in the end beat the other AVC encoders (with the exception of the Ateme one) by a hair.
XviD AVC, while being a good AVC codec nonetheless, doesn't quite deliver the same results as the other three AVC codecs in this group. I found it had a slightly stronger tendency towards blocking, and while the difference is generally small, you can still spot it. Consequently, XviD AVC does not make the final round.
So here's the AVC + non MPEG-4 group again, listed in the order of the results of the quality review
As you can see, I put two codecs on the first and third line. This is because with my review scenes, I cannot reliably conclude which one of the codecs on the same line looks better, and that despite spending several hours on the review part. I hope that the inclusion of two other movies will make that distinction possible in the final round. And, even though 3 codecs had to go, if placed in the other group, they might just have made the finals instead of the ones qualified now.
If we look at the 4 finalists, I would rate them as follows when it comes to Matrix 3:
Where the difference between the two ASP codecs is smaller than the difference between the AVC codecs and XviD.
Best quality per fps
Even with a dual core machine, you can see by the FPS numbers on page one that while there are fast codecs, there are those that could use a good kick in the behind to get moving.
With its 100% lead over the winner in the ASP group, the award still goes to NeroDigital ASP. Both ASP finalists also deliver a good performance in this category, and the Ateme encoder does best when it comes to AVC encoders.
This is a 2005 novelty as well. I experience many things during a comparison which may or may not be interesting, but I feel are still somewhat pertinent for such a comparison, hence I'll put them here.
First of all, what's the worst thing that can happen in a codec comparison? You discover a problem in your slowest participant on a weekend. So happened with VC-1 and XviD. It got worse with VC-1 as it was not possible to provide a workaround or fix for the problems found on short notice. And even if it could, there's the time difference and the fact that upon receiving a fix, it would take another 12 hours for the first movie to be encoded. While you do not want any codec to require a re-encoding session (each participant has the right to one such session, and has two business days to reply to provide a workaround or fix), if your slowest participant requires a second attempt, your whole schedule goes belly up big time.
As in case of VC-1, the problems with XviD were also bitrate related. The first file came out significantly too small, but I received a fix in time. XviD AVC suffered from the same issue and I also had to re-encode Matrix 3 with that codec. Since it was a late submission, this further delayed the comparison (as I previously noted, it only took 48 hours until every codec had encoded Matrix 3 for the first time, courtesy of having both this year's dual core machine, and last year's single core machine available).
Another point I'd like to bring up is the fourCC configuration in ffdshow. It would be advantageous if those could be modified as this would've made the ASP group easier.
The AVC group this year lived up to my worst fears as a reviewer. If you look at a scene once and don't spot any differences, you can go back as many times you like, focus on specific details in order to find differences (they are always there, but if you have something on screen that magically attracts your eyes, it's hard to look someplace else), but in the end you start asking yourself who else is going to spot those differences. In past years, for each movie it generally took only a look at the first few scenes (even in movies where I look at whole chapters, I generally divide them into smaller bits and look at them separately), I normally get an idea of how codecs do compared to each other. If you need to start a point system and count points at the end, it is much more likely that other people who reproduce the encoded movies and have a look themselves might not come to the same conclusion.
Ateme, in the last possible moment, managed to complete an MP4 parser filter which allows perfectly frame accurate navigation in MPC. I could open any MP4, jump to frame X, and it would match frame X decoded by DGDecode (I checked directly via DGDecode to rule out any issues that might occur when opening an AviSynth script in MPC). However, the same cannot be said for our codecs in the AVI container. With DivX, XviD and VP7 I had a negative frame delay of 1, (so frame N in the source became frame N-1 in the encoded video), in HDX4 the frame delay was 1 frame in the positive sense, and LMP4 was frame accurate. When I made ffdshow decode XviD and DivX, I suddenly was able to parse the frame accurate as well, which seems to hint that the decoder has an influence on such navigation, not only the parser.
Last but not least, this year another nasty issue crept up. What do you do with a participant where the main programmer previously asked you to use the same filter for all ASP codecs, then finds a bunch of reason why this is unfair to his codec when you just end up doing what was asked before? And what about discarding my results without investigation even if I send in screenshots and sample video clips? I will always provide backup for my findings to each participant interested, but I expect that such content is properly analyzed before any statements, especially to a public mailing list, are being made. Just as I will not make negative comments about any participant unless I can properly back those up (with content from the review in question), I expect the same from each participant, or they will face disqualification. Last but not least, I find it highly inappropriate if a participant alleges irregularities unless those claims can be backed up with sources that match the contents of the comparison. Now in case you're wondering, this is about LMP4, I've never had any similar problems with any other contestant in the 5 years I'm doing codec comparisons.
The results of the final round are in so check them out as well.
Since HD DVD and Blu-ray will finally be released in 2006, VC-1/AVC compatible hardware will finally be available. I suspect that while prices will be higher than comparable DVD players initially, this, along with HDTV broadcasts using AVC in Europe, will give the more powerful codecs a boost. Consequently, we'll see a move towards AVC in areas where ASP is currently being used. With the latest generation of GFX cards capable of accelerating the decoding of such content (or in case of ATI even accelerate encoding - assuming they manage to deliver), and with Windows Vista requiring more powerful GFX cards (meaning newer cards, and thus more likely to have this video acceleration feature), I think AVC stands to become mainstream sooner than later.
Considering how the three groups played out (the ASP group having a major advantage because competition in that group was considerably less fierce than in the other group, and if the AVC encoders were distributed over both groups, the finals might look differently), and considering that more ASP encoder makers move to AVC, the next comparison is likely to see some changes in the groups (perhaps a random distribution of all participants into a number of groups), and a submission limit (this year we have multiple entries from two teams.. since every entry means additional work I'd like to cut this down to one entry per team).
Now if you have any questions, please have a look at the FAQ.
This document was last updated on December 25, 2005