Audio video codecs: a video editor's tutorial introduction to codecs

What's a codec?

Raw uncompressed digital video contain a huge amount of information: 3 bytes per pixel translates into roughly 240Mbit/s for standard definition video, 504Mbit for standard HD video (720p), and 1136Mbit/s for full HD video (1080p).

Even at just standard definition (DVD) you'd need over 200GB to store a two hour uncompressed movie which isn't practical for most applications.

General purpose compression algorithms (e.g., zip, gzip, bzip2) don't work well on video/audio, but fortunately we've invented these marvelous domain-specific CODEC (short for COmpression and DECompression) algorithms which can achieve 100:1 compression ratio (and higher) without a visible loss of quality. I've even seen examples of videos that achieve an almost magical 1000:1 compression ratio while still maintaining very good quality.

One the things that are likely to confuse a newcomer (it sure confused me) is the huge variety and codecs available and their configuration options.

How do I choose a codec?

It depends on your purpose. Codecs are not created equal. They're often different because they are designed to be well suited for different purposes (e.g., real-time video conferencing vs streaming vs video editing). Each Codec has its own limitations.

In general, you always want to use the best tool for the job and codecs are no exception.

Codecs can generally be divided into two categories: lossless and lossy.

Lossless vs lossy codecs

1. Lossless (e.g., FFV1, HuffYuv): Like zip for general files or PNG for images, there is no loss of information (I.e., quality) so decompressed video is bit-for-bit identical to the original video.

   Lossless codecs are good for editing and archiving "master" copies.

2. Lossy codecs (e.g., H.264, MP4): can achieve much better bitrates but the decompressed video is not identical to the original video though it may look like that (or very close) to the naked eye.

For example, to simulate editing with a lossy codec I re-encoded the same video a few dozen times to see what the quality degradation would look like and after a few passes it is definitely visible to the naked eye. You don't want to do that.

In general codecs are designed to make trade-offs between:

1. CPU requirements (how many FPS you can encode/decode)
2. bitrate for a given resolution/quality
3. implementation complexity

So you can optimize a codec for streaming to squeeze out the best possible video quality for a given bitrate, but that means you need to throw much more computational resources at it to encode the video and maybe to decode it as well.

Best lossless codecs for editing and storage

Audio: WAV (PCM) or FLAC

Video editing: FF HuffYuv (supported by Avidemux). Poor compression (112mbps) but it's fast and it's not an intra-frame (no key frames) format which makes it easier to work with in a video editor (e.g., you can quickly access each frame individually without having to calculate it in reference to a key frame)

Video archiving: FFV1. An intra-frame lossless format. Much better compression than HuffYuv (20Mbit) but much slower to encode/decode and harder to work with.

Best lossy codec for streaming

Audio: AAC (open format, better quality for a given bitrate than MP3). For simple speech audio I personally can't hear the difference between high bitrates and low bitrates.

Video: H.264 (AKA MP4 AVC). Beats everything else hands down in terms of quality for a given bitrate but much more computationally intensive than algorithms such as MP4 ASP.

H.264 is highly configurable so your results may vary depending on the configuration settings and the type of video. I find it much easier to use Avidemux to configure H.264 encoding than FFmpeg.

It's an amazing codec. With clever configuration it seems to be possible to get some 720p HD videos down to under 550 kbps (over 1000:1 compression!) while still providing very good quality.