{"id":54312,"date":"2019-12-23T00:00:00","date_gmt":"2019-12-22T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/kak-rabotaet-videokodek-chast-1-osnovy"},"modified":"2020-02-18T14:02:18","modified_gmt":"2020-02-18T11:02:18","slug":"kak-rabotaet-videokodek-chast-1-osnovy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/kak-rabotaet-videokodek-chast-1-osnovy","title":{"rendered":"How a video codec works. Part 1. Fundamentals","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Part Two: Principles of Video Codec Operation<\/a><\/noindex><\/h3>\n

Any raster image<\/b> can be represented as a two-dimensional matrix<\/b>. When it comes to colors, the idea can be expanded by considering the image as a three-dimensional matrix<\/b>, where additional dimensions are used to store data for each of the colors.<\/p>\n

If we consider the final color as a combination of so-called primary colors (red, green, and blue), in our three-dimensional matrix we define three planes: the first for red, the second for green, and the last for blue.
\n\"How<\/a><\/noindex>
\nWe will call each point in this matrix a pixel (element of the image). Each pixel contains information about the intensity (usually in the form of a numerical value) of each color. For example, a red pixel<\/b> means that it contains 0 green, 0 blue, and maximum red. A pink pixel<\/b> can be formed using a combination of three colors. Using a numerical range from 0 to 255, a pink pixel is defined as Red = 255<\/b>, Green = 192<\/b> and Blue = 203<\/b>.<\/a><\/noindex><\/p>\n

\"How<\/a><\/noindex>

\nThe article is published with the support of EDISON.<\/p>\n

We develop applications for video surveillance, streaming video<\/a><\/noindex>, and we also specialize in video recording in the operating room<\/a><\/noindex>.<\/p><\/blockquote>\n

Alternative Methods of Encoding Color Images<\/h3>\n

\nThere are several other models for representing the colors that make up the image. For example, one could use an indexed palette, which requires only one byte to represent each pixel, instead of the three needed when using the RGB model. In such a model, a 2D matrix can be used instead of a 3D matrix to represent each color. This saves memory but offers a smaller color range.<\/p>\n

\"How
\n<\/p>\n

RGB<\/h2>\n

For example, take a look at the image below. The first face is fully colored. The others are the red, green, and blue planes (the intensity of the respective colors is shown in grayscale).<\/p>\n

\"How
\n
\nWe observe that the shades of red in the original will be in the same places where the brightest parts of the second face are observed. In contrast, the contribution of blue can primarily be seen only in Mario's eyes (the last face) and elements of his clothing. Note where all three color planes contribute the least (the darkest parts of the images) \u2014 this is in Mario's mustache.<\/p>\n

A certain number of bits is required to store the intensity of each color \u2014 this value is called bit depth<\/b>. Let\u2019s assume 8 bits (ranging from 0 to 255) are spent on one color plane. Then we have a color depth of 24 bits (8 bits * 3 planes R\/G\/B).<\/p>\n

Another property of an image is resolution<\/b>, which represents the number of pixels in one dimension. It is often denoted as width \u00d7 height<\/b>, as demonstrated below in the image example of 4 by 4.
\n\"How
\n
\nAnother property we deal with when working with images\/videos is aspect ratio<\/b>, describing the usual proportional relationship between the width and height of an image or pixel.<\/p>\n

When a movie or picture is said to have a size of 16 by 9, it usually refers to the display aspect ratio<\/b> (DAR<\/b> \u2014 from Display Aspect Ratio<\/i>). However, there may sometimes be different forms of individual pixels \u2014 in this case, we refer to the pixel aspect ratio<\/b> (PAR<\/b> \u2014 from Pixel Aspect Ratio<\/i>).<\/p>\n

\"How
\n
\n\"How<\/p>\n

A note for the housekeeper: DVD<\/b> corresponds to DAR 4 by 3<\/b><\/p>\n

Although the actual resolution of a DVD is 704\u00d7480, it maintains an aspect ratio of 4:3 because the Pixel Aspect Ratio (PAR) is 10:11 (704\u00d710 \/ 480\u00d711).<\/p><\/blockquote>\n

\nFinally, we can define video<\/b> as a sequence of n<\/b> frames over a period time<\/b>, which can be considered an additional dimension. And n<\/b> then \u2014 this is the frame rate or number of frames per second (FPS<\/b> \u2014 from Frames per Second<\/i>).<\/p>\n

\"How
\n
\nThe number of bits per second required to display video is its bitrate<\/b> \u2014 bitrate = width * height * bit depth * frames per second<\/b>.<\/p>\n

For example, for a video with 30 frames per second, 24 bits per pixel, and a resolution of 480\u00d7240, it will require 82,944,000 bits per second or 82,944 Mbps (30x480x240x24) \u2014 but this is without using any compression methods.<\/p><\/blockquote>\n

\nFor example, for video at 30 frames per second, 24 bits per pixel, with a resolution of 480\u00d7240, it would require 82,944,000 bits per second or 82,944 Mbps (30x480x240x24) \u2014 but this is without using any compression methods.<\/p>\n

is almost constant , it is called<\/i>constant bitrate CBR<\/b> (constant bit rate<\/b> \u2014 from ). But it can also vary; in this case, it is called<\/i>). But it can also vary; in this case, it's referred to as variable bit rate<\/b> (VBR<\/b> \u2014 from variable bit rate<\/i>).<\/p>\n

This graph shows limited VBR, where not too many bits are spent in the case of a completely dark frame.<\/p>\n

\"How
\n
\nInitially, engineers developed a method to double the perceived frame rate of a display without using additional bandwidth. This method is known as interlaced video<\/b>; primarily, it sends half of the screen in the first \"frame\" and the other half in the next \"frame.\"<\/p>\n

Currently, scene visualization mainly occurs using progressive scanning technology<\/b>. This is a method of displaying, storing, or transmitting moving images, where all lines of each frame are drawn sequentially.<\/p>\n

\"How
\n
\nWell, now we know how an image is represented digitally, how its colors are arranged, how many bits per second we spend to show video, whether the bit rate is constant (CBR) or variable (VBR). We understand the specified resolution using a given frame rate and are acquainted with many other terms, such as interlaced video, PAR, and some others.<\/p>\n

Removing redundancy<\/h2>\n

\nIt is known that uncompressed video cannot be used effectively. An hour of video at 720p resolution and 30 frames per second would occupy 278 GB. We arrive at this value by multiplying 1280 x 720 x 24 x 30 x 3600 (width, height, bits per pixel, FPS, and time in seconds).<\/p>\n

Using lossless compression algorithms<\/b>, like DEFLATE (used in PKZIP, Gzip, and PNG), will not provide sufficient reduction in required bandwidth. Therefore, we must look for other video compression methods.<\/p>\n

To this end, we can leverage the characteristics of our vision. We perceive brightness better than color. Video is a series of sequential images repeated over time. The differences between adjacent frames in the same scene are minor. Furthermore, each frame contains many areas using the same (or similar) color.<\/p>\n

Color, brightness, and our eyes<\/h2>\n

\nOur eyes are more sensitive to brightness than to color. You can verify this yourself by looking at this image.<\/p>\n

\"How<\/p>\n

If you cannot see what colors the squares are on the left half of the image. A<\/b> and B<\/b> If they are actually the same, that's normal. Our brain causes us to pay more attention to light and shadow rather than color. On the right side between the marked squares, there is a bridge of the same color \u2014 hence we (i.e., our brain) can easily determine that, in fact, there is the same color there.<\/p>\n

Let's break down (in simplified terms) how our eyes work. The eye is a complex organ made up of many parts. However, we are most interested in the cones and rods. The eye contains about 120 million rods and 6 million cones.<\/p>\n

Let's consider color and brightness perception as separate functions of certain parts of the eye (in reality, it's a bit more complex, but we'll simplify). Rod cells primarily respond to brightness, while cone cells respond to color. Cones are divided into three types, depending on the pigment they contain: S-cones (blue), M-cones (green), and L-cones (red).<\/p>\n

Since we have significantly more rods (brightness) than cones (color), it can be concluded that we are better at distinguishing transitions between darkness and light than colors.<\/p>\n

\"How
\n<\/p>\n

Contrast Sensitivity Functions<\/h2>\n

\nResearchers in experimental psychology and many other fields have developed numerous theories of human vision. One of them is called contrast sensitivity functions<\/b>. They are related to spatial and temporal illumination. In short, it concerns how many changes are needed before an observer notices them. Note the plural form of the word 'function.' This is because we can measure contrast sensitivity functions not only for black-and-white images but also for color images. Results from these experiments show that in most cases, our eyes are more sensitive to brightness than to color.<\/p><\/blockquote>\n

Since it is known that we are more sensitive to the brightness of an image, we can attempt to use this fact.<\/p>\n

Color Model<\/h2>\n

\nWe have somewhat understood how to work with color images using RGB scheme. There are other models as well. There is a model that separates brightness from chroma, known as YCbCr<\/b>. By the way, there are other models that perform a similar separation, but we will only consider this one.<\/p>\n

In this color model Y<\/b> \u2014 it represents brightness, and two color channels are used: Cb<\/b> (saturated blue) and Cr<\/b> (saturated red). YCbCr can be derived from RGB, and the reverse transformation is also possible. Using this model, we can create full-color images, as shown below:<\/p>\n

\"How
\n<\/p>\n

Conversion between YCbCr and RGB<\/h2>\n

\nSomeone might argue: how is it possible to obtain all colors without using green?<\/p>\n

To answer this question, let\u2019s convert RGB to YCbCr. We will use the coefficients accepted in the standard BT.601<\/b>, which was recommended by the ITU-R<\/b>. This division defines standards for digital video. For example: what is 4K? What should be the frame rate, resolution, color model?<\/p>\n

First, we will calculate the brightness. We will use the constants proposed by the ITU and substitute the RGB values.<\/p>\n

Y<\/b> = 0.299R<\/b> + 0.587G<\/b> + 0.114B<\/b> <\/p>\n

After obtaining the brightness, we will separate the blue and red colors:<\/p>\n

Cb<\/b> = 0.564(B<\/b> \u2014 Y<\/b>)<\/p>\n

Cr<\/b> = 0.713(R<\/b> \u2014 Y<\/b>) <\/p>\n

And we can also convert back and even obtain green using YCbCr:<\/p>\n

R<\/b> = Y<\/b> + 1.402Cr<\/b><\/p>\n

B<\/b> = Y<\/b> + 1.772Cb<\/b><\/p>\n

G<\/b> = Y<\/b> \u2014 0.344Cb<\/b> \u2014 0.714Cr<\/b><\/p>\n

Generally, displays (monitors, TVs, screens, etc.) use only the RGB model. But this model can be organized in different ways:<\/p>\n

\"How
\n<\/p>\n

Color Subsampling<\/h2>\n

\nWith an image represented as a combination of brightness and chrominance, we can take advantage of the human visual system's higher sensitivity to brightness than to chrominance when selectively removing information. Color subsampling is a method of encoding images, using lower resolution for chrominance than for brightness.<\/p>\n

\"How<\/p>\n

How much can we reduce chrominance resolution?! It turns out there are already some schemes that describe how to handle resolution and merging (Final color = Y + Cb + Cr).<\/p>\n

These schemes are known as subsampling systems<\/b> and are expressed as a 3-to-1 ratio\u2014 a<\/b>:x<\/b>:y<\/b>, which defines the number of samples of luminance and chrominance signals.<\/p>\n

a<\/b> \u2014 a reference horizontal sampling (usually equal to 4)
\nx<\/b> \u2014 the number of chrominance samples in the first row of pixels (horizontal resolution relative to a<\/b>)
\ny<\/b> \u2014 the number of chrominance sample variations between the first and second rows of pixels.<\/p>\n

An exception is 4<\/b>:1<\/b>:0<\/b>, which provides one chrominance sample in every block of luminance resolution 4 by 4.<\/p><\/blockquote>\n

Common schemes used in modern codecs:<\/p>\n