Shape shifting and shadow changing: the story behind the color effect of short video


Author: Chang Qing graduated from Tencent in 2008 and has been engaged in client research and development. He has successively participated in PC QQ, mobile QQ, QQ IOT and other product projects. At present, Tencent’s video cloud team is responsible for the optimization and implementation of audio and video terminal solutions, helping customers to obtain the first-class audio and video solutions in the industry with controllable R & D costs. At present, we Our product lines include: interactive live broadcast, on-demand, short video, real-time video call, image processing, AI, etc.

From the evolution of eyes

About 540 million years ago in the Cambrian period, some organisms on the earth began to evolve photosensitive cells, which can sense the intensity of light, and directly drive the body’s moving cells to take necessary avoidance actions to ensure that vulnerable cells are not injured by ultraviolet light.

Later, individual photoreceptor cells began to cluster and form a depression. With the gradual deepening of the depression, these photoreceptor cells gradually gathered into a spherical structure with pores in the front. Thus, the prototype of “eyeball” was formed.

Shape shifting and shadow changing: the story behind the color effect of short video

With the formation of spherical eye structure, the principle of “pinhole imaging” in middle school physics textbooks will play a more and more role: as long as the opening is small enough, light can accurately reach these gathered photoreceptor cells for imaging, and the objects in front of them will become clear. Nautilus, one of the mascots of Tencent culture, has adopted this evolutionary strategy of “taking small as beautiful”, and has been sticking to it until now.

But this smaller and smaller strategy will also result in less and less light, so the world of Nautilus must be very dark. Our ancestors didn’t want to give up a bright world, so we installed a high-level optical component lens for our eyes in order to have a bright and clear world.

However, when everyone can see the world clearly, the ability of color discrimination has evolved into a new round of arms race.

When our ancestors walked on the same kind of corpses that insisted that “I want to be healthy and healthy as a vegetarian”, they didn’t know that the reason why they lived longer was because of the eyes that could distinguish whether the leaves were young leaves or mature leaves. Because mature leaves often contain a lot of phytotoxins, which will cause great harm to the body of the eater. So we are all descendants of that wave of “lecherous” monkeys.

And the eye’s ability to distinguish colors, not only in ancient times let us survive, but also today we can see more colorful effects. For example, the basic principle of shape shifting effect in short video is to make full use of color.

Modification and combination of three primary colors

There are three kinds of visual cone cells in the human body, so the colors we see are all composed of three primary colors. It’s not that there are only these colors in the world, but the more gorgeous color space we can’t perceive.

Therefore, the imaging principle of LCD is based on the combination of R (red) g (green) B (blue) primary colors. Tencent cloud short video (ugsv) has made some articles on these three color spaces

Let’s take a static picture as an example:

Shape shifting and shadow changing: the story behind the color effect of short video

Now remove the red component from the picture, enlarge it by 10%, and move it a little further

Shape shifting and shadow changing: the story behind the color effect of short video

Remove the blue component, enlarge by 10%, and move some distance

Shape shifting and shadow changing: the story behind the color effect of short video

Remove the green component, zoom in 10%, and move some distance

Shape shifting and shadow changing: the story behind the color effect of short video

Then, the three images are superimposed with 33% transparency and the source image to form a shape shifting illusion effect

Shape shifting and shadow changing: the story behind the color effect of short video

Give it to the computer

The above pictures are simply processed by image processing software, but if they are video files, it is obvious that they should be handed over to the computer for automatic solution, how to do it?

First of all, we define coordinates for this picture. In order to facilitate processing, we define the center point of the picture as (0,0), and the maximum value of the XY axis of the picture is 1, as shown in the following figure:

Shape shifting and shadow changing: the story behind the color effect of short videoCoordinate definition

Then we start to process the two transformations mentioned above, one is amplification, the other is movement.

  • enlargeIf any pixel coordinate on the definition image is(x,y)The coordinates after the magnification of s times are(x',y') = s(x,y) = (sx, sy)
  • moveIf any pixel coordinate on the definition image is(x,y), move it(?x,?y), the coordinates after moving are(x',y') = (x+?x, y+?y)
  • mergeCombine the above two formulas, and the new coordinate point is(x',y') = (s(x+?x), s(y+?y))
  • superpositionThen we will overlay the modified image on the original image. The formula of alpha overlay issrc * (1 - alpha) + overlay * alpha

OpenGL based version

For the special effects processing of short video, we need to process dozens or even more video images per second, so the simple C language processing algorithm can not meet the performance requirements. We need to use the hardware acceleration ability of mobile phones. At present, in addition to the common OpenGL and other APIs, there are also options such as Vulkan, DirectX, metal and so on. At present, OpenGL has the best generality on all platforms, and there are abundant information on the Internet. Both desktop platform and mobile platform support it. This paper will talk about the implementation of special effects with OpenGL.

RealizationShape shiftingIn special effects, we will use the fragment shader of OpenGL. The function of fragment shader is to return the color value of each point of the image. Fragment shader also has its own programming language, which is called Shading Language for short glsl. Glsl and C are very similar, adding some vector data types and some image related processing functions. One of the special features of shading compared with C is that it does not support implicit type conversion, such as floating-point integer conversion (when using floating-point numbers, you must add decimal points, such as 1.0).

I don’t list all the special effect codes here for the reason of space. I only attach the partial implementation of amplification and displacement:

varying vec2 textureCoordinate;
uniform sampler2D inputImageTexture;

void main() {
    vec2 offset = vec2(0.05, 0.05);
    float scale = 1.1;
    vec2 coordinate;
    coordinate.x = (textureCoordinate.x - offset.x) / scale;
    coordinate.y = (textureCoordinate.y - offset.y) / scale;
    gl_FragColor = texture2D(inputImageTexture, coordinate);

Considering that not all readers have learned Shading Language, here is a simple interpretation:

  • We define a variable herecoordinateThis is what we mentioned earlier(x,y), whose value is((x - ?x)/s, (y-?y)/s)
  • functiontexture2DIs to get the color of a coordinate from the picture. Here frominputImageTextureMediumcoodinateThe coordinate point gets the color.
  • gl_FragColor, which is the output value of fragment shader,gl_FragColorIs a glsl predefined global variable of typevec4Both (R, G, B, a) these four quantities, that is, an RGBA color.
  • After the program runstextureCoordinateThis coordinate will be displayedgl_FragColorThis color.

As mentioned above, we can make more interesting effects for video. For example, we can overlay the current frame and the previous frame, so that we can make some more interesting effects.

Shape shifting and shadow changing: the story behind the color effect of short video

More special effects

This paper introduces one of many video special effects of Tencent cloud short video (ugsv). If we want to achieve more complex special effects, we need to continue to study the knowledge of OpenGL, face recognition and other related fields, which requires a period of study and effort, and inevitably need to step on many holes.

Does it take only a few minutes to build a quick short video application with various magic effects?

Long press to identify the QR code in the figure below, and go to our Tencent cloud short video official website to learn about our short video products.

Shape shifting and shadow changing: the story behind the color effect of short video

You can install a full version of demo that we prepared for you to show our product features. It is worth mentioning that the source code of this demo is completely free and open source, which can be downloaded to the official website of Tencent cloud video service short video (ugsv) document area.

Tencent cloud short video SDK does not charge extra fees, but if you want to use it, you need to buy Tencent cloud on demand traffic resource package flagship 2 (1W / year), flagship 3, and get one-year license for free. At the same time, you can also get one month trial period without paying any fees. Welcome your guidance. More > > short video official website

Questions and answers

How to identify yellow in short video?

Related reading

The engine that drives the short video boom

From QQ video call to enterprise level fusion audio and video

How to realize the PK scheme of seamless switching

This article has been released by Tencent cloud + community authorized by the author. The original link is:

Welcome to Tencent cloud + community or pay attention to WeChat community (QcloudCommunity).