Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Time:2022-3-20

This article is from the official account number QbitAI, and if it is infringing, it can be deleted.

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

When watching the animation (especially at Lipan / P station), I always feel that the picture quality is not good enough. Even if there are 1080p, I still feel that it is not clear enough and affects the experience too much.

Therefore, the world needs a set of magic to save the resolution.

Now, there is an open source algorithm called anime4k, which can turn the picture into 4K in real time during animation playback, with a delay as low as 3 milliseconds.

It can change 720p / 1080p into 2160p and 480p into 1080p.

The team said that this is the strongest real-time super-resolution method of animation at present, which can be implemented in any programming language.

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Now, the project has picked more than 3700 stars in GitHub and once ranked on the trend list.

So, how does this algorithm benefit mankind?

Only animation

The team said in the paper:

Traditional super-resolutionAlgorithms (such as bicubic), the results are not very good, because they are not born for animation at all< br/>
The traditional deblurring or sharpening method will overshoot when it is close to the edge of the object, distract the audience and reduce the perceptual quality of the image< br/>
Machine learning methods (such as waifu2x) are too slow to be real-time (< 30 milliseconds), especially when ultra-high definition is required.

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Anime4k, on the other hand, only deals with animation, regardless of other video types. This is very important.

Animation does not have as many textures as real video. It is basically objects and lines processed by flat shading.

As long as the picture quality gets a little better, the audience can see it。 Therefore, the team wisely thought that instead of improving the image quality of the whole picture, it was enough to focus on refining the edge, and details such as texture were not important.

How to do this should start with the principle of super-resolution:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

First, a picture can be divided into two parts:

everythingThe low-frequency component is a blurred low-resolution image< br/>
Second isHigh frequency residuals represent the difference between the two resolutions.

Enter a low definition map and turn it into a lower definition version to get a residual.

Thin and sharpen the residuals, and then add them to the low-definition map to get a high-definition map.

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

However, if the residual error is slightly wrong, it will cause ringing and overshoot and affect the effect. This is also the defect of our predecessors.
So the team found a new way:

First putResidual thickness minimizationAs a goal, this is no problem< br/>
However, it is not feasible to directly use the residuals obtained by arbitrary transformation on a low-definition map. The low resolution map should be changed accordingly in order to live in peace with the residual and obtain the ideal super-resolution results< br/>
Therefore, after inputting a graph and its residual, the “push” pixel of the residual makes the residual line thinner< br/>
At the same time, each push should perform the same operation on the color low-definition map.

In this way, the blur can be minimized without ringing and overshoot, which reduce the image quality.

Let’s have a competition

Anime4k (rightmost) competes against the non open source algorithm NGU (second from the left) from madvr and the open source machine learning algorithm waifu2x (third from the left).

First question, eyes:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Second question, ears:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Third question, jade hand:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Question 4, full face:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

The effect of waifu2x elder is obviously inferior to anime4k, and virtual shadow is common. The speed also has obvious defects, and each picture takes more than 1 second.

The image quality generated by NGU predecessors is similar to anime4k, but it is often defeated by anime4k.

More than that, each NGU takes ~ 6 milliseconds, and anime4k takes only ~ 3 milliseconds, which is twice as fast, which is more suitable for real-time generation.

If the effect is similar, why not use NGU directly? Because it’s not open source.

If you think 720p / 1080p animation doesn’t need to be so luxurious as 4K, you can save 480p to 1080p:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Still, anime4k is on a par with the non open source NGU.

Finally, the team did not abandon the power of machine learning even though they had acquired exquisite image quality improvement skills.

Because when saving still paintings (rather than animation), anime4k’s short board appeared. At this time, let the machine learning players waifu2x fight side by side with it and be more successful:

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds

If you need to improve the resolution in real time, you’d better use anime4k.

It’s open source anyway:

Project portal:
https://github.com/bloc97/Anime4K

Fierce! Look, XX artifact is coming! Turn the image quality into 4K HD in real time, with a delay of only 3 milliseconds