Content source: Huawei Developer Conference 2021 HMS core 6 graphics technology forum, keynote speech CG kit exploring high performance rendering at mobile terminal
Speaker: Huawei Hisilicon Qilin GPU team Engineer
Hello, I’m a GPU R & D engineer from Huawei Hisilicon team. Today’s theme is “CG kit exploring high-performance rendering at the mobile terminal”.
Huawei graphics computing service(computer graphics kit, referred to as “CG kit”) provides a rendering framework in the cutting-edge field of computer graphicsPlug in SDK and GPU extension interface, assist developers to tap the ultimate performance of hardware, greatly reduce the difficulty of development, and help developers create products with better experience.
So,CG KitWhat solutions can be provided?
1、 Vulkan rendering framework practice
VulkanIt is the latest graphics standard API, which has certain advantages over OpenGL, and OpenGL has stopped evolving. In the future, graphics will basically evolve with Vulkan. For example, ray tracing at the mobile end is described in Vulkan and is not available in OpenGL. However, the problem is that Vulkan is very flexible. Compared with OpenGL, it has a certain use threshold. Therefore, a high-performance rendering framework based on Vulkan is needed to provide practical architecture and solutions and give full play to hardware performance.
2、 High performance rendering plug-in
CG kit provides a series of plug-ins with rendering capabilities such as multi-threaded rendering, photo super division, volume cloud and occlusion elimination, so as to facilitate the smooth progress of tripartite cooperation. For example, last year’s Netease Tianyu mobile game integrated the photo super division algorithm.
3、 Developer tool chain
CG kit also provides a complete developer tool chain. We can help developers quickly locate and solve rendering function and performance problems. You can imagine that if the future tools can automatically identify some function / performance problems in the game scene and feed back to the developers, it will greatly improve our development efficiency.
4、 Exploration of cutting edge graphics technology
For a long time, CG kit team has been exploring cutting-edge graphics technology. For example, how to apply AI technology to graphics rendering. At present, we have made research achievements in AI super division, automatic 3D face modeling, AI collaborative animation generation and so on.
Vulkan rendering framework practice
Vulkan rendering framework practice provides developers with Vulkan rendering solution optimization, rendering enhancement plug-ins and rendering technical documentsFor example, some API usage strategies, resource cache usage optimization, Vulkan compatibility issues, etc. will be introduced in the document.
As mentioned earlier, Vulkan is a relatively new API. Its command buffer mechanism can naturally support multi-threaded rendering. Because Vulkan is more flexible, CG kit will simplify the encapsulation of Vulkan’s API and make it easier to use.
Secondly, CG kit also supports physics based rendering. PBR can truly reflect the attributes of object surface illumination. The objects we render through PBR will be more realistic.
At the same time, CG kit provides a resource manager, which can reuse a large number of pipeline resources as much as possible, so as to reduce overhead and improve rendering performance.
Finally, CG kit also supports the display capability of hdr10. With the display optimization of HDR screen, the final drawing will have a better dynamic range.
Static super component
Let’s introduce the plug-in of game photo super score. The game has social attributes, and sharing is a very important link in social networking. In the course of the game, we often encounter some wonderful moments that are worth sharing, such as completing a milestone task, obtaining a limited edition beautiful skin, or completing the customization of a virtual character… At these moments, game players are very eager to share these wonderful moments with their friends, so they need to use the function of taking photos, But how to make the effect of taking pictures better?
CG kit provides two solutions for photographing effects.
The first solution is for high-end mobile phones with independent npus, such as those corresponding to Kirin 980, 990 and Kirin 9000 chips. These mobile phones with independent high-end npus can directly call the AI super division interface, so that the picture details can be significantly increased and the picture style is clearer and natural. As can be seen from the comparison diagram below, The facial details of the characters in the processed pictures have been significantly improved.
For those mobile phones without independent NPU chip, we also provide a general solution called filtersr, which is an algorithm based on general GPU. After applying this algorithm, mobile phones without NPU can greatly improve the picture effect, and its sense of sawtooth and picture definition are also greatly improved compared with traditional methods. It is worth mentioning that the oracle of heaven integrates our camera super score algorithm.
CG KitIt also provides a high-performance volume cloud plug-in for the mobile terminal. The volume cloud technology can dynamically render the global illumination scene in a highly realistic cloud sea in real time, and support players to shuttle freely in the cloud. The picture can truly restore the physical characteristics of the cloud. For example, the sense of particles coming from the player’s face during the shuttle process, as well as the real lighting effect, including the details of the edge of the cloud, will be presented very well. As can be seen from the figure below, the color of this cloud changes gradually with time, because we have adapted the light for 24 hours.
In terms of the shape of volume cloud, we support two styles: real and cartoon. We also support the conversion of mash with arbitrary shape into volume cloud, which can meet the different customization needs of developers.
From the perspective of performance, on the Kirin 980 platform, we can render a frame within 4 milliseconds. On the plug-in display, we also provide plug-ins on the PC and mobile terminals. Generally speaking, developers edit and debug on the mobile terminal and integrate it in the way of so on the mobile terminal, and its package size is only about 400k.
Let’s introduce the body fog component. In some games with specific game styles, the highly realistic effects of light and fog will bring you a very good experience. For example, there was a very popular game called cyberpunk 2077 last year, which has a lot of fog effects.
In the main engine 3A masterpiece, volume fog is generally realized through volume rendering. Unlike surface rendering, it only needs to render the surface of the object. Volume rendering needs to render the interior of the object, and the rendering workload may be dozens or hundreds of times that of surface rendering. Volume rendering requires very high performance of hardware, which is also the reason why it has not landed on the mobile terminal.
The CG kit volume fog component has made many optimizations for traditional volume rendering. I will briefly introduce two optimizations.
The first optimization is for multi light scene. When there are multiple light sources, the rendering workload increases tens of times. To solve this problem, we introduce the light source elimination technology to make the lighting calculation only occur in the local range affected by the light source, which can greatly reduce the rendering workload.
The second optimization is to reduce the sampling rate in order to improve the rendering efficiency, but there will be a problem of insufficient picture details after reducing the sampling rate. In order to solve this problem, we introduce the time domain filtering technology to fuse the results of the current frame and the previous frame, which will also get a very good result. The data structure is also relatively adapted for these optimizations.
After a series of previous optimizations, we can perform real-time high-performance volume fog rendering on the mobile terminal. From the perspective of performance, we can render a frame in 4 milliseconds on the Kirin 980 platform; In terms of integration form, PC and mobile plug-ins can be provided at the same time to support PC-side editing and debugging. The mobile end is integrated in so mode, and the package of mobile plug-ins is less than 500kb.
Finally, let’s introduce AI pinch face. If we can achieve a virtual image of thousands of people and thousands of faces in the game scene, and its virtual image can have some facial features of game players, it will enhance the sense of substitution of users. AI kneading face is input in the form of front self photographing. After calling our AI interface, a set of kneading face parameters that highly restore the user’s facial features will be generated, and the 3D model will be automatically generated based on the kneading face parameters. At the same time, while pursuing geometric similarity, we also integrate the real texture and model texture, so that its texture can present some facial features of the player and has higher reducibility. We provide a very simple interface for integration. Developers can easily adapt and greatly improve their development effect.
Because we are pure end-to-side operation, in the medium and high-end models, the AI kneading time is less than 4S, and the stability is very high. At the same time, our face pinching range not only supports the face, but also supports facial features, hair style, glasses, etc; We also integrate the highly restored real texture, have better restoration degree, and provide the beauty adjustment function, which can control the beauty degree of the final output image. In terms of development efficiency, developers only need two steps to call our functions. The first step is to freely adapt the model at the PC end, and the second step is to call our interface with the user’s self photographing as the input at the mobile end. The whole process is very simple and convenient.
I hope you can have in-depth cooperation, thank you!
Learn more > >
Follow us and learn the latest technical information of HMS core for the first time~