Perfdog helps to explore automation performance testing

Time:2020-11-29

Background:
The game project adopts agile development, version development iteration is very fast, basic 1-2 weekly version

Necessity of performance test

Number of performance issues in phases throughout the project
Perfdog helps to explore automation performance testing
Performance problems do not occur at the beginning or suddenly appear one day, but accumulate with our development progress;
Later, we hope to solve problems for months or even years in a few days, but in fact, the results are not satisfactory. And the same problem, the same person, at different times to deal with the experience and time is completely different.
Therefore, the performance problem seems to be a technical problem of the R & D team, but in essence it is the development process of the R & D team

If we can standardize the process so that each version has a data display, and once problems are found and handled in a timely manner, the optimization time in the future can be greatly reduced. However, the performance of each version of human resources is relatively weak, so it can be handled in an automatic way. Will the automatic operation affect the performance data we get, Now let’s explore;

Impact of automation on application performance data

Comparison of the first group of tests

Test background:
1. Open perfdog to record the performance data of manual running function and automatic running function
2. The automation function used this time is airtest

Test case:
1. The airtest ide connection is not opened, and the function of manual running is enabled
2. Open airtest ide connection and run manually
3. Open airtest ide connection and use the automatic script running function
4. Disconnect airtest ide
5. Close airtest ide process

Automation script:
Will only run a small combat function, very short time
Perfdog helps to explore automation performance testing
The disconnection of the following test cases refers to:
Perfdog helps to explore automation performance testing

Let’s take a look at FPS first

Perfdog helps to explore automation performance testing
Obviously, we find that whether or not to run the game functions automatically has little effect on FPS

Let’s look at the memory

Perfdog helps to explore automation performance testing
It is found that automation has no effect on memory, and it is almost the same for memory if automation cannot be opened

Let’s look at the CPU
Perfdog helps to explore automation performance testing
We found that the total CPU utilization rate increased significantly when the IDE connection of airtest was opened, and the utilization rate of total CPU was also increasing when running automation scripts. However, the CPU utilization rate of APP has almost no impact.
This is because when the airtest IDE is connected, the IDE will use minicap service to obtain the screen shot of the phone, which will affect the overall utilization rate of CPU. When running the script, airtest needs to search and match images, so it also takes up CPU. However, the usage rate of app will not be affected.

Comparison of the second group of tests

This test does not apply to automated scripts, the impact of IDE is compared separately

Test case:
1. The static page is not connected to the airtest ide
2. Connect the static page to airtest ide
3. The static page disconnects airtest IDE and does not exit ide
4. Disconnect airtest ide from static page and exit ide

FPS data
Perfdog helps to explore automation performance testing
Whether IDE is turned on or not has no effect on the FPS of the application

Memory
Perfdog helps to explore automation performance testing
Memory doesn’t matter

CPU utilization
Perfdog helps to explore automation performance testing

Just like the conclusion of the first group, it is also true that turning on the IDE will affect the total CPU utilization rate. It should be noted that the performance consumption will still remain after the IDE is disconnected from the mobile phone, because the mincap service has not been interrupted in practice. It is necessary to exit and shut down the IDE CPU before it can return to normal.

The third set of data

The selected one is a mobile app, not a game

FPS
Perfdog helps to explore automation performance testing
Memory
Perfdog helps to explore automation performance testing
CPU
Perfdog helps to explore automation performance testing
We found the same conclusion as above

Standardized CPU utilization recommended

Why is this value recommended as a measure of CPU utilization? Because it is found that normalization is more suitable for automation and is more accurate. Documents on normalized utilization rate:
Introduction to standardized utilization rate
Perfdog helps to explore automation performance testing
Perfdog helps to explore automation performance testing

conclusion

We can use the automatic way to obtain the performance data of the application. This is because the data we get is for a single application, so the automatic operation will not be within the application. However, the access to the automatic SDK should be considered separately, and the resources consumed by the SDK will be counted on the application head.