Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a “physical examination”

Time:2021-10-27

Abstract: Huawei cloud AI development platform modelarts black technology supports ai r & D, makes model development more efficient and simpler, and reduces the landing threshold of AI in the industry. Comprehensive visual evaluation and intelligent diagnosis function enable developers to intuitively understand all aspects of the performance of the model, so as to carry out targeted tuning.

This year, the newly released Huawei cloud modelarts has attracted much attention, and innovative black technology has become the favorite of the AI community.

Model building has always been one of the challenges encountered by the AI landing industry, especially how to evaluate and diagnose the models to be deployed online, so as to ensure whether the R & D meets the business needs, ensure the quality level of AI products, directly put the models with excellent evaluation results into the production environment, and optimize the unsatisfactory models.

Huawei cloud AI development platform modelarts black technology supports ai r & D, makes model development more efficient and simpler, and reduces the landing threshold of AI in the industry. Comprehensive visual evaluation and intelligent diagnosis function enable developers to intuitively understand all aspects of the performance of the model, so as to carry out targeted tuning.

Comprehensive physical examination for AI model to reduce the threshold of AI R & D

For a long time, model tuning usually relies too much on the personal experience of data scientists or Algorithm Engineers, which leads to high threshold and high cost of developing an optimal AI model. On the other hand, in addition to the common indicators such as model accuracy, in order to deploy the model in the production environment, we must consider its performance, interpretability, robustness, anti attack and other indicators. For most data scientists or Algorithm Engineers, the analysis and optimization of these indicators are not very intuitive. This makes it difficult for AI models to enter the production environment.

In order to solve the above two problems, Huawei cloud modelarts helps users automatically analyze and evaluate various indicators of the model through the built-in tools of the platform, which is equivalent to generating the “physical examination report” of the model and giving optimization suggestions. Users can further improve the effect of the model or decide whether to deploy it to the production environment according to the optimization suggestions.

Model evaluation and diagnosis are more intelligent and tuning is more targeted

Modelarts can provide analysis and diagnosis of accuracy, sensitivity, computational performance, antagonism and other indicators, and assist in thermal diagram, abstract feature distribution and other means to understand the model and data distribution.

1. Visually present the predicted value and real value of the sample, and the data analysis is more intuitive

In terms of accuracy, modelarts provides common indicators such as confusion matrix and ROC to visualize the predicted value and real value of each sample, which is convenient for users to conduct in-depth analysis based on the original data. For example, in the flower recognition scene, due to the wide variety of flowers and the small difference of flowers under the same category, the accuracy requirements of AI model will be higher. Modelarts accurately evaluates the accuracy of image recognition model to ensure the accuracy of flower recognition.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Display of prediction results of image classification model based on modelarts

2. Segment data sets, refine model sensitivity analysis, and focus more on optimization suggestions

In terms of sensitivity analysis, modelarts divides the original data set into different subsets according to some basic characteristics of the data set (such as image brightness, clarity and other characteristics, or text length and other characteristics), and then evaluates the accuracy indicators of the model on different subsets to analyze the sensitivity of the model to different features, so as to provide direction for model optimization. When the influence of image brightness on model accuracy fluctuates greatly, the brightness needs to be enhanced when retraining.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Sensitivity analysis of abstract features based on modelarts and its diagnosis suggestions

3. Go deep into the operator optimization of AI underlying system and analyze the computational performance of concrete model

In terms of computational performance, modelarts can conduct fine-grained performance analysis on the models trained by common frameworks, analyze the reasoning duration and parameter distribution of operators, and provide direction for operator optimization and model compression.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Performance analysis of model computing based on modelarts

4. Built in a variety of attack algorithms, multi-dimensional evaluation model antagonism

In terms of adversarial evaluation, modelarts has built-in a variety of attack algorithms to attack the model and output the attack success rate and the accuracy index of the attacked model, as shown in Figure 4. According to the diagnostic recommendations, it can be found that the model is more vulnerable in some categories and can be optimized through further data enhancement. Modelarts can also automatically prompt potential vulnerable operators in the model. As can be seen from Figure 5, the ROC curve after being attacked is obviously not as good as that before being attacked. Therefore, it is not recommended to deploy the model directly to the production environment, but to further optimize it according to the diagnostic recommendations of modelarts.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Figure 4. Evaluation and diagnosis of model adversary attack based on modelarts

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

(a)

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

(b)

Figure 5. ROC curve of model before (a) and after (b) attack

Intelligent evaluation and diagnosis of three scenarios, and targeted model tuning

At present, model intelligent evaluation and diagnosis supports three types of models and data sets, “image classification”, “object detection” and “semantic segmentation”, so that users can fully understand the adaptability of the model to different data features, so that model optimization can be targeted.

If you want to classify the images of five categories of food, use RESNET subscribed by modelarts market_ V1_ 50 algorithm training, the model obtained from the first training is evaluated on the test set, and the accuracy is 54.8%. In the sensitivity analysis, it can be found that the model has a very high sensitivity to image brightness, and the variance of model accuracy in different brightness intervals reaches 0.249. It is recommended to expand the brightness data.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

After the data were amplified, the model was retrained and evaluated with the same test set. The accuracy increased to 63.4%, and the variance of model accuracy in different brightness intervals decreased to 0.217.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

In the helmet detection task, the object detection efficientdet algorithm subscribed by modelarts market is used for training, and 74.8% map is obtained on the verification set.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

Expand the data according to the suggestions, click the second training button, and the recommended training parameters will be automatically loaded into the new training task.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

After the training, check the evaluation results and verify that the set map has been significantly improved.

Modelarts black technology reveals the secret of intelligent evaluation and diagnosis of the model, so that the model can have a

The black technology function of Huawei cloud modelarts is more than automatic model evaluation and diagnosis. Using modelarts, the robot dog can realize industrial patrol inspection, automatically identify fire, plan path and close fire button; Forest rangers in the rainforest conservation organization can better protect endangered animals such as spiders and monkeys; Students of Tongji University have established a set of wetland digital twin system based on modelarts platform to help migratory birds in Hangzhou Bay get better protection and scientific research… In the future, we will bring you more secrets of modelarts innovative technology.

Huawei cloud modelarts creates the strongest AI black technology through continuous innovation, so that AI can better enter production and life, accelerate the landing and deepening of AI in the industry, expand the coverage of AI applications, further release AI technology dividends, and enable more enterprises to use AI for transformation and upgrading.

Click focus to learn about Huawei cloud’s new technologies for the first time~