Since the model is developed based on the samples of a specific period, whether the model is suitable for the ethnic groups outside the development samples must be tested for stability. The population stability index (PSI) can measure the distribution differences of the scores of test samples and model development samples, and is the most common indicator of model stability evaluation. In fact, psi refers to whether the population distribution has changed for different samples or samples at different times after grading according to scores, that is, whether there is a significant change in the proportion of the number of people in each score range to the total number of people. The formula is as follows:
Here, AC and ex are the model output scores of different time periods. If psi is too large, it means that the score distribution of model output has changed greatly and the model needs to be updated.
Examples of PSI application:
1) Out of sample testing
Testing the stability of the model for different samples, such as training set and test set, can also see the training situation of the model. I understand that it is to see the variance of the model.
2) Out of time testing
The further the distance between the test base date and the modeling base date, the greater the difference between the risk characteristics of the test samples and the modeling samples, so the PSI value is usually higher. At this point, we can see that the time of building the model is too long, and whether we need to use new samples to build the model again.
PSI calculation of variables:
PSI: test the stability of variables. When the PSI value of a variable is greater than 0.0001, the variable is unstable.
For a variable, group its value according to the quantile, subtract the proportion of customers in the training model from the proportion of customers in the test model in each group, and then multiply by the logarithm of the division of the two,
This is the stability coefficient psi of this group, and then the PSI coefficient of the variable is the sum of the PSI of all groups of this variable.
Population stability index of PSI
PSI = sum ((actual proportion – expected proportion) / ln (actual proportion / expected proportion))
For example, a scorecard model calculates psi by day. We take the proportion of users in each score segment on the first day (set as a base period) of the model as the expected proportion P1, and then the proportion of users in each score segment every day as the actual proportion P2. In this way, we can calculate the daily psi value according to the formula, and observe the size and trend of these psi, So as to monitor the stability of the score card. Usually, psi is calculated in the dimensions of day, week and month, and each characteristic variable in the scorecard model is monitored by psi.
The change of model scores may be caused by the change of characteristics, or the instability of the model itself. If the total number of high segments does not change and the PSI value changes greatly, it is considered that the model needs to be retrained.
If the PSI value does not change, the total number of high segments will increase, and the overall user will be better.