How is the question answering robot with 90% matching rate refined?


Author: Tian ye, natural language processing expert of Netease smart enterprise

The core task of customer service robot is to effectively answer users’ various questions. Answering questions requires knowledge. Usually, the knowledge used in customer service system is expressed in the form of question answer pairs. This information is usually sorted out by students who understand the actual business scenario. In customer service system, the collection of these question answer pairs is usually named knowledge base, The following example is a fragment of a typical knowledge base in the HRSSC scenario.

  • Question 1: how many days of paid annual leave do employees have?

Answer: seven days.

  • Question 2: when is the payday?

Answer: the 5th of every month.

  • Question 3: what is the proportion of provident fund paid by the company?

Answer: 12% of the payment base.

It can be imagined that different scenarios will have completely different knowledge bases. Even in a fixed scenario, different companies will set up different knowledge bases due to different policies. For example, the HRSSC system knowledge base of a company that has purchased additional commercial insurance for its employees may have such a question and answer dialogue:

  • Question 4: how to apply for the additional commercial insurance presented by the company?

Answer: it will take effect automatically during the period of employment, and no special application is required.

In addition, additional functions can be easily realized through the knowledge base. For example, the knowledge base of an e-commerce company may have the following knowledge points:

  • Question 1: why hasn’t the goods been delivered? The service is terrible!

Answer: I’m sorry, pro, it brought you a bad shopping experience. Please don’t be impatient. The goods are running towards you.

The standard question here is not a question in the traditional sense, but a complaint and complaint. Through the answer of the knowledge point, the customer service robot provides emotional comfort function.

Matching rate of robot question answeringIt is the core index to measure the intelligence of customer service system

The matching rate of robot Q & A = the number of user’s questions correctly matched to several knowledge points in the knowledge base / the number of all questions input by the user

In other words,The matching rate of robot Q & A is the rate that the customer service robot can select the right question from the knowledge base and give a reply, that is, the rate that the robot can answer correctly.

It is not difficult to understand that this indicator directly determines the customer satisfaction of the customer service system.

It is a great challenge for the customer service system that this index reaches a high value. The reason is that for the same meaning, the broad and profound Chinese has almost infinite expressions. For example, let’s list some expressions to describe the beauty of a young lady

The man by the fireside is like the moon, his white wrist is frozen with snow and frost.
When you look back, you can see that Liugong pink and Dai have no color.
Quiet like delicate flowers shining on the water, action like weak willows supporting the wind.
The birds are startled by the sinking fish and the falling geese, and the flowers tremble with shame.
Under the moonlight, you come to me with a smile. Between the moonlight and the snow, you are the third kind of beauty.
Whenever you come to me, you see the stars and the sea.
You put the blue sky on your body and light up my heart.

Obviously, there are infinite expressions of the same meaning, which cannot be exhausted in the knowledge base in advance. When the robot queries the knowledge base, it must be able to correctly deal with the user problems that are not encountered in advance, but there are similar knowledge points in the knowledge base. This is the key and difficult point to improve the matching rate of Q & A.As an expert in integrated service marketing solutions with leading AI capabilities in the industry, Netease Qiyu’s robot Q & a matching rate is as high as 90%. How is it achieved?

1. Knowledge base construction

The rationality and scope of the knowledge base classification system of customer service system is the most important basis of matching rate. According to the above formula, we can imagine two extreme cases

  • Without any knowledge input, the cost of knowledge construction is zero, and the probability of users’ questions being correctly matched is zero, that is, the matching rate of robot question answering is zero;
  • The amount of knowledge in the knowledge base is infinite, and the construction cost of knowledge is infinite, that is, it contains all the questions and answers that users may ask. Then, using a simple string perfect matching algorithm, the matching rate of robot Q & A is 100%;

The actual knowledge base is between these two extremes, and the construction of knowledge base system suitable for customer service needs professional knowledge. Most customers have no practical experience in building knowledge base when they try intelligent customer service system for the first time. Netease Qiyu provides professional robot assistant trainer service. These trainers have dealt with the cold start construction of knowledge base in a variety of industries and scenarios, and have rich experience. They can help customers quickly build knowledge base in the cold start phase of the project, so as to quickly form a set of available customer service system. The following figure shows the cold start process of a typical customer’s knowledge base

How is the question answering robot with 90% matching rate refined?

Let’s compare the knowledge base built by the team of seven fish trainers with the knowledge base created by customers who have no actual cold start experience for intelligent robots

Knowledge base before sorting out:

How is the question answering robot with 90% matching rate refined?

The knowledge base organized by the trainer is as follows:

How is the question answering robot with 90% matching rate refined?

It can be seen that the knowledge base sorted out by the trainer contains more effective similar questions with rich expressions, and eliminates the mistakes contained in the case of the customer building the knowledge base by himself (for example, “employee opens a letter of introduction” and “salary asks who” are arranged to be the same knowledge point).

2. Knowledge base maintenance

After the completion of the cold start phase, the system starts to run online. How to continuously improve the knowledge base in the operation of customer service system becomes the key problem to maintain and improve the matching rate of robot Q & A. Knowledge base maintenance of online system needs to be considered from two dimensions of stock knowledge and incremental knowledge.

For stock knowledge

Due to the large mobility of personnel in the customer service industry, the operation of the system side often changes. Different customer service operators have information gap with each other, which is prone to highly repetitive knowledge points being added to the knowledge base, and eventually the knowledge base becomes redundant and inefficient. In order to solve this problem, Netease seven fish provides the function of knowledge duplicate checking, which can effectively find the knowledge points with high repetition and reduce the system self-interference. The following figure is an example:

How is the question answering robot with 90% matching rate refined?

It can be seen that the knowledge duplication function detects that a large number of similar questions are distributed under different knowledge points, which indicates that it is necessary to sort out the knowledge base.

For incremental knowledge

According to the online operation, Qiyu provides two function points: “guess what you want to ask, similar question summary” and “unknown question clustering” to help the knowledge base operators improve their work efficiency.

Guess the similar question you want to ask:

The following figure is an example of “guess what you want to ask”. When does the canteen open? “The user asks. There are no very close knowledge points in the knowledge base of the system, but the algorithm finds three relatively close knowledge points and submits them to the user as” guess what you want to ask “candidates. If the user makes a choice, such as “1” in the example, the question “restaurant opening time” selected by the user and the question “when will the canteen open” entered by the user will be retained in the background. After that, the operation students can decide whether they need to add “when the canteen opens” to the knowledge point of “restaurant opening time”.

How is the question answering robot with 90% matching rate refined?

Unknown problem discovery

During the operation of the system, there are many unanswered problems in the knowledge base, which are called unknown problems. Seven fish system provides the ability of text clustering algorithm, which can gather these sentences with similar semantics

  • Where can new crown vaccine hit?
  • When can we start vaccination?
  • How to apply for new crown vaccination?

This kind of new users’ question probability generated with the change of the scene will not be included by the existing knowledge points. Seven fish’s unknown problem discovery function can effectively find and classify such unknown problems to the operation students, and then the operation students can decide whether to add corresponding knowledge points in the knowledge base. At the same time, the discovery of unknown problems also plays a role in public opinion monitoring. For example, text clustering finds the following similar problems:

  • Today’s apples are so rotten.
  • The quality of apples is not good.
  • Why are your apples so poor in quality.
  • The apples are not fresh and many of them are rotten.

The above unknown questions are concentrated to prompt businesses that there may be quality control problems. Businesses can quickly make management actions to solve the problems according to the alarm information.

3. Core algorithm of sentence similarity matching

With a wide range of content and reasonable structure of the knowledge base, the next step is the responsibility of the algorithm module to find the correct matching knowledge points and return the answers. For the same knowledge point, the Chinese language, which is the result of five thousand years of culture, provides endless ways of expression. The similarity matching algorithm of user’s question and knowledge point sentence in knowledge base is the core module of seven fish robot. The ability of this module directly determines the question answering matching rate of customer service system. The reason why the Q & a matching rate of Qiyu customer service system remains at the industry-leading 90 +% level mainly depends on continuous R & D investment to ensure that the algorithm ability of Qiyu is always at the forefront of the industry. The following is a list of some similarity matching algorithms used in Qiyu customer service system, which are described from two dimensions: the modeling of a single sentence (using number to represent a single sentence) and the modeling of the relationship between two sentences (using number to represent the relevance of two sentences to be matched).

An algorithm that represents a single statement

Before calculating the similarity of two sentences, it is necessary to find an appropriate mathematical model to represent the sentence (that is, to use a certain number to represent the sentence). Seven fish internal sentence representation methods include bag of words model, vector space model, static word embedding model and pre training based model.

Bag of words model uses set to represent a single sentence, each word is an element in the set, and the words are independent of each other, that is, the order and dependence between words are not considered. Usually, we define a dictionary in advance for Chinese. The sentence representation of the bag of words model is a vector whose length is the same as the size of the dictionary, and each element represents the number of times the corresponding word appears

  • Dictionary: [Lao Wang, Lao Zhang, want to go, Beijing, Shanghai, travel]
  • Lao Wang wants to travel to Beijing
  • Sentence representation: [1, 0, 1, 1, 0, 1]

The example sentence contains four words “Lao Wang”, “want to go”, “Beijing” and “tourism”, and these words only appear once. The indexes of these words in the dictionary are 0, 2, 3 and 5 respectively, so the element value of these positions in the sentence representation vector is 1, and the other elements are 0.

The bag of words model uses the word frequency as the value of the elements in the sentence vector, and can further generate the vector space model representation of the sentence by using indicators with richer semantic information, such as TF-IDF indicator which represents the importance of words. TF-IDF is the product of TF: term frequency and IDF: inverse document frequency. The idea is that if a word appears more frequently in the current document than in other documents, then it has a good discrimination. The word frequency can be obtained by counting the current sentence, while the inverse document frequency needs to traverse all documents, which is calculated according to the following formula:

How is the question answering robot with 90% matching rate refined?

After the TF-IDF value of each word is calculated, sentence vectors are constructed like bag of words model.

The semantic information of a word is often reflected in its relationship with other words. For example, words similar to chopsticks are more likely to be “eating” than “sleeping”. The static word vector model is a method to model the co-occurrence relationship between words, so as to obtain the word vector representation. Then, the vector representation of the sentence can be obtained by averaging the static word vectors of all the words in the sentence or through other more complex neural network structures (such as CNN, LSTM). “Static” in the static word vector means that for a given input sentence, the value of the word vector is fixed and independent of the corresponding context of the word. In other words, after the training of static word vector model, the static word vector of each word is fixed and does not change with the specific input sentence.

The pre training model uses large-scale corpus to learn the relationship between words in a specific context, and the word vector obtained is generally dynamic, that is, the value of a word vector depends on the context. In other words, the same word in different sentences will correspond to different dynamic word vectors. In order to acquire semantics accurately, the pre training model usually constructs learning tasks from the granularity of morphology and syntax

  • Mask a word and use the information above to predict what the word is, such as GPT.
  • Mask a word and use context information to predict what the word is, such as Bert.

The main tasks of syntactic features are as follows:

  • Given two sentences S1 and S2, judge whether their positions in the corpus are continuous, such as Bert.
  • Given two sentences S1 and S2, judge which sentence appears first in the corpus, such as Albert.
  • Given two sentences S1 and S2, judge whether S2 can be used as the reply of S1 in the chat scene, such as Ernie 1.
  • Given a set of sentences, the order of each sentence in the corpus is predicted, such as Ernie 2.

An algorithm for representing the relationship between two statements

When calculating the similarity between sentences, we need to consider not only how to express a single sentence with mathematical model, but also how to model the relevance between two sentences, especially the way and time of association between two sentences. According to the different time when two sentences interact, the similarity algorithm can be divided into three structures: double tower model, single tower model and interaction aggregation model.

Double tower structure is a similarity algorithm based on the representation of single sentence, which means that we first make an abstract understanding of two sentences in turn, and then compare the understanding of both sides. The model uses two encoders to encode the two sentences that need to be matched, and then calculates the similarity (usually cosine similarity) between the two sentences. Double tower means that two statements that need to be compared, such as two towers, have no interaction before the final similarity result is output, such as qa-lstm, bcnn, etc.

The single tower structure emphasizes the interaction between two sentences, which means to understand the two sentences together as a whole, and then draw a conclusion whether they are similar or not. This kind of model stitches two sentences together in some way, and then inputs them into the encoder to calculate the similarity. Single tower means that the two sentences to be matched are a whole, just like an isolated tower. The two sentences to be matched have information interaction from beginning to end, such as match pyramid, bet, etc.

Interactive aggregation structure can be regarded as a compromise of the above structure, which is similar to that we read two sentences in turn and make a preliminary single sentence understanding. Based on the results of single sentence understanding, we carry out the second step of joint understanding of two sentences at the global level, and finally get the result of sentence matching. The model first uses two encoders to encode the two sentences that need to be matched, then fuses the information of the two parts through some method and makes further reasoning, and finally outputs the similarity between the two sentences. The meaning of interactive aggregation is that the overall structure is like two towers with one or more “flyovers”. The lower layer is independent of each other, and the upper layer is associated, such as ESIM, bimpm, etc.

4 Summary

In this article, we detail the secret of the seven fish robot’s question and answer matching rate up to 90%

  • During the cold start, under the guidance of Qiyu professional AI trainer, the knowledge base is quickly built to realize the service online;
  • Effective maintenance of knowledge base and discovery of new knowledge points in the operation of the system;
  • Complete algorithm stack to serve various scenarios.

With the blessing of the above three links, the problem matching accuracy of seven fish robot has always occupied an absolute advantage in the industry.

I don’t know if you’ve ever stood in line in the bank just to pay the electricity bill, had a quarrel with a shop assistant when you waited for an hour in the store but failed to run the business well, and had the experience of waiting in line for a long time in the canteen and finally being told that the dishes were sold out. If none of the above is unforgettable, have you ever dragged your sick body to get up early in the morning for registration, or starved for a train ticket to go home on a cold day.

In the era of technology changing life and mobile Internet, all of the above needs can be completed on mobile phones without leaving home, and question answering robots play a key role in them.

As an expert in integrated service marketing solutions with leading AI capabilities in the industry, Netease Qiyu has been established in e-commerce, education, manufacturing, automobile and other industries to help the digital transformation of enterprise services. We are fortunate to serve thousands of families, improve the quality of life and create a better life. We are grateful for this era.If your enterprise is still troubled by the above pain points, you might as well make friends with Qiyu, and we will work together to embrace the star sea!

More technical dry cargo, welcome to NetEase [WeChat] + official account]