The use of seqeval, the evaluation module of NLP (23) sequence annotation algorithm

Time:2020-3-24

   in NLP, sequence annotation algorithm is a common deep learning model, but are we really familiar with the evaluation of sequence annotation algorithm?
In this paper, the author will evaluate the model effect of sequence annotation algorithm andseqevalUse.

Model effect evaluation of sequence annotation algorithm

In the sequence annotation algorithm, we will generally form the following sequence list, as follows:

['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']

The formats of general sequence annotation algorithms areBIO IOBESBMESAnd so on. Among them,entityIt refers to the non-O continuous tag sequence of the same type (such as per / LOC / ORG) starting from the B tag.
                          

  • Accuracy: accuracy = number of elements predicted / total number of elements
  • Precision: precision = number of predicted correct entities / total number of predicted entities
  • Recall rate: recall = number of predicted correct entities / total number of labeled entities
  • F1 value: F1 = 2Accuracy rateRecall rate / (accuracy rate + recall rate)

For example, we have the following real sequencey_trueAnd prediction seriesy_predAs follows:

y_true = ['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']
y_pred = ['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']

If there are 9 elements in one of the list, and the number of predicted elements is 6, then the accuracy is 2 / 3. The total number of labeled entities is 2, the total number of predicted entities is 3, and the number of predicted correct entities is 1, then precision = 1 / 3, recall = 1 / 2, F1 = 0.4.

Use of seqeval

In general, our algorithm of sequence annotation usesconlleval.plScript implementation, but this is done in Perl. In Python, there is also a third-party module for the model effect evaluation of the corresponding sequence annotation algorithm, which isseqeval, its official website is https://pypi.org/project/seqeval/0.0.3 /.
  seqevalSupportBIO IOBESAnnotation mode can be used to evaluate tasks such as named entity recognition, part of speech tagging, semantic role tagging, etc.
Two examples are given in the official website documents. The author has modified them as follows:
Example 1:

# -*- coding: utf-8 -*-
from seqeval.metrics import f1_score
from seqeval.metrics import precision_score
from seqeval.metrics import accuracy_score
from seqeval.metrics import recall_score
from seqeval.metrics import classification_report

y_true = ['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']
y_pred = ['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER']

print("accuary: ", accuracy_score(y_true, y_pred))
print("p: ", precision_score(y_true, y_pred))
print("r: ", recall_score(y_true, y_pred))
print("f1: ", f1_score(y_true, y_pred))
print("classification report: ")
print(classification_report(y_true, y_pred))

The output results are as follows:

accuary:  0.6666666666666666
p:  0.3333333333333333
r:  0.5
f1:  0.4
classification report: 
           precision    recall  f1-score   support

     MISC       0.00      0.00      0.00         1
      PER       1.00      1.00      1.00         1

micro avg       0.33      0.50      0.40         2
macro avg       0.50      0.50      0.50         2

Example 2:

# -*- coding: utf-8 -*-
from seqeval.metrics import f1_score
from seqeval.metrics import precision_score
from seqeval.metrics import accuracy_score
from seqeval.metrics import recall_score
from seqeval.metrics import classification_report

y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER']]
y_pred =  [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER']]

print("accuary: ", accuracy_score(y_true, y_pred))
print("p: ", precision_score(y_true, y_pred))
print("r: ", recall_score(y_true, y_pred))
print("f1: ", f1_score(y_true, y_pred))
print("classification report: ")
print(classification_report(y_true, y_pred))

The output is the same as above.

Using seqeval in keras

                         .
Download project on GitHubDL_4_NER, website: https://github.com/percent4/dl_4_ner. Modify the folder path in utils.py and the code of the model training part (DL ﹤ ner / Bi ﹤ LSTM ﹤ model ﹤ training. Py) as follows:

# -*- coding: utf-8 -*-
import pickle
import numpy as np
import pandas as pd
from utils import BASE_DIR, CONSTANTS, load_data
from data_processing import data_processing
from keras.utils import np_utils, plot_model
from keras.models import Sequential
from keras.preprocessing.sequence import pad_sequences
from keras.layers import Bidirectional, LSTM, Dense, Embedding, TimeDistributed


#Model input data
def input_data_for_model(input_shape):

    #Data import
    input_data = load_data()
    #Data processing
    data_processing()
    #Import dictionary
    with open(CONSTANTS[1], 'rb') as f:
        word_dictionary = pickle.load(f)
    with open(CONSTANTS[2], 'rb') as f:
        inverse_word_dictionary = pickle.load(f)
    with open(CONSTANTS[3], 'rb') as f:
        label_dictionary = pickle.load(f)
    with open(CONSTANTS[4], 'rb') as f:
        output_dictionary = pickle.load(f)
    vocab_size = len(word_dictionary.keys())
    label_size = len(label_dictionary.keys())

    #Process input data
    aggregate_function = lambda input: [(word, pos, label) for word, pos, label in
                                            zip(input['word'].values.tolist(),
                                                input['pos'].values.tolist(),
                                                input['tag'].values.tolist())]

    grouped_input_data = input_data.groupby('sent_no').apply(aggregate_function)
    sentences = [sentence for sentence in grouped_input_data]

    x = [[word_dictionary[word[0]] for word in sent] for sent in sentences]
    x = pad_sequences(maxlen=input_shape, sequences=x, padding='post', value=0)
    y = [[label_dictionary[word[2]] for word in sent] for sent in sentences]
    y = pad_sequences(maxlen=input_shape, sequences=y, padding='post', value=0)
    y = [np_utils.to_categorical(label, num_classes=label_size + 1) for label in y]

    return x, y, output_dictionary, vocab_size, label_size, inverse_word_dictionary


#Define deep learning model: Bi LSTM
def create_Bi_LSTM(vocab_size, label_size, input_shape, output_dim, n_units, out_act, activation):
    model = Sequential()
    model.add(Embedding(input_dim=vocab_size + 1, output_dim=output_dim,
                        input_length=input_shape, mask_zero=True))
    model.add(Bidirectional(LSTM(units=n_units, activation=activation,
                                 return_sequences=True)))
    model.add(TimeDistributed(Dense(label_size + 1, activation=out_act)))
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model


#Model training
def model_train():

    #The data set is divided into training set and test set, accounting for 9:1
    input_shape = 60
    x, y, output_dictionary, vocab_size, label_size, inverse_word_dictionary = input_data_for_model(input_shape)
    train_end = int(len(x)*0.9)
    train_x, train_y = x[0:train_end], np.array(y[0:train_end])
    test_x, test_y = x[train_end:], np.array(y[train_end:])

    #Model input parameters
    activation = 'selu'
    out_act = 'softmax'
    n_units = 100
    batch_size = 32
    epochs = 10
    output_dim = 20

    #Model training
    lstm_model = create_Bi_LSTM(vocab_size, label_size, input_shape, output_dim, n_units, out_act, activation)
    lstm_model.fit(train_x, train_y, validation_data=(test_x, test_y), epochs=epochs, batch_size=batch_size, verbose=1)


model_train()

The results of model training are as follows (the middle process is omitted):

......
12598/12598 [==============================] - 26s 2ms/step - loss: 0.0075 - acc: 0.9981 - val_loss: 0.2131 - val_acc: 0.9592

We modify the code in the line of lstm_model.fit as follows:

    lables = ['O', 'B-MISC', 'I-MISC', 'B-ORG', 'I-ORG', 'B-PER', 'B-LOC', 'I-PER', 'I-LOC', 'sO']
    id2label = dict(zip(range(len(lables)), lables))
    callbacks = [F1Metrics(id2label)]
    lstm_model.fit(train_x, train_y, validation_data=(test_x, test_y), epochs=epochs,
                   batch_size=batch_size, verbose=1, callbacks=callbacks)

The output result is:

12598/12598 [==============================] - 26s 2ms/step - loss: 0.0089 - acc: 0.9978 - val_loss: 0.2145 - val_acc: 0.9560
 - f1: 95.40
           precision    recall  f1-score   support

     MISC     0.9707    0.9833    0.9769     15844
      PER     0.9080    0.8194    0.8614      1157
      LOC     0.7517    0.8095    0.7795       677
      ORG     0.8290    0.7289    0.7757       745
       sO     0.7757    0.8300    0.8019       100

micro avg     0.9524    0.9556    0.9540     18523
macro avg     0.9520    0.9556    0.9535     18523

This is the strength of seqeval.
   for the use of seqeval in keras, please refer to the GitHub website of the project: https://github.com/chakki-works/seqeval.

summary

Thank you for reading. This sharing is over.
Welcome to my WeChat official account:Python crawler and algorithm

Reference website

  1. Calculation of accuracy and recall rate of sequence annotation: https://xuanlan.zhihu.com/p/56582082
  2. Official document of seqeval: https://pypi.org/project/seqeval/0.0.3/