Tensorflow chapter | tensorflow 2. X model construction based on keras

Time:2020-10-28

Tensorflow chapter | tensorflow 2. X model construction based on keras

IntroductionSince tensorflow version 2.0 was released, keras has been deeply integrated into tensorflow framework, and keras API has become the first choice for building deep network model. Using keras for model development and iteration is a basic skill that every data developer needs to master. Let’s explore the world of keras together.

Introduction to keras

  1. KerasIt’s a utilityPythonHigh level neural networkAPI, which is a stand-alone library that enablesTensorFlowCNTKperhapsTheanoRun as a back end.TensorFlowfrom1.0Version started to try andKerasIntegration, to2.0After the release, it has been deeply integratedKeras, and rely on it closelytf.kerasAs a senior member of the central governmentAPIIt is highly recommended by the governmentkeras APITo complete the construction of depth model.
  2. tf.kerasIt has three key advantages:

    1. User friendly to Xiaobai:KerasIt has a simple and consistent interface, and has clear and feasible suggestions to correct the user’s errors.TensorFlow 2.0In previous versions, due to the complexity of the code,APIThe interface is confused and the compatibility between different versions is poorKerasAfter unification, the workload of developers will be greatly reduced.
    2. Modular and composable:KerasThe model can be easily constructed through any model that can be easily linked together without any constraints.
    3. Easy to expand: when writing a new user-defined module, it can be very convenient to expand based on the existing interface.
  3. KerasbringTensorFlowEasier to use without losing flexibility and performance.

Construction of keras model

stayTensorFlow 2.xThere are three ways to buildKerasThe models areSequentialFunctional APIas well asCustom model (subclassed)。 The following describes the three construction methods.

Sequential Model

Tensorflow chapter | tensorflow 2. X model construction based on keras

  1. stayKerasIn, it is usually used to combine multiple layers(layer)Put it together to form a model(model)The most common way is to stack layers. You can usetf.keras.SequentialTo make it easy. The model shown in the above figure is an example, and its code implementation is as follows:

    import tensorflow as tf
    from tensorflow.keras import layers
    
    model = tf.keras.Sequential()
    # Adds a densely-connected layer with 64 units to the model:
    model.add(layers.Dense(64, activation='relu', input_shape=(16,)))
    # This is identical to the following:
    # model.add(layers.Dense(64, activation='relu', input_dim=16))
    # model.add(layers.Dense(64, activation='relu', batch_input_shape=(None, 16)))
    # Add another:
    model.add(layers.Dense(64, activation='relu'))
    # Add an output layer with 10 output units:
    model.add(layers.Dense(10))
    # model.build((None, 16))
    print(model.weights)
  2. Pay attention to theSequentialThe first layer added can contain ainput_shapeorinput_dimorbatch_input_shapeParameter to specify the dimension of the input data. See the notes section for details. When theinput_shapeAfter the parameters are equal, each timeaddNew layers and models are in the process of continuous creation, that is to say, at this time, the weight matrix of each layer in the model has been initialized and can be calledmodel.weightsTo print the weight information of the model.
  3. Of course, the first layer can also not contain the dimension information of input data, which is called delay creation mode. That is to say, the model has not been created and the weight matrix does not exist at this time. You can callmodel.build(batch_input_shape)Method to create the model manually. If it is not created manually, thefitOr other training and evaluation methods, the model will be created, the weight matrix will be initialized, at this time, the model will automatically infer its dimension information according to the input data.
  4. input_shapeNot specified inbatchAnd set it toNoneBecause of the training and evaluationbatchThe size may be inconsistent. If it is set to a constant value, errors will be generated during training or evaluation. After this setting, it can be automatically inferred by the modelbatchIt is more robust when the size is calculated.
  5. In addition to this sequential addition(add)In addition, we can also use thelayersPass as a parameter toSequentialTo build the model. The sample code is as follows:

    import tensorflow as tf
    from tensorflow.keras import layers
    
    model = tf.keras.Sequential([
        layers.Dense(64, activation='relu', input_shape=(16, )),
        layers.Dense(64, activation='relu'),
        layers.Dense(10)
    ])
    # model.build((None, 16))
    print(model.weights)

Functional API

  1. KerasOfFunctional APIIt’s better than thatSequentialA more flexible way to create models. It can deal with models with nonlinear topological structure and share layer(layers)And the model of multi input and output. The model of deep learning is usually composed of layers(layers)A directed acyclic graph is formedFunctional APIIt is an effective way to build such a graph.
  2. withSequential ModelFor example, use theFunctional APIThe implementation is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    inputs = keras.Input(shape=(16, ))
    dense = layers.Dense(64, activation='relu')
    x = dense(inputs)
    x = layers.Dense(64, activation='relu')(x)
    outputs = layers.Dense(10)(x)
    model = keras.Model(inputs=inputs, outputs=outputs, name='model')
    model.summary()
  3. And useSequentialThe difference of method construction model is that,Functional APIadoptkeras.InputInput specifiedinputsAnd passedfunction callTo generate the outputoutputs, finally usekeras.ModelMethods the whole model was constructed.
  4. WhyFunctional APIAs you can see from the code, you can use various layers like function calls(layers)For example, it’s defineddenseLayer, you can directly addinputsAsdenseTo get an outputx, and thenxAs the input of the next layer, the final function return value is the output of the whole model.
  5. Functional APIYou can put the same layer(layers)As part of multiple models, the sample code is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    encoder_input = keras.Input(shape=(16, ), name='encoder_input')
    x = layers.Dense(32, activation='relu')(encoder_input)
    x = layers.Dense(64, activation='relu')(x)
    encoder_output = layers.Dense(128, activation='relu')(x)
    
    encoder = keras.Model(encoder_input, encoder_output, name='encoder')
    encoder.summary()
    
    x = layers.Dense(64, activation='relu')(encoder_output)
    x = layers.Dense(32, activation='relu')(x)
    decoder_output = layers.Dense(16, activation='relu')(x)
    
    autoencoder = keras.Model(encoder_input, decoder_output, name='autoencoder')
    autoencoder.summary()

    The code contains two models, an encoder(encoder)And a self encoder(autoencoder)As you can see, the two models are sharedencoder_outLayers, of courseencoder_outAll layers before the layer.

  6. Functional APIAll models generated(models)Can be like layers(layers)Also called. Self encoder is also used(autoencoder)For example, it is now divided into encoders(encoder)And decoder(decoder)Two parts, then useencoderanddecodergenerateautoencoderThe code is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    encoder_input = keras.Input(shape=(16, ), name='encoder_input')
    x = layers.Dense(32, activation='relu')(encoder_input)
    x = layers.Dense(64, activation='relu')(x)
    encoder_output = layers.Dense(128, activation='relu')(x)
    
    encoder = keras.Model(encoder_input, encoder_output, name='encoder')
    encoder.summary()
    
    decoder_input = keras.Input(shape=(128, ), name='decoder_input')
    x = layers.Dense(64, activation='relu')(decoder_input)
    x = layers.Dense(32, activation='relu')(x)
    decoder_output = layers.Dense(16, activation='relu')(x)
    
    decoder = keras.Model(decoder_input, decoder_output, name='decoder')
    decoder.summary()
    
    autoencoder_input = keras.Input(shape=(16), name='autoencoder_input')
    encoded = encoder(autoencoder_input)
    autoencoder_output = decoder(encoded)
    autoencoder = keras.Model(
        autoencoder_input,
        autoencoder_output,
        name='autoencoder',
    )
    autoencoder.summary()

    Two models are first generated in the codeencoderanddecoder, and then generateautoencoderWhen modeling, theModel function callThe way to directlyautoencoder_inputandencodedSeparately asencoderanddecoderThe input of the two models is obtainedautoencoderModel.

  7. Functional APIIt is easy to deal with multiple input and multiple output models, which areSequential APIImpossible. For example, part of our model input isCategorical featuresIn general, you need to go throughEmbeddingProcessing, and part of it isNumerical characteristicsIn general, there is no need for special processing. Obviously, these two features can not be directly combined as a single input for common processing, so multiple input will be used. Sometimes we want the model to return more than one output for subsequent calculation. In this case, we will use the multi output model. Example codes for the multiple input and multiple output models are as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    categorical_input = keras.Input(shape=(16, ))
    numeric_input = keras.Input(shape=(32, ))
    categorical_features = layers.Embedding(
        input_dim=100,
        output_dim=64,
        input_length=16,
    )(categorical_input)
    categorical_features = layers.Reshape([16 * 64])(categorical_features)
    numeric_features = layers.Dense(64, activation='relu')(numeric_input)
    x = layers.Concatenate(axis=-1)([categorical_features, numeric_features])
    x = layers.Dense(128, activation='relu')(x)
    
    binary_pred = layers.Dense(1, activation='sigmoid')(x)
    categorical_pred = layers.Dense(3, activation='softmax')(x)
    
    model = keras.Model(
        inputs=[categorical_input, numeric_input],
        outputs=[binary_pred, categorical_pred],
    )
    model.summary()

    There are two inputs in the codecategorical_inputandnumeric_inputAfter different processing layers, the two pass throughConcatenateFinally, two outputs are obtained through different processing layersbinary_predandcategorical_pred。 The structure of the model is shown in the following figure:

    Tensorflow chapter | tensorflow 2. X model construction based on keras

  8. Functional APIAnother good use is model layer sharing, that is, in a model, the layer is repeatedly used, and it learns different features from different inputs. One common shared layer is the embedded layer(Embedding)The code is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    categorical_input_one = keras.Input(shape=(16, ))
    categorical_input_two = keras.Input(shape=(24, ))
    
    shared_embedding = layers.Embedding(100, 64)
    
    categorical_features_one = shared_embedding(categorical_input_one)
    categorical_features_two = shared_embedding(categorical_input_two)
    
    categorical_features_one = layers.Reshape([16 * 64])(categorical_features_one)
    categorical_features_two = layers.Reshape([16 * 64])(categorical_features_two)
    
    x = layers.Concatenate(axis=-1)([
        categorical_features_one,
        categorical_features_two,
    ])
    x = layers.Dense(128, activation='relu')(x)
    outputs = layers.Dense(1, activation='sigmoid')(x)
    
    model = keras.Model(
        inputs=[categorical_input_one, categorical_input_two],
        outputs=outputs,
    )
    model.summary()

    There are two inputs in the codecategorical_input_oneandcategorical_input_two, they share aEmbeddinglayershared_embedding。 The structure of the model is shown in the following figure:

    Tensorflow chapter | tensorflow 2. X model construction based on keras

Custom keras layer and model

  1. tf.kerasThere are many built-in layers under the module(layers)Like the one we used aboveDenseEmbeddingReshapeEtc. Sometimes we will find that these built-in layers can not meet our needs. At this time, it is easy to create custom layers to expand. Custom layer through inheritancetf.keras.LayerClass, and the subclass implements thebuildandcallmethod. For built-inDenseLayer. If the user-defined layer is used, the code is as follows:

    import tensorflow as tf
    from tensorflow import keras
    from tensorflow.keras import layers
    
    class CustomDense(layers.Layer):
        def __init__(self, units=32):
            super().__init__()
            self.units = units
    
        def build(self, input_shape):
            self.w = self.add_weight(
                shape=(input_shape[-1], self.units),
                initializer='random_normal',
                trainable=True,
            )
            self.b = self.add_weight(
                shape=(self.units, ),
                initializer='random_normal',
                trainable=True,
            )
    
        def call(self, inputs):
            return tf.matmul(inputs, self.w) + self.b
    
        def get_config(self):
            return {'units': self.units}
    
        @classmethod
        def from_config(cls, config):
            return cls(**config)
    
    inputs = keras.Input((4, ))
    layer = CustomDense(10)
    outputs = layer(inputs)
    
    model = keras.Model(inputs, outputs)
    model.summary()
    
    # layer recreate
    config = layer.get_config()
    new_layer = CustomDense.from_config(config)
    new_outputs = new_layer(inputs)
    print(new_layer.weights)
    print(new_layer.non_trainable_weights)
    print(new_layer.trainable_weights)
    
    # model recreate
    config = model.get_config()
    new_model = keras.Model.from_config(
        config,
        custom_objects={'CustomDense': CustomDense},
    )
    new_model.summary()
    1. among__init__Method is used to initialize some basic parameters needed to build the layer,buildMethod is used to create the weight matrix required for this layerwSum deviation matrixbcallMethod is the real Performer of the layer build, which converts input to output and returns. In fact, the weight matrix can also be created in the__init__Method, but in many cases, we can’t predict the dimension of the input data in advance, and we need to delay the creation of the weight matrix at a certain time point in the instantiation layer, so we need to set up the weight matrix in thebuildMethod according to the dimension information of input datainput_shapeTo dynamically create a weight matrix.
    2. The call order of the above three methods is__init__buildcallIn which__init__Is called when the layer is instantiated, andbuildandcallIt is called after the input is determined. actuallyLayerClass has a built-in method__call__The method is called first when the layer is built, and inside the methodbuildandcallAnd only for the first time__call__Triggered only whenbuildThat is to saybuildVariables in can only be created once, andcallIt can be called many times, such as training and evaluation.
    3. If you need to provide serialization support for this layer, you need to implement aget_configMethod to return the constructor parameters of the instance of the layer as a dictionary. In the givenconfigBy calling theClass method from_ configTo recreate the layer,from_configThe default implementation of the layer is shown in the code, and the layer re creation is shown in thelayer recreateThe code part, of course, can also be rewrittenfrom_configClass method to provide new creation methods. And recreate the new model(model)Code andlayerThe reconstructed code is different, and it needs the help ofkeras.Model.from_configMethod to complete the construction, seemodel recreateCode part.
  2. Custom layers can be combined recursively, that is, one layer can be used as an attribute of another layer. It is generally recommended that__init__Method, because the sublayer has its ownbuildThe method will be in the outer layerbuildWhen called, it is triggered to perform the construction task of the weight matrix, without displaying the creation in the parent layer. Also withSequential ModelAs an example, the code is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    class MLP(layers.Layer):
        def __init__(self):
            super().__init__()
            self.dense_1 = layers.Dense(64, activation='relu')
            self.dense_2 = layers.Dense(64, activation='relu')
            self.dense_3 = layers.Dense(10)
    
        def call(self, inputs):
            x = self.dense_1(inputs)
            x = self.dense_2(x)
            x = self.dense_3(x)
            return x
    
    inputs = keras.Input((16, ))
    mlp = MLP()
    
    y = mlp(inputs)
    print('weights:', len(mlp.weights))
    print('trainable weights:', len(mlp.trainable_weights))

    As you can see from the code, we will have threeDenseLayer asMLPAnd then use them to completeMLPThe construction of, can achieve andSequential ModelIn addition, the weight matrix of all sub layers will exist as the weight matrix of new layer.

  3. Layer(layers)During the build process, the losses generated during this creation process are collected recursively(losses)。 In rewritingcallMethod, you can call theadd_lossTo increase the loss of custom methods. All the losses of a layer include the loss of its sublayer, and they can be passed throughlayer.lossesProperty, which is a list(list)Note that the loss of regular terms is automatically included. The sample code is as follows:

    import tensorflow as tf
    from tensorflow import keras
    from tensorflow.keras import layers
    
    class CustomLayer(layers.Layer):
        def __init__(self, rate=1e-2, l2_rate=1e-3):
            super().__init__()
            self.rate = rate
            self.l2_rate = l2_rate
            self.dense = layers.Dense(
                units=32,
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
    
        def call(self, inputs):
            self.add_loss(self.rate * tf.reduce_sum(inputs))
            return self.dense(inputs)
    
    inputs = keras.Input((16, ))
    layer = CustomLayer()
    x = layer(inputs)
    print(layer.losses)
  4. Of a layer or modelcallThe method preset has onetrainingParameter, which is aboolType of variable, indicating whether it is in the training state. It will set the value according to the method called. When training, it isTrueAt the time of evaluationFalse。 Because there are some layersBatchNormalizationandDropoutGenerally, it can only be used in the training process, but not in the process of evaluation and prediction. Therefore, the parameter can be used to control the different calculation process of the model in different states.
  5. The implementation of a custom model is similar to that of a custom layer, but the model needs to be inherited fromtf.keras.ModelModelSome of the classAPIYes andLayerFor example, the user-defined model should also be implemented__init__buildandcallmethod. But there are also differences between the twoModelIt has training, evaluation and prediction interfaces, and secondly it can pass themodel.layersView all built-in layer information, andModelClass also provides an interface for model saving and serialization. withAutoEncoderFor example, the example code of a complete custom model is as follows:

    from tensorflow import keras
    from tensorflow.keras import layers
    
    class Encoder(layers.Layer):
        def __init__(self, l2_rate=1e-3):
            super().__init__()
            self.l2_rate = l2_rate
    
        def build(self, input_shape):
            self.dense1 = layers.Dense(
                units=32,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
            self.dense2 = layers.Dense(
                units=64,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
            self.dense3 = layers.Dense(
                units=128,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
    
        def call(self, inputs):
            x = self.dense1(inputs)
            x = self.dense2(x)
            x = self.dense3(x)
            return x
    
    class Decoder(layers.Layer):
        def __init__(self, l2_rate=1e-3):
            super().__init__()
            self.l2_rate = l2_rate
    
        def build(self, input_shape):
            self.dense1 = layers.Dense(
                units=64,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
            self.dense2 = layers.Dense(
                units=32,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
            self.dense3 = layers.Dense(
                units=16,
                activation='relu',
                kernel_regularizer=keras.regularizers.l2(self.l2_rate),
            )
    
        def call(self, inputs):
            x = self.dense1(inputs)
            x = self.dense2(x)
            x = self.dense3(x)
            return x
    
    class AutoEncoder(keras.Model):
        def __init__(self):
            super().__init__()
            self.encoder = Encoder()
            self.decoder = Decoder()
    
        def call(self, inputs):
            x = self.encoder(inputs)
            x = self.decoder(x)
            return x
    
    model = AutoEncoder()
    model.build((None, 16))
    model.summary()
    print(model.layers)
    print(model.weights)

    The above code implements aAutoEncoder ModelClass, which consists of two layers, namelyEncoderandDecoder, and these two layers are also customized. By callingmodel.weightsYou can view all the weight information of the model, including all the weight information in the sub layer.

  6. For a custom layer or model, call thesummary, weights, variables, trainable_weightslossesWhen you wait for a method or property, you must first ensure that the layer or model has been created, or you may report an error or return to null. Pay attention to this when debugging the model.

Configuration layer

staytf.keras.layersThere are many predefined layers under the module, most of which have the same constructor parameters. The following describes some commonly used parameters. For the unique parameters of each layer and the meaning of parameters, you can query the official documents when using them, and the explanation of the documents is generally very detailed.

  1. activationRefers to the activation function, which can be set to a string such asreluoractivationsobjecttf.keras.activations.relu(), by defaultNoneThat is to say, linear relationship.
  2. kernel_initializerandbias_initializer, indicating the initialization method of weight matrix and deviation matrix in the layer, which can be set as string such asGlorotuniform
    perhapsinitializersobjecttf.keras.initializers.GlorotUniform(), by defaultGlorotuniformInitialization mode.
  3. kernel_regularizerandbias_regularizer, which represents the regularization method of weight matrix and deviation matrix. As described above, it can beL1orL2Regularization, such astf.keras.regularizers.l2(1e-3)By default, there is no regular entry.

Comparison of model creation methods

  1. When building a simpler model, useSequentialOf course, the most convenient way is to use the existingLayerComplete the process of rapid construction and verification.
  2. If the model is complex, it is best to useFunctional APIOr custom model. usuallyFunctional APIIt is a more advanced, easier and safer way to implement. It also has some features that the custom model does not have. However, when building models that are not easily represented as directed acyclic graphs, custom models provide more flexibility.
  3. Functional APIModel validation can be done in advance because it passesInputMethod specifies the input dimension of the model in advance, so when the input is not standardized, it will be found earlier, which is helpful for us to debug. However, the user-defined model does not specify the dimension of input data at the beginning, and it infers itself from the input data during the running process.
  4. useFunctional APIWriting code modularity is not strong, it is difficult to read, and through the user-defined model, you can clearly understand the overall structure of the model, easy to understand.
  5. In practical use, theFunctional APICombined with custom model, it can meet our various model building requirements.

Keras model creation skills

  1. When writing model code, you can refer to other people’s model building methods, and sometimes there will be a lot of harvest.
  2. Find the requiredtensorflowMethod, ifkerasIf there is an implementation provided under the module, the method is preferred. If not, thentfThe method under the module can make the code more compatible and robust.
  3. In the process of model creation, the built-in methods and properties of models and layers are often used, such assummaryweightsIn this way, we can look at the structure of the model from a global perspective, and help to find some potential problems.
  4. becauseTensorFlow 2.xThe model is used by defaultEager ExecutionDynamic graph mechanism is used to run code, so it can be printed anywhere in the codeTensorTo view its value and dimension information is very helpful in model debugging.

reference material

  1. Keras sequential model
  2. Keras functional API
  3. Keras writes custom layers and models

Recommended Today

Queue chain storage

Linked list realizes queue Create three files: queuelinked. H, queuelinked. C, queuelinkedtest. C queueLinked.h #ifndef QUEUE_LINKED_H_ #define QUEUE_LINKED_H_ #ifdef __GNUC__ #define DEPRECATED __attribute__( (deprecated) ) #elif defined(_MSC_VER) #define DEPRECATED __declspec( deprecated ) #else #define DEPRECATED #endif #ifndef PTOI #define PTOI( p ) ((int32_t)(int64_t)(p)) #endif #ifndef ITOP #define ITOP( i ) ((void *)(int64_t)(i)) #endif #define ADT […]