Tensorflow 2.0 implements complex neural networks (multiple input multiple output NN, RESNET)

Time:2021-10-17

Common ‘fusion’ operations

The realization of complex neural network model is inseparable from “fusion” operation. Common fusion operations are as follows:

(1) Sum, difference

  1. #Sum
  2. layers.Add(inputs)
  3. #Difference
  4. layers.Subtract(inputs)

Inputs: a list of input tensors (the size of the list is at least 2). The shapes of the list must be the same to perform the sum (difference) operation.

example:

  1. input1 = keras.layers.Input(shape=(16,))
  2. x1 = keras.layers.Dense(8, activation=’relu’)(input1)
  3. input2 = keras.layers.Input(shape=(32,))
  4. x2 = keras.layers.Dense(8, activation=’relu’)(input2)
  5. added = keras.layers.add([x1, x2])
  6.  
  7. out = keras.layers.Dense(4)(added)
  8. model = keras.models.Model(inputs=[input1, input2], outputs=out)

(2) Multiplication

  1. #The element by element product of the input tensor (the corresponding position element is multiplied, and the input dimension must be the same)
  2. layers.multiply(inputs)
  3. #Dot product between input tensor samples
  4. layers.dot(inputs, axes, normalize=False)

Dot is matrix multiplication. Example 1:

  1. x = np.arange(10).reshape(1, 5, 2)
  2.  
  3. y = np.arange(10, 20).reshape(1, 2, 5)
  4.  
  5. #For three-dimensional input, dot usually specifies axes like this, indicating that the first and second dimensions of the matrix participate in matrix multiplication, and the 0 dimension is batchsize
  6. tf.keras.layers.Dot(axes=(1, 2))([x, y])
  7. #The output is as follows:
  8. <tf.Tensor: shape=(1, 2, 2), dtype=int64, numpy=
  9. array([[[260, 360],
  10. [320, 445]]])>

Example 2:

  1. x1 = tf.keras.layers.Dense(8)(np.arange(10).reshape(5, 2))
  2. x2 = tf.keras.layers.Dense(8)(np.arange(10, 20).reshape(5, 2))
  3. dotted = tf.keras.layers.Dot(axes=1)([x1, x2])
  4. dotted.shape
  5. TensorShape([5, 1])

(3) Joint:

  1. #All input tensors are output tensors connected in series through the axis axis.
  2. layers.add(inputs,axis=-1)
  • inputs: the input tensor of a list (the list size is at least 2).
  • axis: axis in series.

example:

  1. x1 = tf.keras.layers.Dense(8)(np.arange(10).reshape(5, 2))
  2. x2 = tf.keras.layers.Dense(8)(np.arange(10, 20).reshape(5, 2))
  3. concatted = tf.keras.layers.Concatenate()([x1, x2])
  4. concatted.shape
  5. TensorShape([5, 16])

(4) Statistical operation

Average layers. Average ()

  1. input1 = tf.keras.layers.Input(shape=(16,))
  2. x1 = tf.keras.layers.Dense(8, activation=’relu’)(input1)
  3. input2 = tf.keras.layers.Input(shape=(32,))
  4. x2 = tf.keras.layers.Dense(8, activation=’relu’)(input2)
  5. avg = tf.keras.layers.Average()([x1, x2])
  6. # x_ 1 x_ 2 as the output
  7. print(avg)
  8. # <tf.Tensor ‘average/Identity:0’ shape=(None, 8) dtype=float32>
  9.  
  10. out = tf.keras.layers.Dense(4)(avg)
  11. model = tf.keras.models.Model(inputs=[input1, input2], outputs=out)

The usage of layers. Maximum() is the same.

Model with multiple inputs and outputs

Suppose you want to construct such a model:

(1) The model has the following three inputs

Work order title (text input), work order text body (text input), and any labels added by the user (classification input)

(2) The model will have two outputs:

  • Priority score between 0 and 1 (scalar sigmoid output)
  • The Department that should process the work order (softmax output within the Department).

The model looks like this:

Tensorflow 2.0 implements complex neural networks (multiple input multiple output NN, RESNET)

Next, start creating the model.

(1) Model input

  1. num_tags = 12
  2. num_words = 10000
  3. num_departments = 4
  4.  
  5. title_input = keras.Input(shape=(None,), name=”title”) # Variable-length sequence of ints
  6. body_input = keras.Input(shape=(None,), name=”body”) # Variable-length sequence of ints
  7. tags_input = keras.Input(shape=(num_tags,), name=”tags”) # Binary vectors of size `num_tags`

(2) Embed each input word into a 64 dimensional vector

  1. title_features = layers.Embedding(num_words,64)(title_input)
  2. body_features = layers.Embedding(num_words,64)(body_input)

(3) The processing results are input into LSTM model to obtain 128 dimensional vector

  1. title_features = layers.LSTM(128)(title_features)
  2. body_features = layers.LSTM(32)(body_features)

(4) Concatenate combines all features

  1. x = layers.concatenate([title_features, body_features, tags_input])

(5) Model output

  1. #Output 1, regression problem
  2. priority_pred = layers.Dense(1,name=”priority”)(x)
  3.  
  4. #Output 2, classification problem
  5. department_pred = layers.Dense(num_departments,name=”department”)(x)

(6) Define model

  1. model = keras.Model(
  2. inputs=[title_input, body_input, tags_input],
  3. outputs=[priority_pred, department_pred],
  4. )

(7) Model compilation

When compiling this model, you can assign different losses to each output. Each loss can even be assigned a different weight to adjust its contribution to the total training loss.

  1. model.compile(
  2. optimizer=keras.optimizers.RMSprop(1e-3),
  3. loss={
  4. “priority”: keras.losses.BinaryCrossentropy(from_logits=True),
  5. “department”: keras.losses.CategoricalCrossentropy(from_logits=True),
  6. },
  7. loss_weights=[1.0, 0.2],
  8. )

(8) Model training

  1. # Dummy input data
  2. title_data = np.random.randint(num_words, size=(1280, 10))
  3. body_data = np.random.randint(num_words, size=(1280, 100))
  4. tags_data = np.random.randint(2, size=(1280, num_tags)).astype(“float32”)
  5.  
  6. # Dummy target data
  7. priority_targets = np.random.random(size=(1280, 1))
  8. dept_targets = np.random.randint(2, size=(1280, num_departments))
  9.  
  10. #Fit the data to the model in the form of a dictionary
  11. model.fit(
  12. {“title”: title_data, “body”: body_data, “tags”: tags_data},
  13. {“priority”: priority_targets, “department”: dept_targets},
  14. epochs=2,
  15. batch_size=32,
  16. )

RESNET model

The basic structure of the model is as follows:

  1. #Implement the first block
  2. _input = keras.Input(shape=(32,32,3))
  3. x = layers.Conv2D(32,3,activation=’relu’)(_input)
  4. x = layers.Conv2D(64,3,activation=’relu’)(x)
  5. block1_output = layers.MaxPooling2D(3)(x)
  6.  
  7. #Implement the second block
  8. x = layers.Conv2D(64,3,padding=’same’,activation=’relu’)(block1_output)
  9. x = layers.Conv2D(64,3,padding=’same’,activation=’relu’)(x)
  10. block2_output = layers.add([x,block1_output])
  11.  
  12. #Implement the third block
  13. x = layers.Conv2D(64, 3, activation=”relu”, padding=”same”)(block2_output)
  14. x = layers.Conv2D(64, 3, activation=”relu”, padding=”same”)(x)
  15. block_3_output = layers.add([x, block2_output])
  16.  
  17. #Enter the full connection layer
  18. x = layers.Conv2D(64,3,activation=’relu’)(block_3_output)
  19. x = layers.GlobalAveragePooling2D()(x)
  20. x = layers.Dense(256, activation=”relu”)(x)
  21. x = layers.Dropout(0.5)(x)
  22. outputs = layers.Dense(10)(x)

Tensorflow 2.0 implements complex neural networks (multiple input multiple output NN, RESNET)

Model definition and compilation:

  1. model = keras.Model(_input,outputs,name=’resnet’)
  2.  
  3. model.compile(
  4. optimizer=keras.optimizers.RMSprop(1e-3),
  5. loss=’sparse_categorical_crossentropy’,
  6. metrics=[“acc”],
  7. )

Model training

  1. (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
  2. #Normalization
  3. x_train = x_train.astype(“float32”) / 255
  4. x_test = x_test.astype(“float32”) / 255
  5. model.fit(tf.expand_dims(x_train,-1), y_train, batch_size=64, epochs=1, validation_split=0.2)

Note: when loss = = keras. Losses. Categoricalcrossentropy (from_logits = true), the label needs to be one-hot:

  1. y_train = keras.utils.to_categorical(y_train, 10)

This is the end of this article about the implementation of complex neural networks (multiple input multiple output NN, RESNET) in tensorflow 2.0. For more information about tensorflow 2.0 complex neural networks, please search the previous articles of developer or continue to browse the relevant articles below. I hope you will support developer in the future!