Tensorflow variable scope

Time:2021-3-23

Give an example

The variables in tensorflow are generally the parameters of the model. When the model is complex, shared variables are extremely complex.

The official website gives a case. When creating a two-layer convolution filter, the corresponding variable of the filter will be created for each image input. However, we hope that all images share the same filter variable. There are four variables in total: conv1_ weights,conv1_ biases,conv2_ weights, and conv2_ biases。

The common practice is to set these variables as global variables. However, the problem is to break the encapsulation. These variables must be documented and referenced by other code files. Once the code changes, the caller may also need to change.

Another way to ensure encapsulation is to encapsulate the model into classes.

However, tensorflow provides a unique mechanism of variable scope to share variables. There are two main mechanisms involved in this function

tf.get_ Variable (< name >, < Shape >, < initializer >) creates or returns a variable with a given name
tf.variable_ scope(<scope_ Name >) to get_ The scope of the variable name of variable()

In the following code, through the tf.get_ Variable () creates two variables named weights and biases.


def conv_relu(input, kernel_shape, bias_shape):
  # Create variable named "weights".
  weights = tf.get_variable("weights", kernel_shape,
    initializer=tf.random_normal_initializer())
  # Create variable named "biases".
  biases = tf.get_variable("biases", bias_shape,
    initializer=tf.constant_initializer(0.0))
  conv = tf.nn.conv2d(input, weights,
    strides=[1, 1, 1, 1], padding='SAME')
  return tf.nn.relu(conv + biases)

But we need two convolution layers, which can be passed at this time tf.variable_ Scope () specifies the scope for differentiation, such as with tf.variable_ Scope (“conv1”) specifies that the scope of the first volume layer is conv1,

Under this scope, there are two variables weights and bias.


def my_image_filter(input_images):
  with tf.variable_scope("conv1"):
    # Variables created here will be named "conv1/weights", "conv1/biases".
    relu1 = conv_relu(input_images, [5, 5, 32, 32], [32])
  with tf.variable_scope("conv2"):
    # Variables created here will be named "conv2/weights", "conv2/biases".
    return conv_relu(relu1, [5, 5, 32, 32], [32])

Finally, in image_ Filters this scope repeatedly uses the variable created when the first image is input, and calls the function reuse_ Variables (), the code is as follows:


with tf.variable_scope("image_filters") as scope:
  result1 = my_image_filter(image1)
  scope.reuse_variables()
  result2 = my_image_filter(image2)

tf.get_ Variable () working mechanism

tf.get_ The working mechanism of variable() is as follows:

When tf.get_ variable_ Scope (). Reuse = = false, calling this function will create a new variable


with tf.variable_scope("foo"):
  v = tf.get_variable("v", [1])
assert v.name == "foo/v:0"

When tf.get_ variable_ Scope (). Reuse = = true, calling this function will reuse the created variables


with tf.variable_scope("foo"):
  v = tf.get_variable("v", [1])
with tf.variable_scope("foo", reuse=True):
  v1 = tf.get_variable("v", [1])
assert v1 is v

Variables are identified by scope / variable name. Later, you will see that scope can be nested like file path.

tf.variable_ Scope understanding

tf.variable_ Scope() is used to specify the scope of a variable as the prefix of the variable name. Nesting is supported, as follows:


with tf.variable_scope("foo"):
  with tf.variable_scope("bar"):
    v = tf.get_variable("v", [1])
assert v.name == "foo/bar/v:0"

The scope of the current environment can be defined by the function tf.get_ variable_ Scope (), and the reuse flag can be obtained by calling reuse_ Variables () is set to true, which is very useful, as follows


with tf.variable_scope("foo"):
  v = tf.get_variable("v", [1])
  tf.get_variable_scope().reuse_variables()
  v1 = tf.get_variable("v", [1])
assert v1 is v

The default value of reuse in the scope is false, and the function reuse is called_ Variables() can be set to true. Once it is set to true, it cannot return to false, and the subspace reuse of the scope is true. If you don’t want to reuse variables, you can go back to the upper scope, which is equivalent to the exit current scope, such as


with tf.variable_scope("root"):
  # At start, the scope is not reusing.
  assert tf.get_variable_scope().reuse == False
  with tf.variable_scope("foo"):
    # Opened a sub-scope, still not reusing.
    assert tf.get_variable_scope().reuse == False
  with tf.variable_scope("foo", reuse=True):
    # Explicitly opened a reusing scope.
    assert tf.get_variable_scope().reuse == True
    with tf.variable_scope("bar"):
      # Now sub-scope inherits the reuse flag.
      assert tf.get_variable_scope().reuse == True
  # Exited the reusing scope, back to a non-reusing one.
  assert tf.get_variable_scope().reuse == False

A scope can be used as a parameter of another new scope, such as:


with tf.variable_scope("foo") as foo_scope:
  v = tf.get_variable("v", [1])
with tf.variable_scope(foo_scope):
  w = tf.get_variable("w", [1])
with tf.variable_scope(foo_scope, reuse=True):
  v1 = tf.get_variable("v", [1])
  w1 = tf.get_variable("w", [1])
assert v1 is v
assert w1 is w

No matter how nested the scopes are, when using with tf.variable_ When scope () opens an existing scope, it will jump to the scope.


with tf.variable_scope("foo") as foo_scope:
  assert foo_scope.name == "foo"
with tf.variable_scope("bar"):
  with tf.variable_scope("baz") as other_scope:
    assert other_scope.name == "bar/baz"
    with tf.variable_scope(foo_scope) as foo_scope2:
      assert foo_scope2.name == "foo" # Not changed.

Initializers of variable scope can create subspaces and tf.get_ The variable() function does not change unless there is a function change in the middle.


with tf.variable_scope("foo", initializer=tf.constant_initializer(0.4)):
  v = tf.get_variable("v", [1])
  assert v.eval() == 0.4 # Default initializer as set above.
  w = tf.get_variable("w", [1], initializer=tf.constant_initializer(0.3)):
  assert w.eval() == 0.3 # Specific initializer overrides the default.
  with tf.variable_scope("bar"):
    v = tf.get_variable("v", [1])
    assert v.eval() == 0.4 # Inherited default initializer.
  with tf.variable_scope("baz", initializer=tf.constant_initializer(0.2)):
    v = tf.get_variable("v", [1])
    assert v.eval() == 0.2 # Changed default initializer.

Operators (OPS) are affected by variable scope, which is equivalent to implicitly opening the name scope of the same name. For example, the name of + is foo / add


with tf.variable_scope("foo"):
  x = 1.0 + tf.get_variable("v", [1])
assert x.op.name == "foo/add"

In addition to the variable scope, you can also explicitly open the name scope. The name scope only affects the name of the operator, not the name of the variable. In addition, if tf.variable_ Scope() passes in a character parameter. When creating a variable scope, a name scope with the same name will be created implicitly. As shown in the following example, the scope of variable V is foo, while the operator of operator x becomes foo / bar, because there is an implicit name scope foo


with tf.variable_scope("foo"):
  with tf.name_scope("bar"):
    v = tf.get_variable("v", [1])
    x = 1.0 + v
assert v.name == "foo/v:0"
assert x.op.name == "foo/bar/add"

Note: if tf.variable_ If scope () passes in a scope object instead of a string, a name scope with the same name will not be implicitly created.

The above is the whole content of this article, I hope to help you learn, and I hope you can support developer more.

Recommended Today

Swift advanced 08: closure & capture principle

closure closurecanCapture and storageOf any constants and variables defined in their contextquote, this is the so-calledClose and wrap those constants and variablesTherefore, it is called“closure”Swift can handle everything for youCaptured memory managementOperation of. Three forms of closure [global function is a special closure]: a global function is a closure that has a name but does […]