So how do we deal with the data reasonably?
We know that the state in Q (s, a) represents the state of the snake. This state includes the position of the apple, the position of the snake, the position of the boundary, the distance between the snake and the boundary, and so on. How to express so much content and accurately transfer it into neural network for calculation?
We use a very important data structure – “tensor“.
This time, our theme is: “what is tensor and how does it flow?”
What is tensor
First of all, we declare that tensor is the most basic data structure in tensorflow. It is the most important concept of tensorflow. As the name suggests, flow means flow, and tensorflow is the flow of tensor It is not the same concept as tensor in physics.
What is tensor? In short, tensor is the universal concept of multidimensional array. Usually one-dimensional array is called vector, two-dimensional array is called matrix, these are tensor. We also have three-dimensional tensors, four-dimensional tensors, five dimensional tensors and so on. A zero dimensional tensor is a concrete number.
The basic concept of tensor
The figure below shows the epidemic situation of some cities in China, which is a three-dimensional tensor.
From the above figure, we can analyze the basic concept of tensor
Dimension is the number of data axes. As shown in the previous figure, the data has three axes, namely city, classification and time. By using the knowledge of life science, the genus and species of phylum can represent the seven dimensions of biological classification.
Shape represents the size (number of elements) of the tensor along each axis, that is, shape. The shape of the previous graph matrix example is (3, 5), and the shape of the 3D tensor example is (3, 5, 3).
“Data type” is the type of data contained in a tensor. For example, the type of tensor can be float32, int32, float64, etc. In rare cases, char tensors are encountered
The figure below is a three-dimensional graph of tensor
Tensor can be increased or decreased according to the actual situation. For example, I can easily add a statistical City, statistical classification and statistical time. With the development of the epidemic situation, we add a dimension to count the epidemic situation of each country.
At this point, some people may regard n-dimensional tensors as a tree graph, and each element of the dimension will have the following branches and further branches. To put it bluntly, it’s a special kind of tree view. Due to the characteristics of shape, the number of N-1-dimensional elements contained in n-dimensional elements is equal. For example, if the shape is a tensor of (2,2,3) and there are two elements in two dimensions, then the number of elements in one dimension is equal. This is different from that each branch of the tree can be expanded irregularly.
Tensor may have some defects, but it is still one of the best carriers to deal with data, especially in the game production. In the presence of multivariate tensor, it is easier to use library function for various operations.
“The form of tensor”
In mathematics, there are n-dimensional vectors. In fact, they are all one-dimensional tensors. The n-dimensional vector in mathematics refers to the number of components. For example, the dimension of [1,2] is 2, and it has two components: 1 and 2[ 1,2,3,…, 1000] the dimension of this vector is 1000. In the concept of tensors, they are all one-dimensional tensors.
What about the dimension and shape of tensor?
The shape of tensor itself is a tuple. The number of tuple elements represents the number of dimensions. Starting from tuple , it represents the number of elements in each dimension (from high dimension to low dimension). For example, (2,3) is a two-dimensional tensor with three elements in one dimension and two elements in two dimensions.
The advantage of tensor in tensorflow
In the neural network constructed by tensorflow, the values of input layer and output layer are tensor. Because tensor model can deal with the multi-dimensional data of index set (element item), it is closer to the attributes of the actual problem than matrix model, so it can better describe the actual problem, so as to ensure that the neural network algorithm is effective
At the same time, tensorflow has the function of dimension reduction, such as www.cungun.comDQN The input is a multi-dimensional tensor describing the environment, which contains many complex decimals. After processing, the output is a number representing four optional actions.
Tensor realizes data flow through numpy
Numpy is the basic package of scientific computing in Python. It provides multi-dimensional array objects, various derived objects (such as mask array and matrix), and various routines for array fast operation, including mathematics, games, logic, shape operation, sorting, selection, I / O discrete Fourier transform, basic linear algebra, basic statistical operation, random simulation, etc.
The biggest problem of a tenor is that it can’t calculate data easily. For example, to operate a tenor, we need to start a session first. Otherwise, we can’t simply assign or judge a tenor. This restriction is fatal to the program. Data flow requires a lot of complex operations. Therefore, with the help of numpy’s powerful computing power and the convenient data conversion between tensor and numpy, they perfectly realize the complex computing work of neural network.
The general operation process is: tensorflow defines all the calculation processes, that is, the calculation flow graph, establishes the neural network, and creates the input tensor. At this time, it represents a defined calculation process, and does not really carry out the calculation; Next, tensor is automatically converted to numpy to perform operations through explicit or implicit conversion. This is the time to give full play to numpy’s computing power.
“The use of tensors in the greedy snake program”
The construction of neural network mentioned in the last part is one of the applications of tensor. Next, let me briefly introduce some other applications.
Create and call initialization tensor
self.STATE = self.SNAKE.look(self.FOOD_X, self.FOOD_Y, boundaries)）
The position coordinates of food are transformed into numpy storage
NP. Hstack is used to stack the elements horizontally. These elements are mainly affected by the snake head direction and the refresh position of the fruit, that is, the top, bottom, left and right of the snake head direction.
“Memory of experience”
states, actions, rewards, next_states, dones = zip(*experiences)
\#Convert information into numpy format
states = np.array(states).reshape(self.BATCH_SIZE, state_shape)
actions = np.array(actions, dtype=’int’).reshape(self.BATCH_SIZE)
rewards = np.array(rewards).reshape(self.BATCH_SIZE)
next_states = np.array(next_states).reshape(self.BATCH_SIZE, state_shape)
dones = np.array(dones).reshape(self.BATCH_SIZE)
return states, actions, rewards, next_states,
We first use zip * to split the tensor (the variable here is the core of the snake principle), and then convert it into numpy form to take it out, which is convenient for subsequent data processing with numpy.
Dimension reduction computing
As can be seen from the above figure, there are often 18 items of stored data. Finally, through tensorflow and numpy data processing, we turn 18 float8 data into action, that is, 0, 1, 2 and 3 are used to represent the direction. From the above figure, since the training batch is 64, there are 64 numbers in the action, which is the 64 best actions of the snake calculated at that time.