Clash Royale CLAN TAG#URR8PPP

Tensorflow, Keras: What is the best way to build time-varing models

For example, given the time varying model with initial value x0 and transition matrix A, where the size is (batch_size, m, 1) and (T, m, m) respectively.

x0

A

(batch_size, m, 1)

(T, m, m)

The data flow is like: x1 = A1 x0, x2 = A2 x1, x3 = A3 x2 etc. And we save every x into Y which has size (batch_size, m, T)

x1 = A1 x0

x2 = A2 x1

x3 = A3 x2

x

Y

(batch_size, m, T)

x0 --A1--> x1 --A2--> x2 --A3--> x3 --A4--> ... xT
| | | | |
| | | | |
| | | | |
↓ ↓ ↓ ↓ ↓
Y[:,0] Y[:,1] Y[:,2] Y[:,3] Y[:,T]


Currently, my code for such process is:

# Code just for simple demonstration
import tensorflow as tf
import numpy as np

A = tf.Variable(np.ones(T, m, m), dtype='float64') # Use identity matrix just for demo

x0 = tf.placeholder((None, m, 1), dtype='float64')
x = tf.matmul(A[0], x0) # For broadcasting matmul
Y = x

for i in range(1, T):
x = tf.matmul(A[i], x)
Y = tf.concat([Y, x], axis=-1)

g = tf.gradients(Y, A)
print(Y)
print(g)


In that code, I use tf.concat() to save the results from every time step. But actually, in my application:

tf.concat()

I found my style of code is not effective. When the depth (time step) goes deep, e.g. T=200, the forward pass and gradients derivation became awfully slow and the memory exploded (if I reduce A to (1, m, m), memory became fine).

T=200

A

(1, m, m)

So, my question is:

How will you build such models? Are there any more efficient ways to achieve that x(t+1) = A(t)x(t) like models? For example, what about initialize a fixed size Y, and use Y[:,:,i].assign(x) (not sure)?

x(t+1) = A(t)x(t)

Y

Y[:,:,i].assign(x)

Is that possible to merge those operations into one operation, and manually define the gradients? If yes, what gradients should we derive? (Y is the output, and A contains trainable weights)

Y

A

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.