Clash Royale CLAN TAG#URR8PPP

In-place operations with PyTorch

I was wondering how to deal with in-place operations in PyTorch. As I remember using in-place operation with autograd has always been problematic.

And actually I’m surprised that this code below works, even though I haven’t tested it I believe this code would have raised an error in version 0.3.1.

0.3.1

Basically I want do is set a certain position of a tensor vector to a certain value in a like:

my_tensor[i] = 0


Working example code:

# test parameter a
a = torch.rand((2), requires_grad=True)
print('a ', a)
b = torch.rand(2)

# calculation
c = a + b

# performing in-place operation
c[0] = 0
print('c ', c)
s = torch.sum(c)
print('s ', s)

# calling backward()
s.backward()

# optimizer step
optim = torch.optim.Adam(params=[a], lr=0.5)
optim.step()

# changed parameter a
print('changed a', a)


Output:

a tensor([0.2441, 0.2589], requires_grad=True)
c tensor([0.0000, 1.1511], grad_fn=<CopySlices>)
s tensor(1.1511, grad_fn=<SumBackward0>)
changed a tensor([ 0.2441, -0.2411], requires_grad=True)


So obviously in version 0.4.1. this works just fine without warnings or errors.

0.4.1

Referring to this article in the documentation: autograd-mechanics

Supporting in-place operations in autograd is a hard matter, and we
discourage their use in most cases. Autograd’s aggressive buffer
freeing and reuse makes it very efficient and there are very few
occasions when in-place operations actually lower memory usage by any
significant amount. Unless you’re operating under heavy memory
pressure, you might never need to use them.

But even though it works, the use of in-place operations is discouraged in most cases.

So my questions are:

How much does the usage of in-place operations affect performance?

How do I get around using in-place operations in such cases where I want to set one element of a tensor to a certain value?

Thanks in advance!

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.