RuntimeError: Trying to backward through the graph a second time when using dataloader

[vence-andersen]

I am working on Chapter 3 excersive, I wrote a model

class TinyVGG(nn.Module):
    def __init__(self,input_features,
                 output_features,
                 hidden_units,
                 len_classes) -> None:
        super().__init__()

        self.conv_relu_layer_1 = nn.Sequential(
            nn.Conv2d(in_channels=input_features,
                      out_channels=output_features,
                      kernel_size=3,
                      padding=1,
                      stride=1),
            nn.ReLU()
        )
        self.conv_relu_maxpool_1 = nn.Sequential(
            nn.Conv2d(in_channels=output_features,
                      out_channels=output_features,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3)
        )
        self.conv_relu_layer_2 = nn.Sequential(
            nn.Conv2d(in_channels=output_features,
                      out_channels=output_features,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU()
        )
        self.conv_relu_maxpool_2 = nn.Sequential(
            nn.Conv2d(in_channels=output_features,
                      out_channels=output_features,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3)
        )
        self.classfier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(in_features=hidden_units*3*3,
                      out_features=len_classes)
        )
    def forward(self, x):
        x = self.conv_relu_layer_1(x)
        x = self.conv_relu_maxpool_1(x)
        x = self.conv_relu_layer_2(x)
        x = self.conv_relu_maxpool_2(x)
        x = self.classfier(x)

        return x

To train it

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(params=tinyvgg_GPU.parameters(),
                             lr=0.01)

epochs = 5

for epoch in range(epochs):
    accuracy=0; loss=0
    for data, target in train_dataloader:
        tinyvgg_GPU.train()
        
        data = data.to(device)
        target = target.to(device)
    
        y_logits = tinyvgg_GPU(data)
        
        loss += loss_fn(y_logits, target)

        optimizer.zero_grad()

        loss.backward()

        optimizer.step()
    
    loss = loss/len(train_dataloader)

    tinyvgg_GPU.eval()
    with torch.inference_mode():
        test_loss = 0; test_accuracy = 0
        for data, target in test_dataloader:

            data = data.to(device)
            target = target.to(device)

            test_logits = tinyvgg_GPU(data)

            test_loss += loss_fn(test_logits, target)

        test_loss = test_loss/len(test_dataloader)

    print(f"Epoch: {epoch} | loss: {loss} | test_loss : {test_loss}")

This throwed me the error message:
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

Then I ran the model in the solutions notebook, which ran without any errors. Can someone please help me understand what's the problem in my code to get such an error? Thanks in advance.

[vence-andersen]

So, I fixed it, well now I am even more confused.

I changed the loss computation from

loss += loss_fn(y_logits, target)

to

loss = loss_fn(y_logits, target)
train_loss += loss

Can I get a clarification of how this solved the problem as I see no difference. Because in my code I am directly adding the loss and, in the solution, the loss is stored in a variable before being added. How could this be the problem?

[mrdbourke]

Hi @vence-andersen ,

You've found a great gotcha in PyTorch!

While I'm not sure I fully 100% understand it the in most correct way, here's how I think of it.

The key is in the error message:

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

So what happens here is when you calculate the loss with:

loss += loss_fn(y_logits, target)

And then run:

loss.backward()

PyTorch behind the scenes only remembers going through the loss variable once per loop.

So when you try to recompute it with += it goes, wait, I've already seen the previous loss, why am I seeing it again?

Essentially calling loss.backward(), PyTorch expects it to be a new loss value.

This is why your first solution errors but your second solution passes (even though they seem similar in regular Python).

For a more technical explanation of this and how it works, I'd refer to the following:

• http://blog.ezyang.com/2019/05/pytorch-internals/
• https://pytorch.org/tutorials/beginner/introyt/autogradyt_tutorial.html