Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement torch_tensor_backward #286

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Conversation

jwallwork23
Copy link
Collaborator

@jwallwork23 jwallwork23 commented Feb 18, 2025

Closes #152.

This PR:

  • Sets the requires_grad argument to tensor constructors up properly. (Done in Hook up requires_grad #288.)
  • Wraps the tensor->backward method as torch_tensor_backward.
  • Provides a method for retrieving the gradient of a tensor (with respect to a tensor on which the user has called torch_tensor_backward): torch_tensor_get_gradient.
    • For similarity with PyTorch, I also provided a method torch_tensor%grad() (c.f. tensor.grad).
  • Provides unit tests for the gradients of the overloaded operators.
  • Updates the autograd example accordingly.
  • Revises the scalar multiplication/division approach as described below. (Done in Rework scalar multiplication and division #289.)

I spent far too much time trying to get the gradients to be computed correctly for scalar multiplication and division as it is currently implemented. However, I discovered that everything works nicely if we just define scalars as rank-1 tensors with a single value. The revised approach to scalar multiplication/division adds a bit of boilerplate, but perhaps this could be reduced by introducing a torch_scalar class (see #285 for discussion).

@jwallwork23 jwallwork23 added enhancement New feature or request autograd Tasks towards the online training / automatic differentiation feature labels Feb 18, 2025
@jwallwork23 jwallwork23 self-assigned this Feb 18, 2025
@jwallwork23 jwallwork23 changed the title 152 tensor backward Implement torch_tensor_backward Feb 18, 2025
@jwallwork23 jwallwork23 mentioned this pull request Feb 18, 2025
@jwallwork23
Copy link
Collaborator Author

I'll break this PR down into more manageable ones as it's got rather unwieldy.

@jwallwork23 jwallwork23 changed the base branch from main to rework-scalar-ops March 10, 2025 16:06
Base automatically changed from rework-scalar-ops to main March 12, 2025 16:00
@jwallwork23 jwallwork23 marked this pull request as ready for review March 12, 2025 16:06
@jwallwork23
Copy link
Collaborator Author

Ah, just noticed that I haven't updated pages/autograd.md for this PR yet

### The `backward` operator

*Not yet implemented.*

Will do that.

@jwallwork23
Copy link
Collaborator Author

Ah, just noticed that I haven't updated pages/autograd.md for this PR yet

### The `backward` operator

*Not yet implemented.*

Will do that.

Done in 883654a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autograd Tasks towards the online training / automatic differentiation feature enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement torch_tensor_backward
1 participant