docs: Add intuitive explanation of JD #494
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
| ================ | ||
|
|
||
| This guide briefly explains what Jacobian descent is and on what kind of problems it can be used. | ||
| For a more theoretical explanation, please read our article |
Contributor
There was a problem hiding this comment.
Maybe
Suggested change
| For a more theoretical explanation, please read our article | |
| For a detailed explanation, take a look at |
|
|
||
| **Introduction** | ||
|
|
||
| The goal of Jacobian descent is to train models with multiple conflicting losses. When you have |
Contributor
There was a problem hiding this comment.
Maybe?
Suggested change
| The goal of Jacobian descent is to train models with multiple conflicting losses. When you have | |
| The goal of Jacobian descent is to train models when considering multiple conflicting losses. When you have |
| losses and then computing the gradient, so doing that is equivalent to doing gradient descent. | ||
|
|
||
| If you have two gradients with a negative inner product and quite different norms, their sum will | ||
| have a negative inner product with the smallest gradient. So, given a sufficiently small learning |
Contributor
There was a problem hiding this comment.
pic or fake (Figure 1)
| are traditionally considered as single-objective can actually be seen as multi-objective. Here are a | ||
| few examples: | ||
|
|
||
| - We can consider separately the loss of each element in the mini-batch, instead of averaging them. |
Contributor
There was a problem hiding this comment.
I understand what you did there, I guess you want to avoid the awkwardness of SJD
Suggested change
| - We can consider separately the loss of each element in the mini-batch, instead of averaging them. | |
| - We can consider separately the loss of each element in the dataset, instead of averaging them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This also needs to improve the monitoring examples so that they're easier to understand for someone who is just getting started with torchjd, and so that they show:
TODO: