Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a warning in the defaults.conf for BERTAdam, and remove it as a default from all scripts #1096

Open
jeswan opened this issue Sep 17, 2020 · 0 comments
Open

Add a warning in the defaults.conf for BERTAdam, and remove it as a default from all scripts #1096

jeswan opened this issue Sep 17, 2020 · 0 comments
Labels
high-priority Fix this before addressing any other major issue.

Comments

@jeswan
Copy link
Contributor

jeswan commented Sep 17, 2020

Issue by sleepinyourhat
Thursday Jun 11, 2020 at 21:52 GMT
Originally opened as nyu-mll/jiant#1096

BERTAdam considered harmful in fine-tuning:

https://arxiv.org/abs/2006.05987
https://arxiv.org/abs/2006.04884

In the spirit of Reasonable Defaults:

  • Add a warning in the defaults.conf where BERTAdam is mentioned
  • Remove it as a default from all example scripts
  • Make sure that our support for conventional Adam is working, and use it as the replacement

(Relevant to @claravania)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high-priority Fix this before addressing any other major issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jeswan