New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning #10

Open

nocotan opened this issue May 28, 2021 · 0 comments

Assignees

Labels

gradient quantization TernGrad

Member

nocotan commented May 28, 2021 •

edited

Loading

一言でいうと

分散学習時に勾配を{-1, 0, 1}で量子化するTernGradを提案．

論文リンク

https://proceedings.neurips.cc/paper/2017/file/89fcd07f20b6785b92134bd6c1d0fa42-Paper.pdf

著者/所属機関

Duke University

投稿日付(yyyy/MM/dd)

NeurIPS2017

概要

新規性・差分

分散学習時に勾配を{-1, 0, 1}で量子化するTernGradを提案
適当な仮定を置くことでTernGradの収束性を理論的に証明
導出されたバウンドから，収束性を向上させるlayer-wise ternarizingとgradient clippingを提案

手法

TernGradアルゴリズム

収束性の理論解析

一般的な勾配アルゴリズムでは以下の二つが仮定されている：

これに加えて，TernGradの理論解析のために以下の追加の仮定を置く．

結果

コメント

https://github.com/wenwei202/terngrad

nocotan added TernGrad gradient quantization labels

nocotan self-assigned this

nocotan changed the title ~~[WIP] TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning~~ TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment