Correction: `memory_ops` for `encoder_forward` #797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

MasterSkepticista wants to merge 1 commit into karpathy:master from MasterSkepticista:enc_fwd_mem

MasterSkepticista commented Feb 26, 2025

Two changes:

sizeof(floatX), otherwise bandwidth calculation is off by 2x when using bf16
3C memory ops instead of 4C: For each input token we have 2C reads and C writes.

On RTX-3090 (936 GB/s):

# Before
...
block_size 1024 | time 0.0483 ms | bandwidth 2082.91 GB/s

# After
...
block_size 1024 | time 0.0483 ms | bandwidth 782.40 GB/s


          Fix napkin math

af84752

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet