TransformerLens v3

[shayansadeghieh]

Problem

Brief description of problem being resolved. If there are multiple or subpoints, use bullet points.

Fix

Brief description of fix being applied. If there are multiple or subpoints, use bullet points.

Testing

Brief description of testing that is done or added. If there are multiple or subpoints, use bullet points.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 5.18%. Comparing base (374a059) to head (6cddecd).

Additional details and impacted files

@@           Coverage Diff            @@
##            main    #153      +/-   ##
========================================
- Coverage   7.93%   5.18%   -2.75%     
========================================
  Files        124     104      -20     
  Lines      17502   16035    -1467     
  Branches     382     169     -213     
========================================
- Hits        1388     832     -556     
+ Misses     16103   15203     -900     
+ Partials      11       0      -11     

Flag	Coverage Δ
autointerp	?
inference	?
webapp	5.18% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[shayansadeghieh]

@hijohnnylin I don't think we need this anymore with tlens >= 3, please correct me if I'm wrong. It is causing infinite loops with model_bridge.

Try running this in a notebook and you'll see the new version of tlens automatically has mlp_block.hook_in attribute so we shouldn't need to add it:

!pip install git+https://github.com/shayansadeghieh/[email protected]
from transformer_lens.model_bridge import TransformerBridge
from transformer_lens import HookedTransformer

print("\n=== NEW: TransformerBridge ===")
new_model = TransformerBridge.boot_transformers("gpt2-small")
new_model.enable_compatibility_mode(disable_warnings=True)

# Test if hook_in exists
mlp_block = new_model.blocks[0].mlp
print(f"MLP has hook_in: {hasattr(mlp_block, 'hook_in')}")
print(f"MLP has hook_out: {hasattr(mlp_block, 'hook_out')}")

# Try to access it
try:
    hook = mlp_block.hook_in
    print(f"hook_in type: {type(hook)}")
    print(f"hook_in name: {hook.name}")
except AttributeError as e:
    print(f"ERROR: {e}")

print("=== OLD: HookedTransformer ===")
old_model = HookedTransformer.from_pretrained("gpt2-small")

# Test if hook_in exists
mlp_block = old_model.blocks[0].mlp
print(f"MLP has hook_in: {hasattr(mlp_block, 'hook_in')}")
print(f"MLP has hook_out: {hasattr(mlp_block, 'hook_out')}")

# Try to access it
try:
    hook = mlp_block.hook_in
    print(f"hook_in type: {type(hook)}")
except AttributeError as e:
    print(f"ERROR: {e}")