Skip to content

[inference] speedup by 84% - async, layer caching #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zazer0 wants to merge 18 commits into
base: main
Choose a base branch
from

feat(CLAUDE): correct inference testing

7690555
Select commit
Loading
Failed to load commit list.
Draft

[inference] speedup by 84% - async, layer caching #103

feat(CLAUDE): correct inference testing
7690555
Select commit
Loading
Failed to load commit list.
Codecov / codecov/project succeeded Jun 5, 2025 in 0s

8.79% (+0.81%) compared to 3063b04

View this Pull Request on Codecov

8.79% (+0.81%) compared to 3063b04

Details

Codecov Report

Attention: Patch coverage is 34.14634% with 270 lines in your changes missing coverage. Please review.

Project coverage is 8.79%. Comparing base (3063b04) to head (7690555).
Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
...pedia_inference/endpoints/steer/completion_chat.py 28.79% 131 Missing and 5 partials ⚠️
...euronpedia_inference/endpoints/steer/completion.py 3.61% 80 Missing ⚠️
...uronpedia_inference/endpoints/activation/single.py 5.55% 17 Missing ⚠️
.../neuronpedia_inference/endpoints/activation/all.py 8.33% 11 Missing ⚠️
...ia_inference/endpoints/activation/topk_by_token.py 10.00% 9 Missing ⚠️
...ce/neuronpedia_inference/layer_activation_cache.py 89.77% 7 Missing and 2 partials ⚠️
apps/inference/neuronpedia_inference/server.py 0.00% 8 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff            @@
##            main    #103      +/-   ##
========================================
+ Coverage   7.98%   8.79%   +0.81%     
========================================
  Files        121     122       +1     
  Lines      17040   17354     +314     
  Branches     362     422      +60     
========================================
+ Hits        1360    1527     +167     
- Misses     15669   15804     +135     
- Partials      11      23      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
View more details on Codecov