Skip to content

[inference] speedup by 84% - async, layer caching #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zazer0 wants to merge 18 commits into
base: main
Choose a base branch
from

feat(CLAUDE): correct inference testing

7690555
Select commit
Loading
Failed to load commit list.
Draft

[inference] speedup by 84% - async, layer caching #103

feat(CLAUDE): correct inference testing
7690555
Select commit
Loading
Failed to load commit list.
Codecov / codecov/patch succeeded Jun 5, 2025 in 1s

34.14% of diff hit (target 7.98%)

View this Pull Request on Codecov

34.14% of diff hit (target 7.98%)

Annotations

Check warning on line 165 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L165 

Added line #L165 was not covered by tests

Check warning on line 177 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L177 

Added line #L177 was not covered by tests

Check warning on line 181 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L180-L181 

Added lines #L180 - L181 were not covered by tests

Check warning on line 184 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L183-L184 

Added lines #L183 - L184 were not covered by tests

Check warning on line 186 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L186 

Added line #L186 was not covered by tests

Check warning on line 188 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L188 

Added line #L188 was not covered by tests

Check warning on line 191 in apps/inference/neuronpedia_inference/endpoints/activation/all.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/all.py#L191 

Added line #L191 was not covered by tests

Check warning on line 157 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L157 

Added line #L157 was not covered by tests

Check warning on line 160 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L160 

Added line #L160 was not covered by tests

Check warning on line 163 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L163 

Added line #L163 was not covered by tests

Check warning on line 167 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L166-L167 

Added lines #L166 - L167 were not covered by tests

Check warning on line 170 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L169-L170 

Added lines #L169 - L170 were not covered by tests

Check warning on line 172 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L172 

Added line #L172 was not covered by tests

Check warning on line 270 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L270 

Added line #L270 was not covered by tests

Check warning on line 273 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L273 

Added line #L273 was not covered by tests

Check warning on line 276 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L276 

Added line #L276 was not covered by tests

Check warning on line 279 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L279 

Added line #L279 was not covered by tests

Check warning on line 281 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L281 

Added line #L281 was not covered by tests

Check warning on line 284 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L284 

Added line #L284 was not covered by tests

Check warning on line 286 in apps/inference/neuronpedia_inference/endpoints/activation/single.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/single.py#L286 

Added line #L286 was not covered by tests

Check warning on line 73 in apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py#L72-L73 

Added lines #L72 - L73 were not covered by tests

Check warning on line 76 in apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py#L76 

Added line #L76 was not covered by tests

Check warning on line 80 in apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py#L79-L80 

Added lines #L79 - L80 were not covered by tests

Check warning on line 83 in apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py#L82-L83 

Added lines #L82 - L83 were not covered by tests

Check warning on line 85 in apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

apps/inference/neuronpedia_inference/endpoints/activation/topk_by_token.py#L85 

Added line #L85 was not covered by tests
View more details on Codecov