Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #113 +/- ##
========================================
- Coverage 7.98% 7.93% -0.05%
========================================
Files 121 121
Lines 17040 17104 +64
Branches 362 361 -1
========================================
- Hits 1360 1358 -2
- Misses 15669 15735 +66
Partials 11 11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
anthonyduong9
changed the title
SrGonao
left a comment
SrGonao
left a comment
There was a problem hiding this comment.
I think I caught everything that I found strange
| example = Example( | ||
| tokens=activation.tokens, # type: ignore | ||
| activations=torch.tensor(activation.values), | ||
| str_tokens=activation.tokens, |
There was a problem hiding this comment.
This does not seem correct. We are expecting str_tokens to be the decoded tokens into strings. Do you have access to those in your request?
There was a problem hiding this comment.
activations.tokens is a list of strings. I think what's confusing is that we're able to pass activations.tokens to Example.tokens despite https://github.com/EleutherAI/delphi/blob/db49cb78120c1926a4a3c4928c76ece6be64dcb3/delphi/latents/latents.py#L66-L73. But https://github.com/EleutherAI/delphi/blob/db49cb78120c1926a4a3c4928c76ece6be64dcb3/delphi/scorers/embedding/embedding.py#L111-L121 shows that Example.tokens can either be a list of integers or strings.
Do we want to keep both Example.tokens and Example.str_tokens in delphi but make changes so tokens can only be a list of strings?
If so, I can update the requests in this repo, in a separate PR, and make the changes to delphi as well.
There was a problem hiding this comment.
Yea this is definitely a mistake on our part that we let pass. Example.tokens should definitely only be a list of integers and Example.str_tokens the corresponding strings. In this case, because you don't care about the integers you can just pass a dummy list?
| example = ActivatingExample( | ||
| tokens=activation.tokens, # type: ignore | ||
| activations=torch.tensor(activation.values), | ||
| str_tokens=activation.tokens, |
|
@hijohnnylin this PR was ready to merge before EleutherAI/delphi#133 was merged. But this PR isn't ready because the latter PR contains a bunch of commits. Feel free to close or edit this. |
2 participants
Problem
We depend on a fork at EleutherAI/delphi@8a6307e.
Fix
eai-delphi 0.0.2, infuzz-detection.yaml.schemas, runningmake setup-all-autointerp VERSION=2.0.0.apps/autointerp/neuronpedia_autointerpto matcheai-delphi 0.0.2.apps/autointerp/pyproject.toml.Testing
Brief description of testing that is done or added. If there are multiple or subpoints, use bullet points.
apps/autointerp, I ranpoetry run uvicorn server:app --host 0.0.0.0 --port 5003 --workers 1, and made requests to POST /score/fuzz-detection, POST /score/embedding, and POST /explain/default, before and after the changes.Fixes #27