feat: add WMDP dataset integration #93
Conversation
Add WMDP cyber dataset configuration for unlearning
|
|
||
| We invite the LLM unlearning community to collaborate by adding new benchmarks, unlearning methods, datasets and evaluation metrics here to expand OpenUnlearning's features, gain feedback from wider usage and drive progress in the field. | ||
|
|
||
| --- | ||
|
|
||
| ### 📢 Updates | ||
|
|
||
| #### [Apr 9, 2025] |
There was a problem hiding this comment.
need to update just before merge
There was a problem hiding this comment.
It's the same as WMDP_cyber_forget.yaml
Is it needed for something? If for testing performance disruption, IIRC MMLU was used.
There was a problem hiding this comment.
@ruidazeng I don't see the path existing for this dataset in hf.
cais/wmdp-corpora
There was a problem hiding this comment.
You mean for MMLU? Yeah, it's not in cais/wmdp-corpora but in https://huggingface.co/datasets/cais/mmlu
From what I understand, in original setup, training is done on:
- "cais/wmdp-corpora" - "cyber-forget-corpus"
- "cais/wmdp-corpora" - "cyber-retain-corpus"
And evaluation on:
- "cais/wmdp" - "wmdp-cyber"
- "cais/mmlu" - "all"
There was a problem hiding this comment.
I see you mentioned in #80 that evaluation won't be done here anyway, but only afterwards, so I guess WMDP_cyber_forget.yaml and WMDP_cyber_retain.yaml just aren't needed.
Add WMDP cyber dataset configuration for unlearning
What does this PR do?
Fixes #80
Before submitting