-
Notifications
You must be signed in to change notification settings - Fork 619
Add CABS Merge Method #568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
All contributors have signed the CLA ✍️ ✅ |
I have read the CLA Document and I hereby sign the CLA |
Thanks for the PR! I'd love to have your method in mergekit. Two things:
|
This is absolutely an amazing merging method, however it seems to need more support with gradients. While technically compatible if you supply as many values to the array as there are blocks, it would be nice to have it round to the closest whole number so it doesn't throw an error. |
Implements CABS (Conflict-Aware and Balanced Sparsification) model merging technique from "CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging".
CABS aims to improve merged model quality by mitigating parameter interference through sequential conflict-aware pruning and applying n:m structural pruning to task vector components.
Key Features:
pruning_order
, masking out parameters already claimed by prior models in the sequence before subsequent pruning. This minimizes destructive overlap.n
largest magnitude weights out of everym
consecutive weights) to the conflict-masked task vector components.n
andm
values are configurable globally (default_n_val
,default_m_val
) and per-model (n_val
,m_val
).weight
(lambda) and added to the base model.cabs.py
implementingCABSMerge
andCABSTask
.examples/cabs.yml
.