Feature Request: DeepSeek V3.2-Exp support

[createthis]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support DeepSeek V3.2-Exp: https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp

Motivation

Because we can.

Possible Implementation

VLLM code is here for reference: vllm-project/vllm#25896

[lovedheart]

I am not an expert on LLM. I put flashmla_sparse.py to Qwen-Max and it shows

[jukofyork]

It's really just the sparse attention that's going to be a problem:

(Ignore the scale_inv as they are for the quantisation)

[jukofyork]

This is the important code in VLLM:

https://github.com/vllm-project/vllm/blob/88ef733ff5125b1b29e7aaae13063b2a109d5e8f/vllm/model_executor/models/deepseek_v2.py#L768

https://github.com/vllm-project/vllm/blob/88ef733ff5125b1b29e7aaae13063b2a109d5e8f/vllm/model_executor/models/deepseek_v2.py#L584

[jukofyork]

https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/DeepSeek_V3_2.pdf

So it looks like it will need an extra cache to store a 128-element k_s value for each token at each layer, but other than this the actual calculation is fairly straightforward (assuming GGML has the needed primitive operation for the sparse multiplication).

[jukofyork]

@slaren is ggml_mul_mat_id() likely to work for this:

• Much larger k (2048) and n (~128k) compared to MoE selection logic.
• It's going to be multiplying a lot of 1xm tensors (specifically lots of 512 or 576 element vectors) instead of nxm tensors compared to the MoE logic.

[jukofyork]

Alternatively, we could just mask out the non-top-k tokens with zeros and do a normal ggml_mul_mat, but this seems a bit crap and doesn't really get any of the gains in speed from the sparsity...

[slaren]

I think it is very likely that at least it will be faster than a masked matrix multiplication, but ultimately it will depend on how well optimized are the backends for this case. You should be able to test the performance by using test-backend-ops in perf mode, and adding the relevant test cases to make_test_cases_perf.

[createthis]

I've uploaded the bf16 here, if anyone needs it: https://huggingface.co/createthis/DeepSeek-V3.2-Exp-bf16/tree/main

[createthis]

(/data/conda-envs/llama_cpp_cu128) jesse@larry:~/llama.cpp$ python     convert_hf_to_gguf.py     --outtype f16     --split-max-size 50G     --outfile /data2/DeepSeek-V3.2-Exp-GGUF/bf16     /data2/DeepSeek-V3.2-Exp-bf16/
INFO:hf-to-gguf:Loading model: DeepSeek-V3.2-Exp-bf16
WARNING:hf-to-gguf:Failed to load model config from /data2/DeepSeek-V3.2-Exp-bf16: The checkpoint you are trying to load has model type `deepseek_v32` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: DeepseekV32ForCausalLM
ERROR:hf-to-gguf:Model DeepseekV32ForCausalLM is not supported
(/data/conda-envs/llama_cpp_cu128) jesse@larry:~/llama.cpp$

Looks like we're blocked on huggingface/transformers#41196 upstream.

It can be hacked around with:

pip install git+https://github.com/hnyls2002/transformers@2a8a2a9

Then we're back to just needing to write code, it seems, so I started a branch:

https://github.com/createthis/llama.cpp/pull/9/files

(/data/conda-envs/llama_cpp_cu128) jesse@larry:~/llama.cpp$ python     convert_hf_to_gguf.py     --outtype f16     --split-max-size 50G     --outfile /data2/DeepSeek-V3.2-Exp-GGUF/bf16     /data2/DeepSeek-V3.2-Exp-bf16/
INFO:hf-to-gguf:Loading model: DeepSeek-V3.2-Exp-bf16
INFO:hf-to-gguf:Model architecture: DeepseekV32ForCausalLM
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-000163.safetensors'
INFO:hf-to-gguf:token_embd.weight,            torch.bfloat16 --> F16, shape = {7168, 129280}
INFO:hf-to-gguf:blk.0.attn_norm.weight,       torch.float32 --> F32, shape = {7168}
INFO:hf-to-gguf:blk.0.ffn_down.weight,        torch.bfloat16 --> F16, shape = {18432, 7168}
INFO:hf-to-gguf:blk.0.ffn_gate.weight,        torch.bfloat16 --> F16, shape = {7168, 18432}
INFO:hf-to-gguf:blk.0.ffn_up.weight,          torch.bfloat16 --> F16, shape = {7168, 18432}
INFO:hf-to-gguf:blk.0.ffn_norm.weight,        torch.float32 --> F32, shape = {7168}
Traceback (most recent call last):
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 9366, in <module>
    main()
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 9360, in main
    model_instance.write()
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 429, in write
    self.prepare_tensors()
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 6687, in prepare_tensors
    super().prepare_tensors()
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 300, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 6684, in modify_tensors
    return [(self.map_tensor_name(name), data_torch)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/llama.cpp/convert_hf_to_gguf.py", line 259, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.layers.0.self_attn.indexer.k_norm.bias'
(/data/conda-envs/llama_cpp_cu128) jesse@larry:~/llama.cpp$

I'll work on this in my spare time, if I have time, and no one more experienced steps in.

[createthis]

Quick update ... I woke up early this morning and did my best to figure out the tensor mappings. I've updated my branch: https://github.com/createthis/llama.cpp/pull/9/files

Using this command the GGUF bf16 generates successfully:

python convert_hf_to_gguf.py \
  --outtype f16 \
  --split-max-size 50G \
  --outfile /data2/DeepSeek-V3.2-Exp-GGUF/bf16 \
  /data2/DeepSeek-V3.2-Exp-bf16/

Shard (30/30): 100%|█████████████████████████████████████████████████████████████████████████████| 7.52G/7.52G [00:16<00:00, 451Mbyte/s]
Writing: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.34T/1.34T [49:23<00:00, 454Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /data2/DeepSeek-V3.2-Exp-GGUF/bf16/

This was monkey work, but I'm not 100% sure I named everything correctly. Can someone double check before we move forward with the hard part? Thanks!

[mattepiu]

Checking https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp there's some cpp kernel code which might help more than python:
deepseek-ai/DeepGEMM@7c95b14
deepseek-ai/FlashMLA@c28eca9

[jukofyork]

Quick update ... I woke up early this morning and did my best to figure out the tensor mappings. I've updated my branch: https://github.com/createthis/llama.cpp/pull/9/files

Using this command the GGUF bf16 generates successfully:

python convert_hf_to_gguf.py
--outtype f16
--split-max-size 50G
--outfile /data2/DeepSeek-V3.2-Exp-GGUF/bf16
/data2/DeepSeek-V3.2-Exp-bf16/

Shard (30/30): 100%|█████████████████████████████████████████████████████████████████████████████| 7.52G/7.52G [00:16<00:00, 451Mbyte/s]
Writing: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.34T/1.34T [49:23<00:00, 454Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /data2/DeepSeek-V3.2-Exp-GGUF/bf16/

This was monkey work, but I'm not 100% sure I named everything correctly. Can someone double check before we move forward with the hard part? Thanks!

I'm downloading it now, but will be a couple of days to 4MB/s.

[createthis]

FYI: Aider polyglot benchmarks indicate the VLLM code is underperforming vs deepseek's own stack. See https://discord.com/channels/1131200896827654144/1422153923493498902/1423627025062690816 for more info. We should be careful when cargo culting VLLM's code and see if we can identify the bug.

[createthis]

With the last two commits I've raised the tensor count error from expected 1391, got 1086 to expected 1391, got 1330. Seems like progress. Still missing something though. I deleted the GGUF I previously posted because I had to regenerate it after the last commit, which makes me think I'll have to do that again at least one more time.

Also, kinda funny... the last two commits have been authored by DeepSeek-V3.1-Terminus running under Open Hands, because I don't really know what I'm looking for at this point, so I'm just giving it the information I have and letting it go to work. I'm impressed it was able to make progress. The last couple of times I tried this method it didn't work.