[REQUEST] Add support for EXAONE 4.0 models

[lesj0610]

Hi DeepSpeed team,

I’d like to suggest adding support for EXAONE 4.0 (LGAI‑EXAONE/EXAONE‑4.0‑32B and 1.2B) in DeepSpeed.
Currently, this model is officially supported by vLLM (via PR #21060), but DeepSpeed doesn’t yet recognize or integrate it.
This means we can’t fully take advantage of ZeRO, offload, and tensor parallel features with EXAONE 4.0.

EXAONE 4.0 is a hybrid LLM (combining reasoning and non‑reasoning modes) and provides strong performance on reasoning tasks and multilingual content (Korean, English, Spanish). Benchmarks like MMLU‑Redux and MATH show impressive numbers (MMLU‑Redux 92.3, MATH/AIME ~85%).

What we’d like to see:

1. Register Exaone4ForCausalLM in DeepSpeed’s supported model registry.
2. Add support for initialization, parameter loading, and optimizer state management for the EXAONE 4.0 architecture.
3. Ensure compatibility with ZeRO-offload (CPU/NVMe) and tensor parallel inference.

I’d be happy to share test logs or work on a draft PR if needed.
Adding EXAONE 4.0 support would be a great benefit for the growing Korean and multilingual AI community.

Thanks for your time and consideration!

[PKUWZP]

Thanks for the proposal and I like the idea of adding EXAONE 4.0 support for DeepSpeed. We would love to collaborate on this. @tjruwase will also provide some guidance on how we should proceed. One quick question: can you share any architectural difference between EXAONE 4.0 to other OSS models such as Gemma, Llama3 and Qwen3 etc.?

[notkisk]

I can point out a couple of important architecture differences that matter for DeepSpeed. First, it uses a hybrid attention mechanism mixing local (sliding window) and global (full) attention in a 3:1 ratio and it skips RoPE in global layers. This is different from standard transformers like LLaMA or Qwen, which usually have a uniform attention setup. Second, it introduces QK-Reorder-Norm, which applies LayerNorm directly to the attention and MLP outputs (instead of the usual Pre-LN style), and adds RMSNorm after the Q and K projections. These changes could affect DeepSpeed ZeRO Stage 3, especially around parameter partitioning, memory usage, and gradient flow. To be safe, I’d recommend testing with ZeRO-1 or ZeRO-2 first since they’re less sensitive to architecture. Watch memory patterns closely and make sure gradient computation is stable. On the bright side, EXAONE 4.0 works with Hugging Face Transformers (v4.54.0+), and DeepSpeed supports standard PyTorch modules, so integration should mostly work out of the box with maybe a few tweaks to handle those unique layers

[notkisk]

See PR #7456 for the feature.