[BUG] GRPO on GSM8K is stable for SGLang but unstable/collapses for vLLM

[adityasoni9998]

Checklist

• The error occurs when using our provided Docker image.
• I can consistently reproduce the bug across multiple trials or random seeds.
• If the error causes experiment abortion, I've verified that this error is the root
  cause, not a secondary error caused by peer workers.

Detailed Information

Hi team - thank you for your work on RL training infrastructure. I have been consistently facing RL instability when running this example using the default config - the only change I make when shifting from SGLang to vLLM is on this line where I replace sglang with vllm.

Interestingly, on both H100 and A6000 GPUs from NVIDIA, I find that SGLang works fine with the reward on validation split close to that given here but I find that the RL run collapses when running on H100 with vLLM. I used the FSDP trainer backend in my experiments. I tried setting the enforce_eager flag in vLLM to True but that didn't help either. Is there any specific set of flags/settings needed to make vLLM work with AReaL? My experiments were run using a relatively recent commit ae8c792fdb5e21f77b3b9bca9c435cb6d1ddf62b.

I have attached the reward curves and grad norm curves for reference (the grad norm seems to explode/be higher for vLLM but not for SGLang)

Describe the bug

See above

Expected behavior

Default config with this simple RL example should work regardless of choice of inference backend.

Full logs

N/A

To Reproduce

Setup AReaL with vLLM/SGLang and then run python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

Commit ID

ae8c792fdb5e21f77b3b9bca9c435cb6d1ddf62b

Environment

NVIDIA H100 and A6000 GPUs. Installation simply required uv sync --extra cuda with Python 3.12 for SGLang. For vLLM, we installed flash-attn from the pre-built wheel: https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.10-cp312-cp312-linux_x86_64.whl for torch 2.10.

Script

https://github.com/inclusionAI/AReaL/blob/main/examples/math/gsm8k_grpo.yaml

[garrett4wade]

Hi @adityasoni9998, thank you for the bug report!

We’re able to reproduce the issue on NVIDIA H200 using our Docker image, but so far we haven’t been able to identify the root cause.

As a temporary workaround, we are using SGLang as the default backend for model training jobs.

A few additional observations that may help with debugging:

• vLLM works as expected on Ascend NPUs
• vLLM shows a lower initial reward compared to other backends (as shown in your curve as well)
• Rollouts generated by vLLM occasionally contain random characters

One hypothesis is that this may be related to the Qwen2 model version being relatively old, potentially interacting poorly with newer versions of vLLM and leading to numerical instability. Or, we should use a different learning rate for vLLM. However, this is not yet confirmed and requires further investigation.

[adityasoni9998]

Hi @garrett4wade! Thank you for looking into this and reproducing this issue. I think the likely reason for this behaviour is training-inference mismatch which is related to your observation Rollouts generated by vLLM occasionally contain random characters (I have seen similar problems when the rewards start collapsing). One stark difference we notice is in ppo_actor/update/compute_logp/kl_div_taylor/avg for different choice of inference engine backend - the inference-trainer logprob mismatch is way worse for vLLM as compared to SGLang which maybe the root cause. There are some workarounds for this in this blog but I am not very sure how helpful this is because this problem will grow worse as you shift to multi-turn tool-use scenarios for long-horizon tasks.

[garrett4wade]

Hi @adityasoni9998, thanks for the quick update!

The methods discussed in the blog you referenced have indeed been helpful for MoE models in our experience. We also have a partial implementation in #1088 that may be relevant here.

Could you try tightening the RejectionSamplingConfig (e.g., using stricter thresholds, or different aggregation level) and see if that resolves the mismatch at the algorithmic level?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity within the last 14 days.

If this issue is still relevant, please add a comment to keep it open. Otherwise, it will be automatically closed in 16 days.

Thank you for your contribution!