Complete MiniCPM-V 4.6 multimodal RL training and checkpoint support

[xsuler]

Motivation

Issue #137 tracks loading and executing the MiniCPM-V 4.6 vision encoder/projector path. A separate issue is needed for the complete reinforcement-learning lifecycle after those components are available in the AReno model adapter.

The current MiniCPM-V 4.6 adapter is text-trunk only: the vision tower and multimodal projector remain outside the AReno model. This prevents RL training from controlling media parameters, propagating gradients through them, synchronizing updated parameters to rollout workers, or saving a complete trained multimodal checkpoint.

Dependency

This issue depends on #137 for processor integration and vision tower/projector loading. It should not duplicate the basic image-to-text loading work tracked there.

Proposed feature

Complete MiniCPM-V 4.6 multimodal RL support after #137 lands:

• preserve processor-produced image features through agentic rollout, packing, training, and recomputation;
• execute the vision tower and projector during training with correct autograd behavior;
• expose MiniCPM-V tower and projector through the common freeze/unfreeze contract;
• support independent tower/projector learning-rate schedules;
• keep frozen modules in evaluation mode and avoid retaining unnecessary activation graphs;
• include unfrozen tower/projector parameters in policy synchronization between train and rollout roles;
• support at least one policy-gradient algorithm such as GRPO or GSPO with multimodal prompts;
• report clear errors for unsupported feature layouts or incomplete multimodal checkpoints.

Checkpoint requirements

RL output checkpoints must be usable as complete MiniCPM-V 4.6 checkpoints:

• save the text trunk, vision tower, resampler/projector, and required multimodal configuration;
• preserve source-compatible parameter names or provide an explicit deterministic mapping;
• save updated media parameters even when only one of tower or projector is unfrozen;
• include optimizer and LR-scheduler state for trainable text/tower/projector groups when resumable training state is requested;
• restore freeze state and optimizer grouping from the run configuration on resume;
• verify save/load round trips under the supported TP/DP layouts;
• ensure the saved checkpoint can be used by both areno train resume and areno serve image requests.

Acceptance criteria

• A one-step MiniCPM-V 4.6 multimodal RL smoke test completes with a valid reward and nonzero policy gradient.
• Frozen tower/projector parameters remain unchanged; each unfreeze option updates only the requested group.
• Rollout workers receive updated trainable media parameters after policy synchronization.
• Saving and reloading preserves text, tower, and projector outputs for a fixed image/text sample within dtype tolerance.
• Resumed RL training restores independent optimizer/LR state for all enabled groups.
• A checkpoint produced by the RL run serves a basic image-to-text request successfully.

Related work

• Parent multimodal tracker: Support multimodal image-to-text training and serving #132
• MiniCPM-V 4.6 loading and image-to-text support: Add MiniCPM-V 4.6 image-to-text support #137
• Multimodal tower/projector control and checkpoint round trips: Generalize multimodal tower/projector training controls and checkpoint round trips #489

[Dayuxiaoshui]

@xsuler I can take this