Is there any guiding principle for choosing --env.scene.num-envs?

[GF-Huang]

As long as GPU memory allows, should we choose a larger --env.scene.num-envs whenever possible? Will an excessively large --env.scene.num-envs cause the overall training speed to decrease?

[MLuc24]

Not quite "bigger whenever it fits", because num_envs doesn't only buy throughput — it also changes your batch size, and the PPO config doesn't compensate for that on its own.

From tasks/velocity/config/g1/rl_cfg.py:

num_learning_epochs=5,
num_mini_batches=4,
...
num_steps_per_env=24,

The rollout per iteration is num_envs × num_steps_per_env, and it's split into a fixed count of minibatches, not a fixed size. So minibatch size is num_envs × 24 / 4, while gradient steps per iteration stay pinned at 5 × 4 = 20 no matter how many envs you run. Double num_envs and you collect twice the experience, then do the same twenty updates on it — more accurate gradients, but fewer updates per sample collected.

That's the mechanism behind the slowdown you're asking about. Steps/sec keeps climbing with more envs (up to the point the GPU saturates), but iterations needed to reach a given policy quality can climb too, so wall-clock to a working policy is a U-shape rather than monotonic.

If you do want to run many more envs, scale num_mini_batches alongside so minibatch size stays roughly where it was — that keeps gradient steps proportional to data. Worth noting the adaptive LR absorbs some of this already:

schedule="adaptive",
desired_kl=0.01,

Bigger batches give lower-variance gradients, so the schedule will tend to push the learning rate up to hold the target KL. It softens the effect but doesn't remove it, since the update count is still fixed.