Question on Adjusting Hyper-parameters for Lightweight Models

[AaronSun2000]

Hello,

Thank you for your work!

I would like to ask whether, if the model is replaced with a more lightweight one such as GPT-4o-mini, the inner-epochs, outer-epochs, or other hyper-parameters would also need to be adjusted.

Really appreciate any insights from your experience. Thank you!

[ViktorAxelsen]

Thank you for your interest in our work.

I think there are two reasonable options here.

The first option is to directly use the skills we provide and evaluate them with other models such as GPT-4o-mini. For example, you can load our released weights in the eval_*.sh scripts and simply change --model to gpt-4o-mini. Since the skills evolved by MemSkill have a certain degree of generalization ability, this is a reasonable and much more cost-efficient way to test the framework on another model.

The second option is to retrain the system. In our view, retraining is especially more meaningful when moving to benchmarks or scenarios beyond the ones where skills have already been evolved, such as conversation, document, or ALFWorld settings. Of course, retraining on the same benchmarks with a different base model is also possible if that is what you want to study.

If you do retrain, some hyperparameters are worth adjusting. A good place to start is --chunk-size, which determines how the interaction history is segmented during training. In practice, using a larger chunk size during training is usually more efficient and can help the system converge faster. We think the default value of 2048 is a reasonable starting point.

For inner-epochs and outer-epochs, these two together control the frequency of skill evolution. In our experience, setting inner-epochs around 50–100 is often a sensible range, because after each evolution step the controller needs some time to adapt to the updated skill set. If this value is too small, evolution may happen before the controller has sufficiently adapted; if it is too large, training cost may increase unnecessarily.

For outer-epochs, it is generally fine to set a relatively larger value, since the training pipeline already uses early stopping by default. In other words, if performance does not improve for several outer epochs, training will stop automatically.

One more detail is that if you change inner-epochs, you should also adjust new-action-bias-steps accordingly. This parameter controls how long newly added skills are encouraged during training, so it is usually better to scale it proportionally with inner-epochs.

As for batch-size, larger is generally better if your budget and API rate limits allow it, although in practice this is often constrained by throughput and cost.

For more implementation details, please refer to src/config.py and the configuration section in the README.

Hope this helps, and thanks again for your interest in MemSkill.

[AaronSun2000]

Thank you for your detailed and thorough explanation.

Really appreciate your response~