Skip to content

Conversation

blogbin
Copy link

@blogbin blogbin commented May 27, 2025

Fix: Improve ParallelExt Compatibility with joblib Backend

Description

This PR fixes the handling of maxtasksperchild in ParallelExt, ensuring better compatibility when using the joblib backend. The update ensures that task management behaves consistently under different multiprocessing settings.

Motivation and Context

Previously, maxtasksperchild was either not respected or caused unexpected behavior under the joblib backend. This change ensures that task recycling and memory/resource constraints are correctly handled when running long or parallel workflows.

How Has This Been Tested?

  • Ran: pytest qlib/tests/test_all_pipeline.py
  • Verified on custom joblib-based scripts for parallel workflows

Please confirm the above after testing.

Screenshots of Test Results (if appropriate)

(You can attach console output or screenshots here if needed)

Types of Changes

  • Bug fix
  • New feature
  • Documentation update

@github-actions github-actions bot added the waiting for triage Cannot auto-triage, wait for triage. label May 27, 2025
Copy link

@Abhijais4896 Abhijais4896 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

def init(self, *args, **kwargs):
maxtasksperchild = kwargs.pop("maxtasksperchild", None)
super(ParallelExt, self).init(*args, **kwargs)
if isinstance(self._backend, MultiprocessingBackend):
if isinstance(self._backend, MultiprocessingBackend) and maxtasksperchild is not None:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting for triage Cannot auto-triage, wait for triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants