[Windows] Runaway worker spawning causes system resource exhaustion (200+ workers)

# Bug Report: Runaway Worker Spawning in pi-messenger Crew (Windows)

## Summary
pi-messenger crew enters an infinite worker spawning loop on Windows, creating 200+ worker instances in minutes, causing system resource exhaustion. Happened twice independently with the same plan/tasks.

## Environment
- **OS:** Windows 11 (running in Git Bash)
- **Node:** v22.x (exact version available if needed)
- **pi-messenger:** v0.13.0
- **pi:** Latest (installed via npm global)
- **Shell:** Git Bash (MinGW)

## Reproduction Steps

### Initial Setup
1. Repository: `scalyr-config` (146 dashboards, 66 parsers)
2. Create crew plan with 8 tasks:
   ```bash
   pi_messenger({ action: "plan", prompt: "<tech debt analysis plan>" })
   ```
3. Plan creates 8 tasks (see attached `plan.json`)

### Trigger Runaway
4. Execute work:
   ```bash
   pi_messenger({ action: "work" })
   # OR
   pi_messenger({ action: "work", autonomous: true })
   ```

### Observed Behavior
- **Incident 1:** 184 workers spawned in ~2 minutes (17:02-17:04 UTC)
- **Incident 2:** 226 workers spawned in similar timeframe (18:07 UTC)
- Multiple workers per second at peak
- All working on same task (task-1)
- System resource exhaustion (30+ visible processes killed manually)

## Evidence

### Worker Artifacts
```
.pi/messenger/crew/artifacts/
├── 226 *_input.md files (226 unique worker instances)
├── Multiple workers assigned to same task
└── Workers appeared to execute but orchestrator kept spawning more
```

### Task State (Stuck)
```json
{
  "id": "task-1",
  "status": "in_progress",
  "assigned_to": "BrightZenith",
  "attempt_count": 1,
  "started_at": "2026-03-12T17:05:33.087Z"
}
```

### Worker Progress Logs
Workers were actually executing:
```
[2026-03-12T17:02:26.967Z] (system) Assigned to crew-worker (attempt 1)
[2026-03-12T17:02:29.143Z] (system) Assigned to crew-worker (attempt 1)
[2026-03-12T17:02:33.489Z] (system) Assigned to crew-worker (attempt 1)
...
[2026-03-12T17:04:40.765Z] (system) Assigned to crew-worker (attempt 1)
```

28 assignments in ~2 minutes, multiple per second.

### Worker Output
Workers checked status and asked for input instead of autonomously claiming tasks:
```
Current task assignments:
- task-1: Assigned to BrightZenith (in progress 🔄)
- task-2 through task-8: Unassigned (⬜)

You (GoldPhoenix) currently have no tasks assigned.

What would you like to do?
```

## Root Cause Analysis

### Primary Issue: Process Lifecycle Tracking Broken
The orchestrator spawns workers but cannot properly track their lifecycle, leading to:
1. Worker spawned via `spawn("pi", args, {...})`
2. Worker process starts successfully
3. **BUG:** Orchestrator loses track of worker (process detachment or exit detection failure)
4. Orchestrator thinks worker crashed/exited immediately
5. Task reset to `todo` state via `proc.on("close")` handler
6. Task appears available again → spawn another worker
7. **Infinite loop**

### Secondary Issues

**A. Windows-specific spawn problem (PARTIALLY FIXED)**

Original code:
```typescript
const proc = spawn("pi", args, {
  cwd,
  stdio: ["ignore", "pipe", "pipe"],
  shell: process.platform === "win32",  // ← WRONG
  env,
});
```

Using `shell: true` causes `cmd.exe` to spawn and exit immediately, detaching the actual `pi` process.

**User applied workaround:**
```typescript
const proc = spawn(process.platform === "win32" ? "pi.cmd" : "pi", args, {
  cwd,
  stdio: ["ignore", "pipe", "pipe"],
  // No shell: true
  env,
});
```

**HOWEVER:** Runaway still happened AFTER this fix, indicating deeper orchestration issues.

**B. Workers not autonomously claiming tasks**

With `autonomous: true`, workers should claim available tasks. Instead they:
- Check task status
- See other tasks available
- Ask "What would you like to do?" and wait for user input
- Never autonomously claim work

**C. No concurrency limits or rate limiting**

No visible safeguards prevent spawning 200+ workers simultaneously:
- No max worker count enforcement
- No spawn rate limiting
- No detection of duplicate work on same task

## Expected Behavior

1. **Worker spawning:**
   - Check if worker already running for task
   - Enforce max concurrent workers (configurable, default 2-4?)
   - Rate limit spawning (e.g., max 1 worker per second)

2. **Worker lifecycle:**
   - Properly track worker process (no detachment)
   - Detect actual worker exit (not shell exit)
   - Only reset task to `todo` if worker genuinely failed

3. **Autonomous mode:**
   - Workers should claim available tasks without user prompts
   - Coordinate via mesh to avoid duplicate work
   - Exit gracefully when no work available

4. **Safeguards:**
   - Max worker count limit
   - Timeout detection for stuck tasks
   - Prevent multiple workers on same task

## Minimal Reproduction

**Any multi-task crew plan triggers this on Windows:**

```javascript
// 1. Create plan with 3+ tasks
pi_messenger({ 
  action: "plan", 
  prompt: "Create 3 simple tasks:\n1. List files\n2. Count files\n3. Summarize results"
});

// 2. Start work
pi_messenger({ action: "work", autonomous: true });

// 3. Observe runaway spawning
// Expected: 1-2 workers
// Actual: 50+ workers in <1 minute
```

## Workarounds

### For Users (Temporary)
1. **DO NOT use crew on Windows** until fixed
2. If accidentally triggered:
   ```bash
   taskkill //F //IM node.exe
   ```
3. Manual task execution:
   ```javascript
   pi_messenger({ action: "task.start", id: "task-1" })
   // Work on task manually
   pi_messenger({ action: "task.done", id: "task-1", summary: "..." })
   ```

### For Maintainers

**Fix 1: Windows spawn (crew/lobby.ts line ~137)**
```typescript
// WRONG
const proc = spawn("pi", args, { shell: process.platform === "win32" });

// CORRECT
const proc = spawn(process.platform === "win32" ? "pi.cmd" : "pi", args, {});
```

**Fix 2: Add worker count limit (crew/spawn.ts)**
```typescript
export function spawnWorkersForReadyTasks(
  cwd: string,
  maxWorkers: number,
): SpawnResult {
  // Add check for currently running workers
  const runningWorkers = getLobbyWorkerCount(cwd);
  if (runningWorkers >= maxWorkers) {
    return { assigned: 0, firstWorkerName: null };
  }
  
  // Limit spawning to available slots
  const slotsAvailable = maxWorkers - runningWorkers;
  // ...
}
```

**Fix 3: Prevent duplicate task assignment**
```typescript
// Before spawning, check if task already has a running worker
const existingWorker = getLobbyWorkers(cwd).find(w => w.assignedTaskId === task.id);
if (existingWorker && existingWorker.proc.exitCode === null) {
  continue; // Skip, already being worked on
}
```

## Additional Context

### Files Available
- `plan.json` - The crew plan that triggered runaway
- `task-1.json` through `task-8.json` - Task definitions
- Sample worker artifacts showing spawn timestamps
- Full incident report with timeline

### System Impact
- 200+ node.exe processes
- High CPU/memory usage
- System became unresponsive
- Required manual process killing

### User Workaround Applied
User patched `crew/lobby.ts` to use `pi.cmd` on Windows, but runaway still occurred, confirming the issue is deeper than just spawn command.

---

## Proposed Solution Priority

1. **P0 - Critical:** Add max worker count enforcement and prevent spawning if already at limit
2. **P0 - Critical:** Fix Windows spawn to use `pi.cmd` (prevents process detachment)
3. **P1 - High:** Prevent duplicate workers on same task
4. **P1 - High:** Add spawn rate limiting
5. **P2 - Medium:** Fix autonomous worker task claiming (remove user prompts in autonomous mode)
6. **P2 - Medium:** Add worker lifecycle logging/debugging
7. **P3 - Low:** Add timeout detection for stuck tasks

## Questions for Maintainers

1. Is crew tested on Windows? (Appears to be Linux/Mac focused)
2. Are there existing integration tests for worker spawning?
3. What's the intended max worker count? (No config option found)
4. Should workers in autonomous mode ever prompt for user input?

---

**Reporter:** Windows user via AI agent investigation  
**Date:** 2026-03-12  
**Severity:** Critical (system resource exhaustion)  
**Frequency:** 100% reproduction rate on Windows with multi-task plans


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Windows] Runaway worker spawning causes system resource exhaustion (200+ workers) #11

Bug Report: Runaway Worker Spawning in pi-messenger Crew (Windows)

Summary

Environment

Reproduction Steps

Initial Setup

Trigger Runaway

Observed Behavior

Evidence

Worker Artifacts

Task State (Stuck)

Worker Progress Logs

Worker Output

Root Cause Analysis

Primary Issue: Process Lifecycle Tracking Broken

Secondary Issues

Expected Behavior

Minimal Reproduction

Workarounds

For Users (Temporary)

For Maintainers

Additional Context

Files Available

System Impact

User Workaround Applied

Proposed Solution Priority

Questions for Maintainers

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Windows] Runaway worker spawning causes system resource exhaustion (200+ workers) #11

Description

Bug Report: Runaway Worker Spawning in pi-messenger Crew (Windows)

Summary

Environment

Reproduction Steps

Initial Setup

Trigger Runaway

Observed Behavior

Evidence

Worker Artifacts

Task State (Stuck)

Worker Progress Logs

Worker Output

Root Cause Analysis

Primary Issue: Process Lifecycle Tracking Broken

Secondary Issues

Expected Behavior

Minimal Reproduction

Workarounds

For Users (Temporary)

For Maintainers

Additional Context

Files Available

System Impact

User Workaround Applied

Proposed Solution Priority

Questions for Maintainers

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions