feat: Add Web Arena as a benchmark

[amirrezaalasti]

Problem Statement

We should integrate WebArena as an additional benchmark in our evals/ suite to measure performance on realistic, complex web tasks. WebArena provides a highly realistic environment for evaluating LLM-native web interfaces, simulating multi-step interactions across varied domains (e-commerce, forums, CMS).

Proposed Solution / API Design

To support WebArena, we need to add a dedicated scenario wrapper that can interface with the WebArena environment, update our main evaluation runner to execute these tasks, and document the required local setup.

Proposed Checklist

• Add scenario integration for WebArena (evals/scenarios/webarena.ts).
• Update evals/run-benchmark.ts to support running the new webarena benchmark.
• Update README.md and documentation with the setup instructions and new benchmark results.

Example Implementation

// evals/scenarios/webarena.ts
export class WebArenaScenario {
  private environmentUrl: string;

  constructor(environmentUrl: string = process.env.WEBARENA_URL || 'http://localhost:8000') {
    this.environmentUrl = environmentUrl;
  }

  async setup(): Promise<void> {
    console.log(`[WebArena] Connecting to environment at ${this.environmentUrl}`);
    // Initialization logic for the WebArena Docker backend
  }

  async runTask(agent: WebAgent, task: WebArenaTask): Promise<EvaluationResult> {
    await agent.navigate(task.startUrl);
    const executionState = await agent.execute(task.intent);
    
    // Evaluate the final DOM state against WebArena's expected criteria
    return this.evaluateState(task.taskId, executionState);
  }
}

// evals/run-benchmark.ts
import { WebArenaScenario } from './scenarios/webarena';

// Usage in the main runner
if (targetSuite === 'webarena') {
  const scenario = new WebArenaScenario();
  await scenario.setup();
  
  for (const task of WEBARENA_TASKS) {
    const result = await scenario.runTask(agent, task);
    metrics.record(result);
  }
}

Alternatives Considered

• MiniWoB++: We considered MiniWoB++ for web task evaluation, but the tasks are largely comprised of simplified HTML pages with basic interactions (e.g., clicking a button or simple drag-and-drop). It does not accurately reflect the complex DOM structures, nested shadow DOMs, or multi-page navigation required for modern applications.
• Mind2Web: Another strong alternative, but Mind2Web operates primarily on offline, static DOM dumps. WebArena allows our agents to interact with a live, stateful environment, which is necessary for testing iterative reasoning and error-recovery capabilities.
• Custom E2E Playwright Suite: Building our own hardcoded test suite on live websites. We rejected this because websites frequently change their layouts, which would break our evaluation baselines.

Additional Context

• Dependencies: Running this benchmark will require contributors to spin up the local WebArena Docker containers. This requires significant local compute/RAM. We should ensure the README.md clearly outlines these hardware requirements.
• Reference: [WebArena GitHub Repository](https://github.com/web-arena-x/webarena) and [WebArena Paper](https://arxiv.org/abs/2307.13854).