feat: Enabling automation of experiments running v2.0

scripts/exp/ablation/README.md

@@ -0,0 +1,9 @@
+# Introduction
+
+| name | .env      | desc |
+| --   | --        | --   |
+| full | full.env |  enable all features    |
+| minicase | minicase.env | enable minicase |
+
+
+

scripts/exp/ablation/env/basic.env

@@ -0,0 +1,3 @@
+KG_IF_USING_VECTOR_RAG=False
+KG_IF_USING_GRAPH_RAG=False
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=False

scripts/exp/ablation/env/max.env

@@ -0,0 +1,3 @@
+KG_IF_USING_VECTOR_RAG=True
+KG_IF_USING_GRAPH_RAG=True
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True

scripts/exp/ablation/env/mini-case.env

@@ -0,0 +1,3 @@
+KG_IF_USING_VECTOR_RAG=True
+KG_IF_USING_GRAPH_RAG=False
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True

scripts/exp/ablation/env/pro.env

@@ -0,0 +1,3 @@
+KG_IF_USING_VECTOR_RAG=True
+KG_IF_USING_GRAPH_RAG=False
+KG_IF_ACTION_CHOOSING_BASED_ON_UCB=True

scripts/exp/tools/README.md

@@ -0,0 +1,148 @@
+### Tools Directory
+
+This directory provides scripts to run experiments with different environment configurations, collect results, and demonstrate usage through an example script.
+
+### Directory Structure
+
+scripts/exp/tools/
+├── run_envs.sh       # Script for running experiments
+├── collect.py        # Results collection and summary
+├── test_system.sh    # Usage script for rdagent kaggle loop
+├── README.md         # This documentation
+
+Tools Overview
+
+	1.	run_envs.sh: Executes experiments with different environment configurations in parallel.
+	2.	collect.py: Collects and summarizes experiment results into a single file.
+	3.	test_system.sh: Demonstrates how to use the above tools together for experiment execution and result collection [for rdagent kaggle loop]
+
+### Getting Started
+
+Prerequisites
+
+1. Ensure the scripts have execution permissions:
+
+```
+chmod +x scripts/exp/tools/run_envs.sh
+chmod +x scripts/exp/tools/test_system.sh
+```
+
+2. Place your .env files in the desired directory for environment configurations.
+
+### Usage
+
+#### 1. Running Experiments with Different Environments
+
+The run_envs.sh script allows running a command with multiple environment configurations in parallel.
+
+**Command Syntax**
+
+```
+./run_envs.sh -d <dir_to_.envfiles> -j <number_of_parallel_processes> -- <command>
+```
+
+**Example Usage**
+
+Basic example:
+
+```
+./run_envs.sh -d env_files -j 1 -- echo "Hello"
+```
+
+Practical example:
+
+```
+dotenv run -- ./run_envs.sh -d RD-Agent/scripts/exp/ablation/env -j 1 -- python RD-Agent/rdagent/app/kaggle/loop.py
+```
+
+**Explanation:**
+
+| Option | Description |
+| --- | --- |
+| `-d` | Specifies the directory containing .env files. |
+| `-j` | Number of parallel processes to run (e.g., 1 for sequential execution). |
+| `--` | Separates script options from the command to execute. |
+| `<command>` | The command to execute with the environment variables loaded. |
+
+
+####2.Collecting Results
+
+The collect.py script processes logs and generates a summary JSON file.
+
+**Command Syntax**
+
+```
+python collect.py --log_path <path_to_logs> --output_name <summary_filename>
+```
+
+**Example Usage**
+
+Collect results from logs:
+
+```
+python collect.py --log_path logs --output_name summary.json
+```
+**Explanation:**
+
+| Option | Description |
+| --- | --- |
+| `--log_path` | Required. Specifies the directory containing experiment logs. |
+| `--output_name` | Optional. The name of the output summary file (default: summary.json). |
+
+#### 3. Example Workflow [for rdagent kaggle loop]
+
+Use the test_system.sh script to demonstrate a complete workflow.
+
+**Steps:**
+
+1.	Ensure the scripts are executable:
+
+```
+chmod +x scripts/exp/tools/run_envs.sh
+chmod +x scripts/exp/tools/test_system.sh
+```
+
+2.	Run the test system:
+
+```
+./scripts/exp/tools/test_system.sh
+```
+
+This will:
+	1.	Load environment configurations from .env files.
+	2.	Execute experiments using the configurations.
+
+3. Find your logs in the logs directory.
+
+4. Use the collect.py script to summarize results:
+
+```
+python collect.py --log_path logs --output_name summary.json
+```
+
+### Troubleshooting
+
+#### Permission Denied
+
+If you encounter a PermissionError when running scripts:
+
+1.Ensure the script has execution permissions:
+
+```
+chmod +x ./scripts/exp/tools/run_envs.sh
+chmod +x ./scripts/exp/tools/test_system.sh
+```
+
+2.Verify file ownership:
+
+``` 
+ls -l ./scripts/exp/tools/
+```
+
+### Notes
+* Scale parallel processes as needed using the -j parameter.
+* Avoid errors by ensuring .env files are correctly formatted.
+* Modify test_system.sh to meet your project's specific needs.
+* Add other metrics interested in collect.py to summarize automatically.
+
+For further assistance, refer to the comments within the scripts or reach out to the development team.

scripts/exp/tools/collect.py

@@ -0,0 +1,83 @@
+import os
+import json
+import argparse
+from pathlib import Path
+from datetime import datetime
+from rdagent.log.storage import FileStorage
+from rdagent.scenarios.kaggle.kaggle_crawler import (
+    leaderboard_scores,
+)
+
+def collect_results(log_path) -> list[dict]:
+    summary = []
+    log_storage = FileStorage(Path(log_path))
+    evaluation_metric_direction = None
+    # Extract score from trace using the same approach as UI
+    for msg in log_storage.iter_msg():
+        if "scenario" in msg.tag:
+            competition_name = msg.content.competition # Find the competition name     
+            leaderboard = leaderboard_scores(competition_name)
+            evaluation_metric_direction = float(leaderboard[0]) > float(leaderboard[-1])
+
+        if "runner result" in msg.tag:
+            if msg.content.result is not None:
+                score = msg.content.result
+                summary.append({
+                    "competition_name": competition_name,
+                    "score": score,
+                    "workspace": msg.content.experiment_workspace.workspace_path,
+                    "evaluation_metric_direction": evaluation_metric_direction
+                })
+    return summary
+
+def generate_summary(results, output_path):
+    summary = {
+        "configs": {}, #TODO: add config? 
+        "best_result": {"competition_name": None, "score": None},
+        "timestamp": datetime.now().strftime("%Y%m%d_%H%M%S"),
+        #Add other metrics that we want to track in the future (eg. is there successive increase?)
+    }
+    for result in results:
+        # Update best result
+        # If the evaluation metric is higher, it is better
+        if result["evaluation_metric_direction"]:
+            if (result["score"] is not None and 
+                (summary["best_result"]["score"] is None or 
+                result["score"] > summary["best_result"]["score"])):
+                summary["best_result"].update({
+                    "score": result["score"],
+                    "competition_name": result["competition_name"]
+                })
+        else:
+            if (result["score"] is not None and 
+                (summary["best_result"]["score"] is None or 
+                result["score"] < summary["best_result"]["score"])):
+                summary["best_result"].update({
+                    "score": result["score"],
+                    "competition_name": result["competition_name"]
+                })
+
+    with open(output_path, "w") as f: 
+        json.dump(summary, f, indent=4)
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='Collect and summarize experiment results')
+    parser.add_argument('--log_path', type=str, required=True,
+                       help='Path to the log directory containing experiment results')
+    parser.add_argument('--output_name', type=str, default='summary.json',
+                       help='Name of the output summary file (default: summary.json)')
+    return parser.parse_args()
+
+if __name__ == "__main__":
+    args = parse_args()
+    log_path = Path(args.log_path)
+
+    # Verify the log path exists
+    if not log_path.exists():
+        raise FileNotFoundError(f"Log path does not exist: {log_path}")
+
+    results = collect_results(log_path)
+    output_path = log_path / args.output_name
+    generate_summary(results, output_path)
+    print("Summary generated successfully at", output_path)
+

scripts/exp/tools/run_envs.sh

@@ -0,0 +1,51 @@
+#!/bin/sh
+cat << "EOF" > /dev/null
+Given a directory with *.env files.  Run each one.
+
+usage for example:
+
+  1) directly run command without extra shared envs
+  ./run_envs.sh -d <dir_to_*.envfiles> -j <number of parallel process> -- <command>
+
+  2) load shared envs `.env` before running command with different envs.
+  dotenv run -- ./run_envs.sh -d <dir_to_*.envfiles> -j <number of parallel process> -- <command>
+
+EOF
+
+# Function to display usage
+usage() {
+  echo "Usage: $0 -d <dir_to_*.envfiles> -j <number of parallel process> -- <command>"
+  exit 1
+}
+
+# Parse command line arguments
+while getopts "d:j:" opt; do
+  case $opt in
+    d) DIR=$OPTARG ;;
+    j) JOBS=$OPTARG ;;
+    *) usage ;;
+  esac
+done
+
+# Shift to get the command
+shift $((OPTIND -1))
+
+# Check if directory and jobs are set
+if [ -z "$DIR" ] || [ -z "$JOBS" ] || [ $# -eq 0 ]; then
+  usage
+fi
+
+COMMAND="$@"
+
+# Before running commands
+echo "Running experiments with following env files:"
+find "$DIR" -name "*.env" -exec echo "{}" \;
+
+# Export and run each .env file in parallel
+find "$DIR" -name "*.env" | xargs -n 1 -P "$JOBS" -I {} sh -c "
+  set -a
+  . {}
+  set +a
+  $COMMAND
+"
+

scripts/exp/tools/test_system.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# Test directory setup
+TEST_DIR="test_run"
+mkdir -p "$TEST_DIR/results"
+mkdir -p "$TEST_DIR/logs"
+
+# Define paths
+ENV_DIR="/home/v-xisenwang/RD-Agent/scripts/exp/ablation/env"
+PYTHON_SCRIPT="/home/v-xisenwang/RD-Agent/rdagent/app/kaggle/loop.py"
+
+# Run the experiment
+echo "Running experiments..."
+dotenv run -- ./scripts/exp/tools/run_envs.sh -d "$ENV_DIR" -j 4 -- \
+    python "$PYTHON_SCRIPT" \
+    --competition "spaceship-titanic" \ 
+
+# Cleanup (optional - comment out if you want to keep results)
+# rm -rf "$TEST_DIR"