LexVerse

A universe where legal agents learn, act, and are evaluated.

Quick Start · Benchmarks · Outputs · Architecture

LexVerse is an evolving runtime for evaluating legal language models and agents. It brings legal benchmarks into a shared workflow for task preparation, execution, and result management while preserving their original task formats, interaction harnesses, and evaluators. We are continuing to integrate more benchmarks and develop environment capabilities for richer agent interactions.

⚖️ Supported Benchmarks

Benchmark	Coverage	Interaction	Verification
LexEval	23 tasks	Single response	Upstream task evaluator
LawBench	20 tasks	Single response	Upstream task evaluator
J1Bench	CI, CR, KQ, LC, CD, DD	Official multi-role harness	Upstream scenario evaluator
PLawBench	3 tasks	Single response	Upstream rubric evaluator
DLawBench	4 personas	Official consultation loop	Upstream session evaluator
Legal-world	6 stages	Official stage pipeline	Upstream stage evaluator
SimuCourt	6 tasks	Single response	Upstream task evaluator
MASER	complaint_generation	Official Law_Fact runner	Upstream scenario evaluator
LeCaRDv2	case_retrieval	Encoder / LLM rewrite retrieval	Recall@K & ranking evaluator

🚀 Quick Start

1. Install LexVerse

Use Python 3.10 or newer. Install LexVerse with the dependencies for the benchmarks you want to run:

python -m pip install -e ".[lexeval,lawbench,j1bench,simucourt,maser,lecardv2,openai]"
python -m lexverse --help

Compatibility note One environment can run all these benchmarks. LexEval and LawBench use different Rouge packages, so some Rouge-based scores may differ slightly from upstream results in a combined installation.

2. Create local configuration

Create local configuration files from the tracked *.example.yaml templates.

Configure at least one OpenAI-compatible connection in configs/secrets.local.yaml.

The benchmark YAML selects the connection with models.default.profile and the actual model with models.default.name. Multi-role and LLM-judged benchmarks also configure models.roles and models.evaluators. Model names belong in benchmark configuration, not in the secrets file.

3. Run LexEval

run --dry-run resolves the selected upstream cases and checks the configuration without calling the model. generate runs model generation only, run performs generation, official evaluation, and aggregation, and evaluate scores an existing run independently.

python -m lexverse run lexeval --config configs/benchmarks/lexeval.example.yaml --dry-run
python -m lexverse run lexeval --config configs/benchmarks/lexeval.local.yaml \
  --output-root runs/lexeval-experiment

4. Run LawBench

python -m lexverse run lawbench --config configs/benchmarks/lawbench.example.yaml --dry-run
python -m lexverse run lawbench --config configs/benchmarks/lawbench.local.yaml \
  --output-root runs/lawbench-experiment

If a run is interrupted, resume it from the frozen manifest snapshot:

python -m lexverse run --resume runs/lawbench-experiment

If generation has completed and only scoring needs to be rerun:

python -m lexverse evaluate --run-dir runs/lawbench-experiment

5. Run J1Bench

J1Bench uses gated data. Accept the terms on the J1-Eval dataset page and authenticate once:

hf auth login

The first run --dry-run or run downloads the selected J1-Eval_<SCENARIO>.jsonl files to .lexverse/datasets/j1bench/. Later runs reuse this cache and do not modify the downloaded files.

python -m lexverse run j1bench --config configs/benchmarks/j1bench.example.yaml --dry-run
python -m lexverse run j1bench --config configs/benchmarks/j1bench.local.yaml \
  --output-root runs/j1bench-experiment

J1Bench runs the official multi-role conversation for every case and invokes the official evaluator once per scenario. It is slower and more expensive than the two single-response benchmarks.

6. Run SimuCourt, MASER, and LeCaRDv2

python -m lexverse run simucourt --config configs/benchmarks/simucourt.example.yaml --dry-run
python -m lexverse run simucourt --config configs/benchmarks/simucourt.local.yaml \
  --output-root runs/simucourt-experiment

python -m lexverse run maser --config configs/benchmarks/maser.example.yaml --dry-run
python -m lexverse run maser --config configs/benchmarks/maser.local.yaml \
  --output-root runs/maser-experiment

python -m lexverse run lecardv2 --config configs/benchmarks/lecardv2.example.yaml --dry-run
python -m lexverse run lecardv2 --config configs/benchmarks/lecardv2.local.yaml \
  --output-root runs/lecardv2-experiment

Adapter details, configuration modes, and local reproduction results are documented in:

• lexverse/benchmarks/maser/README.md
• lexverse/benchmarks/simucourt/README.md
• lexverse/benchmarks/lecardv2/README.md

📂 Outputs

Each --output-root contains manifest.json, summary.json, trial records, and official evaluation artifacts. LexEval and LawBench report task-level scores in evaluation_result.csv; J1Bench and MASER keep per-case intermediate results and a scenario-level final result under trials/<scenario>/verifier/.

collect rebuilds summary.json from existing Trial records and the existing official evaluation_result.csv:

python -m lexverse collect --run-dir runs/lexeval-experiment

For a cheaper smoke test, adjust benchmark.default_limit or each task's limit in a local benchmark YAML (null selects all samples). Start a new run whenever changing models, tasks, or generation parameters.

🏗️ Architecture

LexVerse connects benchmark preparation, execution, evaluation, and result management in one workflow.

Open the interactive architecture map →

📝 Notes

• Do not edit files under .lexverse/datasets/j1bench/.
• Start a new run after modifying a local YAML; --resume reads the frozen manifest snapshot.
• Upstream repositories are pinned in each benchmark's catalog.yaml; patches make them relocatable without replacing evaluator logic.
• Benchmark and dataset licenses remain governed by their original sources.

LexVerse is released under the Apache License 2.0.

Contact Us

For issues and feature requests, use GitHub Issues. You can also email xieh@tsinghua.edu.cn.