Skip to content

Commit 708596c

Browse files
aktasbatuhanclaude
andcommitted
Add the kai CLI: monitor, explore, compare runs and benchmark tooling
A discoverable, plain-language command suite over bench runs (Rich + plotext, arrow-key/numbered menus): - kai monitor live dashboard: per-setup table + focused panel with a Tufte-style progress chart (best-so-far line + per-step dots) - kai explore guided menu; browse runs and drill in - kai show step through a run: idea/change/outcome notes, parent diff, prev/next stepping, fully back-navigable - kai best print or save the winning program of a run - kai compare setups side by side on a shared scale (small multiples) - kai ablation per-feature attribution (leave-one-out) + ABLATION.md - kai export results to CSV/JSON - kai runs / glossary; `kai --help` lists commands; `kai --version` Also: seed-level resume in the bench harness, a 4-model all-flash ablation config matrix, and removal of the Textual prototype TUI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 489b9da commit 708596c

27 files changed

Lines changed: 2963 additions & 324 deletions
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: false
39+
literature_review_enabled: false
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: false
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false

‎configs/ablation/auto_full.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: true
39+
literature_review_enabled: true
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: true
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: true
39+
literature_review_enabled: true
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: false
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false

‎configs/ablation/auto_noHMRD.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: false
39+
literature_review_enabled: true
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: true
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false

‎configs/ablation/auto_noLR.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: true
39+
literature_review_enabled: false
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: true
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false

‎configs/ablation/auto_warmup.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
max_iterations: 20
2+
checkpoint_interval: 10
3+
log_level: INFO
4+
random_seed: 42
5+
diff_based_evolution: true
6+
max_code_length: 30000
7+
llm:
8+
api_base: https://openrouter.ai/api/v1
9+
models:
10+
- name: stepfun/step-3.7-flash
11+
weight: 1.0
12+
- name: google/gemini-3.5-flash
13+
weight: 1.0
14+
- name: deepseek/deepseek-v4-flash
15+
weight: 1.0
16+
- name: qwen/qwen3.6-flash
17+
weight: 1.0
18+
island_models:
19+
- - name: google/gemini-3.5-flash
20+
weight: 1.0
21+
- name: deepseek/deepseek-v4-flash
22+
weight: 1.0
23+
- - name: stepfun/step-3.7-flash
24+
weight: 1.0
25+
- name: qwen/qwen3.6-flash
26+
weight: 1.0
27+
temperature: 0.7
28+
top_p: 0.95
29+
max_tokens: null
30+
timeout: 120
31+
retries: 2
32+
retry_delay: 3
33+
prompt:
34+
num_top_programs: 2
35+
num_diverse_programs: 1
36+
use_template_stochasticity: true
37+
include_artifacts: true
38+
require_hmrd: true
39+
literature_review_enabled: true
40+
literature_review_max_chars: 4000
41+
strategy_clustering_enabled: true
42+
strategy_num_clusters: 6
43+
strategy_epsilon: 0.05
44+
database:
45+
population_size: 60
46+
archive_size: 30
47+
num_islands: 2
48+
migration_interval: 20
49+
migration_rate: 0.1
50+
elite_selection_ratio: 0.2
51+
exploitation_ratio: 0.7
52+
evaluator:
53+
timeout: 60
54+
cascade_evaluation: true
55+
cascade_thresholds:
56+
- 0.3
57+
- 0.6
58+
parallel_evaluations: 2
59+
use_llm_feedback: false
60+
warmup:
61+
enabled: true

0 commit comments

Comments
 (0)