Skip to content

Fix default-weights.yaml: dead key, missing entries, and scoring drift #403

@jwm4

Description

@jwm4

Summary

Audit of src/agentready/data/default-weights.yaml against all registered assessors found three discrepancies that cause silent scoring drift.

Issues

1. Typo: separation_concerns vs separation_of_concerns

The yaml has:

separation_concerns: 0.03

But the assessor (SeparationOfConcernsAssessor) returns attribute_id = "separation_of_concerns". The yaml key never matches any assessor, so it is dead weight. The assessor falls back to default_weight=0.03, which happens to be the right value, but the dead yaml key contributes 3% to the rescaling denominator without any assessor claiming it. This means every other yaml-registered attribute is very slightly overweighted.

Fix: Rename separation_concerns to separation_of_concerns in the yaml.

2. repomix_config not in yaml

RepomixConfigAssessor was recently registered in create_all_assessors() (see #400) but is not in default-weights.yaml. It falls back to default_weight=0.02 and contributes to the scoring denominator outside the yaml's rescaling logic.

Fix: Add repomix_config: 0.02 to the yaml and reduce other weights to keep the total at exactly 1.0.

3. branch_protection not in yaml

BranchProtectionAssessor is not in the yaml and uses default_weight=0.005. In practice this is benign today because the assessor always returns not_applicable (it requires GitHub API access and is skipped from scoring). But it should be documented.

Fix: Either add branch_protection: 0.005 to the yaml, or add a comment noting it is intentionally omitted as a conditional/API-dependent assessor.

How to reproduce

import yaml
from pathlib import Path

with open("src/agentready/data/default-weights.yaml") as f:
    weights = {k: v for k, v in yaml.safe_load(f).items() if isinstance(v, float)}

# Dead key -- no assessor has this ID
assert "separation_concerns" in weights        # passes (shouldn't)
assert "separation_of_concerns" not in weights # passes (wrong)

# Missing entries
assert "repomix_config" not in weights    # passes (wrong)
assert "branch_protection" not in weights # passes (debatable)

Impact

  • All assessments are affected by the separation_concerns typo (slight overweighting of other attributes).
  • Any assessment where repomix_config is not skipped gets its weight applied outside the normalized yaml total.

Opened by Claude Code under the supervision of Bill Murdock.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions