Skip to content

feat(backend): Implement Agent Output Quality Scoring and Feedback Loop #277

Description

@devJaja

Summary

Add automatic quality scoring for agent outputs based on completeness, relevance, and format compliance, feeding scores back to reputation system.

Motivation

Currently, agent quality is only assessed by human review. Automatic scoring provides immediate quality feedback and enables data-driven agent selection.

Acceptance Criteria

  • Quality scoring function evaluates: completeness (has all required sections), relevance (matches prompt intent), format (follows expected schema)
  • Score range: 0-100 (normalized)
  • Scores stored with task execution records
  • Scores feed into on-chain reputation updates
  • Low-scoring outputs flagged for review
  • Quality metrics aggregated per agent (avg, min, max, trend)
  • Scoring rules configurable per agent type
  • Unit tests for scoring logic and edge cases

Files to Modify

File Action
backend/src/services/qualityScorer.ts New: output quality scoring service
backend/src/services/qualityScorer.types.ts New: scoring types
backend/src/coordinator/coordinator.ts Add quality scoring after agent execution
backend/src/agents/research/research.ts Define research output schema for scoring
backend/src/agents/coding/coding.ts Define coding output schema for scoring
backend/src/services/qualityScorer.test.ts New: scoring tests

How to Contribute

  1. Create branch feat/agent-quality-scoring
  2. Build quality scoring service with configurable rules
  3. Integrate scoring into coordinator workflow
  4. Define output schemas per agent type
  5. Write unit tests for scoring logic
  6. Run npm run test && npm run lint before submitting PR

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions