Skip to content

Conversation

@michaelr524
Copy link
Collaborator

No description provided.

michaelr524 and others added 5 commits January 3, 2026 19:59
Based on user feedback from LessWrong/EA Forum about false positives,
aggressive flagging, and missing context issues.

Key changes planned:
- Single-pass full document extraction (replaces chunking)
- Multi-stage filtering (charity, supported elsewhere, dedup)
- Simplified review (summarization only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Was backwards: "defending weak claim by switching to strong one"
Now correct: "defending controversial claim by retreating to defensible one"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add DB-level title search with case-insensitive LIKE query
- Increase document limit from 30 to 100
- Add debounced search input with spinner
- Fix 'q' key quit issue when typing in search field
- Improve date format to human-readable (Dec 27, 2025)
- Fix alignment with fixed-width title padding

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add deleteSeries() to MetaEvaluationRepository
- Add delete confirmation modal in MainMenu (d key, y/n confirm)
- Improve API error handling with human-readable messages
- Switch dev-env.sh from zellij to tmux

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Plugin now passes full documentText for analysis instead of splitting into chunks
- Extractor uses documentText when text param is not provided (single-pass mode)
- Made text param optional in FallacyExtractorInput to support both modes
- Backwards compatible: chunk mode still works when text+chunkStartOffset provided

This reduces code complexity and provides better context to the LLM
by analyzing the full document at once.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Jan 3, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
roast-my-post Error Error Jan 22, 2026 10:32am

Request Review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jan 3, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

- Add SupportedElsewhereFilterTool that checks if flagged issues are
  actually supported/justified elsewhere in the document
- Integrate filter into fallacy-check plugin between extraction and
  comment generation phases
- Add debug logging to fallacy extractor and filter for visibility
- Add restart command to dev-env.sh with buffer clearing
- Update implementation notes with next steps (model testing,
  per-claim verification, extraction prompt improvements)

Results on test document show filter correctly identifies claims that
are justified by technical explanations later in the document. Opus
filters more aggressively (0 issues) vs Sonnet (1-2 issues).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add callOpenRouterWithTool() wrapper for OpenRouter API tool calling
- Add Gemini 3 Pro/Flash model IDs to OPENROUTER_MODELS
- Add temperature normalization per provider (Anthropic 0-1, others 0-2)
- Update supported-elsewhere filter to use OpenRouter for non-Claude models
- Add FALLACY_FILTER_MODEL env var for easy model switching
- Increase max_tokens to 8000 for OpenRouter (Gemini Pro needs more)
- Add error logging for tool call failures

Tested with Gemini 3 Flash ($0.003) and Pro ($0.054) - both agree
with Opus that all 5 issues are supported elsewhere (vs Sonnet keeping 1-2).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… restart

- Add model parameter to FallacyExtractorInput for OpenRouter models
- Support FALLACY_EXTRACTOR_MODEL env var for easy model switching
- Use callOpenRouterWithTool for non-Claude models (Gemini, GPT, etc.)
- Clear visible screen before scrollback in dev-env restart

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
michaelr524 and others added 3 commits January 7, 2026 11:08
- Update model testing results (Opus, Sonnet, Gemini Flash/Pro comparison)
- Document OpenRouter integration for multi-model testing
- Reorganize next steps by pipeline stage (extraction, filtering, review)
- Add planned filters: Principle of Charity, dedup/severity threshold
- Add cross-cutting concerns: multi-expert aggregation, observability, validation
- Add section 3.8: Prioritized implementation plan with 4 phases
- Include risk table with mitigations

Key insight: Phase 1 (observability + validation) must come first -
can't improve what you can't measure.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Meta-eval scoring for comment quality (accuracy, clarity, tone)
- Review stage improvements based on meta-eval feedback
- Feedback loop to iterate on prompts over time

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create telemetry module with StageMetrics, PipelineExecutionRecord types
- Add PipelineTelemetry collector class with fluent API
- Track 5 pipeline stages: extraction, dedup, filter, comment-gen, review
- Persist telemetry to EvaluationVersion.pipelineTelemetry JSON field
- Refactor FallacyCheckPlugin with helper methods for cleaner code

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add validation types (EvaluationSnapshot, DocumentComparisonResult, RegressionFlag)
- Add comment comparison logic with fuzzy matching (Levenshtein similarity)
- Add regression detection: score drop, lost comments, high-importance loss, extraction drop
- Add Validation screen to meta-evals CLI with Corpus/Compare/Results tabs
- Add repository methods for corpus queries and evaluation snapshots
- Clarify Settings UI shows judge model is for Score/Rank flows

TODO: Add baseline selection (pinned golden baseline vs latest run)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ValidationBaseline and ValidationBaselineSnapshot tables
- Add repository methods for baseline CRUD
- Update Validation UI with baseline management:
  - Create/delete/select baselines
  - Run pipeline on baseline documents
  - Compare new results vs saved baseline
  - Save results as new baseline
- Show change summary: "X kept, +Y new, -Z lost" per document
- Use [=] unchanged / [~] changed instead of pass/fail icons
- Clarify main menu labels (Score/Rank vs Validation)
- Remove emoji from menu items

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
michaelr524 and others added 2 commits January 7, 2026 12:55
- MainMenu now only has 4 options: Score/Rank, Validation, Settings, Exit
- Created ScoreRankMenu component with series list, create, delete
- Settings remains as modal overlay in MainMenu
- Updated App.tsx routing for new screen structure
- Navigation: SeriesDetail and CreateBaseline now return to ScoreRankMenu

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ValidationRun and ValidationRunSnapshot tables for persisting runs
- Capture per-item filter reasoning in pipeline telemetry (filteredItems)
- Record filter reasons from supported-elsewhere-filter and review stages
- Display filter reasoning for lost comments in validation UI
- Distinguish filtered comments (⊘) from not-extracted comments (−)
- Simplify UI: remove Results tab, auto-navigate to History after run
- Show all comments in scrollable list (no more "and X more" truncation)
- Add legend and summary breakdown (X filtered, Y not extracted)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…uter direct API

Multi-extractor system:
- Run multiple extractors in parallel with different models/settings
- Optional LLM judge for aggregation (disabled by default, uses simple dedup)
- Per-extractor configuration via FALLACY_EXTRACTORS env var

New extractor config options:
- `thinking: boolean` - Enable/disable extended thinking (Claude) or reasoning (OpenRouter)
- `temperature: number | "default"` - Explicit temp or use model's native default

OpenRouter direct API:
- Replaced OpenAI SDK with direct HTTP calls for full parameter control
- Proper `reasoning_effort` support: none/minimal/low/medium/high/xhigh
- New `callOpenRouterChat()` for non-tool-calling use cases
- Updated claim-evaluator to use new API

Telemetry & UI:
- Track temperatureConfig and thinkingEnabled per extractor
- Display extraction params in validation UI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new Extractor Lab screen to main menu
- Allows running fallacy extraction directly without full pipeline
- Configure multiple extractors with different models/temperatures
- Uses same validation corpus as Validation screen (50 docs)
- Display format matches Create Baseline (numbered, with dates)
- Export @roast/ai/fallacy-extraction module for external use

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update package.json export to use dist files instead of src
- Use static import instead of dynamic import in ExtractorLab
- Fixes ERR_REQUIRE_CYCLE_MODULE error when running extraction

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add UnifiedUsageMetrics type for consistent cost/token tracking across providers
- OpenRouter: Extract cost directly from API response (native_tokens_cost)
- Anthropic: Calculate cost from token usage using pricing table
- Add unifiedUsage to extractor, judge, filter, and review tools
- Pass costUsd to pipeline telemetry endStage() for each stage
- Display per-stage and total costs in Pipeline Flow UI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /api/monitor/lab/model-endpoints API for fetching available models
- Add useModelEndpoints hook for model selection in profile editor
- Fix null safety in multiExtractor deduplication (handle empty exactText)
- Update telemetry types and exports
- Update profile-loader and extraction types
- Update jobs infrastructure (worker, orchestrator, service)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ecution

- Add new Principle of Charity filter tool that interprets arguments
  charitably before critiquing, filtering issues that dissolve under
  generous interpretation
- Refactor pipeline to run filters dynamically based on filterChain order
  instead of hardcoded execution sequence
- Update PipelineView to display all filter stages dynamically with
  proper step numbering
- Add filter type validation and settings UI in FilterChainEditor

The filter order (charity → supported-elsewhere) is intentional:
1. Charity removes invalid critiques that aren't real problems
2. Supported-elsewhere then checks if remaining valid critiques are
   addressed in the document

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add expandable document rows showing all evaluation versions
- Allow selecting specific version per document via radio buttons
- Auto-select latest version when checking document without expanding
- Add /api/monitor/lab/corpus/versions endpoint for fetching versions
- Support beforeDate filter in baselines API for cutoff-based selection
- Add dev-env.sh psql command for database access
- Document dev scripts in CLAUDE.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… filters

- Add modelConfigResolver.ts for unified model configuration across extractors, judge, and filters
- Add reasoningBudget.ts for dynamic reasoning token budget calculation
- Add provider routing support to principle-of-charity and supported-elsewhere filters
- Consolidate single/multi-extractor paths into unified extraction flow
- Fix verbose logging in smart-text-searcher (was logging full document content)
- Add Claude thinking budget calculation from reasoning effort levels
- Refactor UI components to use shared ModelConfigurator and ProviderSelector
- Update CLAUDE.md with database access restrictions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
michaelr524 and others added 2 commits January 21, 2026 17:53
- Add actualApiParams and responseMetrics to judge output
- Capture judge telemetry in fallacy-check plugin
- Add telemetry fields to ExtractionPhaseTelemetry and StageMetrics
- Fix judge config to properly pass reasoning and provider from profile
- Update frontend types to support new telemetry fields

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ActualApiParams and ApiResponseMetrics types to both filter outputs
- Update supported-elsewhere-filter to capture and return API telemetry
- Update principle-of-charity-filter to capture and return API telemetry
- Update endStage calls to pass filter telemetry (actualApiParams, responseMetrics, unifiedUsage)
- Update PipelineView UI to display filter and judge telemetry details

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create types/common.ts with shared types:
  - ReasoningEffort, ReasoningConfig, ProviderPreferences
  - ActualApiParams, ApiResponseMetrics
  - EFFORT_TO_BUDGET_TOKENS constant
  - effortToBudgetTokens utility function

- Update all files to import from shared types:
  - extraction/types.ts
  - profile-types.ts
  - telemetry/types.ts
  - fallacy-judge/types.ts
  - fallacy-extractor/types.ts
  - principle-of-charity-filter/types.ts
  - supported-elsewhere-filter/types.ts
  - openrouter.ts

- Remove duplicate effortToBudgetTokens from filter implementations

This eliminates ~6 duplicate definitions of each type across the codebase.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… tools

Converted debug output in principle-of-charity-filter and supported-elsewhere-filter
from console.log to context.logger.debug for proper log level control.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…lit large files

Code review improvements:

- Fix getDefaultConfig() to use new filterChain array format (was using old
  { filters: [...] } format which caused config parsing issues)

- Consolidate JACCARD_SIMILARITY_THRESHOLD constant in common.ts (was
  duplicated in dedup.ts and multiExtractor.ts)

- Replace console.log/warn with structured logger calls in:
  - reasoningBudget.ts
  - profile-loader.ts (also removed DEBUG logs)
  - PipelineTelemetry.ts (refactored logSummary to use structured logging)

- Create shared LLM filter utility (tools/shared/llm-filter-utils.ts):
  - callLLMFilter<T>() abstracts Claude vs OpenRouter differences
  - buildThinkingConfig(), buildOpenRouterReasoning() helpers
  - truncateDocumentForContext() utility
  - Reduces code duplication between filter tools

- Split PipelineView.tsx (871 → 394 lines, ~55% reduction):
  - pipelineUtils.ts: Format helpers
  - ItemCards.tsx: FilteredItemCard, CommentCard components
  - ExtractorCards.tsx: ExtractorCard, DeduplicationCard components

- Fix duplicated ReasoningEffortLevel type in multiExtractor.ts
  (now uses ReasoningEffort from common.ts)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@qodo-code-review
Copy link

CI Feedback 🧐

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: Build and Push Docker Images (Dockerfile, main)

Failed stage: Build and push Docker image (main) [❌]

Failed test name: ""

Failure summary:

The action failed during the Docker build step when pnpm --filter @roast/web run build ran next
build and Next.js/webpack failed to compile due to a missing Node core module:
-
../../internal-packages/ai/src/shared/jobContext.ts: Module not found: Can't resolve 'async_hooks'
-
Import trace shows this browser bundle path pulling in server-only code: jobContext.tslogger.ts
reasoningBudget.ts../../internal-packages/ai/src/index.ts
./src/app/evaluators/new/CreateAgentForm.tsx
This caused next build to exit with status 1, which
made the Docker RUN ... pnpm --filter @roast/web run build step fail (Dockerfile line 28) and the
overall buildx action to fail.

Relevant error logs:
1:  ##[group]Runner Image Provisioner
2:  Hosted Compute Agent
...

1058:  #19 3.000 > vitest run "generated-readmes.vtest.ts"
1059:  #19 3.000 
1060:  #19 3.369 
1061:  #19 3.369  RUN  v3.2.4 /usr/src/app/apps/web
1062:  #19 3.369 
1063:  #19 4.177  ↓ src/__tests__/generated-readmes.vtest.ts (1 test | 1 skipped)
1064:  #19 4.191 
1065:  #19 4.191  Test Files  1 skipped (1)
1066:  #19 4.191       Tests  1 skipped (1)
1067:  #19 4.191    Start at  19:05:06
1068:  #19 4.192    Duration  821ms (transform 56ms, setup 132ms, collect 8ms, tests 0ms, environment 232ms, prepare 200ms)
1069:  #19 4.192 
1070:  #19 4.998    ▲ Next.js 15.3.6
1071:  #19 4.998 
1072:  #19 5.032    Creating an optimized production build ...
1073:  #19 58.02 Failed to compile.
1074:  #19 58.02 
1075:  #19 58.02 ../../internal-packages/ai/src/shared/jobContext.ts
1076:  #19 58.02 Module not found: Can't resolve 'async_hooks'
1077:  #19 58.02 
1078:  #19 58.02 https://nextjs.org/docs/messages/module-not-found
1079:  #19 58.02 
1080:  #19 58.02 Import trace for requested module:
1081:  #19 58.02 ../../internal-packages/ai/src/shared/logger.ts
1082:  #19 58.02 ../../internal-packages/ai/src/utils/reasoningBudget.ts
1083:  #19 58.02 ../../internal-packages/ai/src/index.ts
1084:  #19 58.02 ./src/app/evaluators/new/CreateAgentForm.tsx
1085:  #19 58.02 
1086:  #19 58.02 
1087:  #19 58.02 > Build failed because of webpack errors
1088:  #19 58.15 /usr/src/app/apps/web:
1089:  #19 58.15  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @roast/web@0.1.0 build: `pnpm --filter @roast/ai run check-schemas && pnpm --filter @roast/ai run check-readmes && next build`
1090:  #19 58.15 Exit status 1
1091:  #19 ERROR: process "/bin/sh -c NEXT_TELEMETRY_DISABLED=1     DOCKER_BUILD=true     DATABASE_URL=\"***localhost:5432/dummy?schema=public\"     AUTH_SECRET=\"build-time-dummy\"     ANTHROPIC_API_KEY=\"build-time-dummy\"     pnpm --filter @roast/web run build" did not complete successfully: exit code: 1
1092:  ------
1093:  > [builder  9/12] RUN NEXT_TELEMETRY_DISABLED=1     DOCKER_BUILD=true     DATABASE_URL="***localhost:5432/dummy?schema=public"     AUTH_SECRET="build-time-dummy"     ANTHROPIC_API_KEY="build-time-dummy"     pnpm --filter @roast/web run build:
1094:  58.02 ../../internal-packages/ai/src/shared/logger.ts
1095:  58.02 ../../internal-packages/ai/src/utils/reasoningBudget.ts
1096:  58.02 ../../internal-packages/ai/src/index.ts
1097:  58.02 ./src/app/evaluators/new/CreateAgentForm.tsx
1098:  58.02 
1099:  58.02 
1100:  58.02 > Build failed because of webpack errors
1101:  58.15 /usr/src/app/apps/web:
1102:  58.15  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @roast/web@0.1.0 build: `pnpm --filter @roast/ai run check-schemas && pnpm --filter @roast/ai run check-readmes && next build`
1103:  58.15 Exit status 1
1104:  ------
1105:  Dockerfile:28
1106:  --------------------
1107:  27 |     # Build web app with dummy env vars for validation only
1108:  28 | >>> RUN NEXT_TELEMETRY_DISABLED=1 \
1109:  29 | >>>     DOCKER_BUILD=true \
1110:  30 | >>>     DATABASE_URL="***localhost:5432/dummy?schema=public" \
1111:  31 | >>>     AUTH_SECRET="build-time-dummy" \
1112:  32 | >>>     ANTHROPIC_API_KEY="build-time-dummy" \
1113:  33 | >>>     pnpm --filter @roast/web run build
1114:  34 |     
1115:  --------------------
1116:  ERROR: failed to build: failed to solve: process "/bin/sh -c NEXT_TELEMETRY_DISABLED=1     DOCKER_BUILD=true     DATABASE_URL=\"***localhost:5432/dummy?schema=public\"     AUTH_SECRET=\"build-time-dummy\"     ANTHROPIC_API_KEY=\"build-time-dummy\"     pnpm --filter @roast/web run build" did not complete successfully: exit code: 1
1117:  ##[error]buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c NEXT_TELEMETRY_DISABLED=1     DOCKER_BUILD=true     DATABASE_URL=\"***localhost:5432/dummy?schema=public\"     AUTH_SECRET=\"build-time-dummy\"     ANTHROPIC_API_KEY=\"build-time-dummy\"     pnpm --filter @roast/web run build" did not complete successfully: exit code: 1
1118:  Post job cleanup.

The client index.ts was pulling in Node.js-only modules (async_hooks)
through this import chain:
  index.ts → openrouter.ts → reasoningBudget.ts → logger.ts → jobContext.ts

Fix:
- Create openrouter-types.ts with client-safe types/constants only
- Create reasoningBudget-client.ts with sync functions only
- Move ToolConfig/ToolContext types to tools/base/types.ts
- Export async API functions from server.ts instead of index.ts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace console.log with logger.debug for OpenRouter API calls
- Remove fire emoji debug output from fallacy extractor
- Include full response body in OpenRouter error messages for better
  debugging of rate limits (429) and other errors

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Prevents false positives where the model thinks recent dates are
"in the future" due to training cutoff. All system prompts now include
the current date as a reference point.

Affects: extractor, judge, filters, and review stages.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
michaelr524 and others added 2 commits January 22, 2026 10:29
- Add new "All Evals" tab to Lab UI showing recent user-facing evaluations
  with their pipeline telemetry (not just validation runs)
- Add API endpoint /api/monitor/lab/evaluations to fetch evaluation versions
  with pipelineTelemetry data
- Track items that pass through filters (not just filtered out items):
  - Add PassedItemRecord type to telemetry
  - Record passed items in principle-of-charity and supported-elsewhere filters
  - Display passed items in PipelineView (collapsed by default)
- New components: AllEvaluationsList, PassedItemCard, useAllEvaluations hook

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add whitespace-nowrap to prevent text wrapping
- Reduce padding and gap for better fit
- Shorten 'All Evals' to 'Evals'
- Add flex-shrink-0 to icons

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants