-
Notifications
You must be signed in to change notification settings - Fork 6
Fallacy checker refactor #385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Based on user feedback from LessWrong/EA Forum about false positives, aggressive flagging, and missing context issues. Key changes planned: - Single-pass full document extraction (replaces chunking) - Multi-stage filtering (charity, supported elsewhere, dedup) - Simplified review (summarization only) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Was backwards: "defending weak claim by switching to strong one" Now correct: "defending controversial claim by retreating to defensible one" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add DB-level title search with case-insensitive LIKE query - Increase document limit from 30 to 100 - Add debounced search input with spinner - Fix 'q' key quit issue when typing in search field - Improve date format to human-readable (Dec 27, 2025) - Fix alignment with fixed-width title padding 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add deleteSeries() to MetaEvaluationRepository - Add delete confirmation modal in MainMenu (d key, y/n confirm) - Improve API error handling with human-readable messages - Switch dev-env.sh from zellij to tmux 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Plugin now passes full documentText for analysis instead of splitting into chunks - Extractor uses documentText when text param is not provided (single-pass mode) - Made text param optional in FallacyExtractorInput to support both modes - Backwards compatible: chunk mode still works when text+chunkStartOffset provided This reduces code complexity and provides better context to the LLM by analyzing the full document at once. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Add SupportedElsewhereFilterTool that checks if flagged issues are actually supported/justified elsewhere in the document - Integrate filter into fallacy-check plugin between extraction and comment generation phases - Add debug logging to fallacy extractor and filter for visibility - Add restart command to dev-env.sh with buffer clearing - Update implementation notes with next steps (model testing, per-claim verification, extraction prompt improvements) Results on test document show filter correctly identifies claims that are justified by technical explanations later in the document. Opus filters more aggressively (0 issues) vs Sonnet (1-2 issues). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add callOpenRouterWithTool() wrapper for OpenRouter API tool calling - Add Gemini 3 Pro/Flash model IDs to OPENROUTER_MODELS - Add temperature normalization per provider (Anthropic 0-1, others 0-2) - Update supported-elsewhere filter to use OpenRouter for non-Claude models - Add FALLACY_FILTER_MODEL env var for easy model switching - Increase max_tokens to 8000 for OpenRouter (Gemini Pro needs more) - Add error logging for tool call failures Tested with Gemini 3 Flash ($0.003) and Pro ($0.054) - both agree with Opus that all 5 issues are supported elsewhere (vs Sonnet keeping 1-2). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… restart - Add model parameter to FallacyExtractorInput for OpenRouter models - Support FALLACY_EXTRACTOR_MODEL env var for easy model switching - Use callOpenRouterWithTool for non-Claude models (Gemini, GPT, etc.) - Clear visible screen before scrollback in dev-env restart 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update model testing results (Opus, Sonnet, Gemini Flash/Pro comparison) - Document OpenRouter integration for multi-model testing - Reorganize next steps by pipeline stage (extraction, filtering, review) - Add planned filters: Principle of Charity, dedup/severity threshold - Add cross-cutting concerns: multi-expert aggregation, observability, validation - Add section 3.8: Prioritized implementation plan with 4 phases - Include risk table with mitigations Key insight: Phase 1 (observability + validation) must come first - can't improve what you can't measure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Meta-eval scoring for comment quality (accuracy, clarity, tone) - Review stage improvements based on meta-eval feedback - Feedback loop to iterate on prompts over time 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create telemetry module with StageMetrics, PipelineExecutionRecord types - Add PipelineTelemetry collector class with fluent API - Track 5 pipeline stages: extraction, dedup, filter, comment-gen, review - Persist telemetry to EvaluationVersion.pipelineTelemetry JSON field - Refactor FallacyCheckPlugin with helper methods for cleaner code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add validation types (EvaluationSnapshot, DocumentComparisonResult, RegressionFlag) - Add comment comparison logic with fuzzy matching (Levenshtein similarity) - Add regression detection: score drop, lost comments, high-importance loss, extraction drop - Add Validation screen to meta-evals CLI with Corpus/Compare/Results tabs - Add repository methods for corpus queries and evaluation snapshots - Clarify Settings UI shows judge model is for Score/Rank flows TODO: Add baseline selection (pinned golden baseline vs latest run) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ValidationBaseline and ValidationBaselineSnapshot tables - Add repository methods for baseline CRUD - Update Validation UI with baseline management: - Create/delete/select baselines - Run pipeline on baseline documents - Compare new results vs saved baseline - Save results as new baseline - Show change summary: "X kept, +Y new, -Z lost" per document - Use [=] unchanged / [~] changed instead of pass/fail icons - Clarify main menu labels (Score/Rank vs Validation) - Remove emoji from menu items 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- MainMenu now only has 4 options: Score/Rank, Validation, Settings, Exit - Created ScoreRankMenu component with series list, create, delete - Settings remains as modal overlay in MainMenu - Updated App.tsx routing for new screen structure - Navigation: SeriesDetail and CreateBaseline now return to ScoreRankMenu 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ValidationRun and ValidationRunSnapshot tables for persisting runs - Capture per-item filter reasoning in pipeline telemetry (filteredItems) - Record filter reasons from supported-elsewhere-filter and review stages - Display filter reasoning for lost comments in validation UI - Distinguish filtered comments (⊘) from not-extracted comments (−) - Simplify UI: remove Results tab, auto-navigate to History after run - Show all comments in scrollable list (no more "and X more" truncation) - Add legend and summary breakdown (X filtered, Y not extracted) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…uter direct API Multi-extractor system: - Run multiple extractors in parallel with different models/settings - Optional LLM judge for aggregation (disabled by default, uses simple dedup) - Per-extractor configuration via FALLACY_EXTRACTORS env var New extractor config options: - `thinking: boolean` - Enable/disable extended thinking (Claude) or reasoning (OpenRouter) - `temperature: number | "default"` - Explicit temp or use model's native default OpenRouter direct API: - Replaced OpenAI SDK with direct HTTP calls for full parameter control - Proper `reasoning_effort` support: none/minimal/low/medium/high/xhigh - New `callOpenRouterChat()` for non-tool-calling use cases - Updated claim-evaluator to use new API Telemetry & UI: - Track temperatureConfig and thinkingEnabled per extractor - Display extraction params in validation UI 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new Extractor Lab screen to main menu - Allows running fallacy extraction directly without full pipeline - Configure multiple extractors with different models/temperatures - Uses same validation corpus as Validation screen (50 docs) - Display format matches Create Baseline (numbered, with dates) - Export @roast/ai/fallacy-extraction module for external use 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update package.json export to use dist files instead of src - Use static import instead of dynamic import in ExtractorLab - Fixes ERR_REQUIRE_CYCLE_MODULE error when running extraction Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add UnifiedUsageMetrics type for consistent cost/token tracking across providers - OpenRouter: Extract cost directly from API response (native_tokens_cost) - Anthropic: Calculate cost from token usage using pricing table - Add unifiedUsage to extractor, judge, filter, and review tools - Pass costUsd to pipeline telemetry endStage() for each stage - Display per-stage and total costs in Pipeline Flow UI Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add /api/monitor/lab/model-endpoints API for fetching available models - Add useModelEndpoints hook for model selection in profile editor - Fix null safety in multiExtractor deduplication (handle empty exactText) - Update telemetry types and exports - Update profile-loader and extraction types - Update jobs infrastructure (worker, orchestrator, service) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ecution - Add new Principle of Charity filter tool that interprets arguments charitably before critiquing, filtering issues that dissolve under generous interpretation - Refactor pipeline to run filters dynamically based on filterChain order instead of hardcoded execution sequence - Update PipelineView to display all filter stages dynamically with proper step numbering - Add filter type validation and settings UI in FilterChainEditor The filter order (charity → supported-elsewhere) is intentional: 1. Charity removes invalid critiques that aren't real problems 2. Supported-elsewhere then checks if remaining valid critiques are addressed in the document Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add expandable document rows showing all evaluation versions - Allow selecting specific version per document via radio buttons - Auto-select latest version when checking document without expanding - Add /api/monitor/lab/corpus/versions endpoint for fetching versions - Support beforeDate filter in baselines API for cutoff-based selection - Add dev-env.sh psql command for database access - Document dev scripts in CLAUDE.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… filters - Add modelConfigResolver.ts for unified model configuration across extractors, judge, and filters - Add reasoningBudget.ts for dynamic reasoning token budget calculation - Add provider routing support to principle-of-charity and supported-elsewhere filters - Consolidate single/multi-extractor paths into unified extraction flow - Fix verbose logging in smart-text-searcher (was logging full document content) - Add Claude thinking budget calculation from reasoning effort levels - Refactor UI components to use shared ModelConfigurator and ProviderSelector - Update CLAUDE.md with database access restrictions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add actualApiParams and responseMetrics to judge output - Capture judge telemetry in fallacy-check plugin - Add telemetry fields to ExtractionPhaseTelemetry and StageMetrics - Fix judge config to properly pass reasoning and provider from profile - Update frontend types to support new telemetry fields Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ActualApiParams and ApiResponseMetrics types to both filter outputs - Update supported-elsewhere-filter to capture and return API telemetry - Update principle-of-charity-filter to capture and return API telemetry - Update endStage calls to pass filter telemetry (actualApiParams, responseMetrics, unifiedUsage) - Update PipelineView UI to display filter and judge telemetry details Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create types/common.ts with shared types: - ReasoningEffort, ReasoningConfig, ProviderPreferences - ActualApiParams, ApiResponseMetrics - EFFORT_TO_BUDGET_TOKENS constant - effortToBudgetTokens utility function - Update all files to import from shared types: - extraction/types.ts - profile-types.ts - telemetry/types.ts - fallacy-judge/types.ts - fallacy-extractor/types.ts - principle-of-charity-filter/types.ts - supported-elsewhere-filter/types.ts - openrouter.ts - Remove duplicate effortToBudgetTokens from filter implementations This eliminates ~6 duplicate definitions of each type across the codebase. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… tools Converted debug output in principle-of-charity-filter and supported-elsewhere-filter from console.log to context.logger.debug for proper log level control. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…lit large files
Code review improvements:
- Fix getDefaultConfig() to use new filterChain array format (was using old
{ filters: [...] } format which caused config parsing issues)
- Consolidate JACCARD_SIMILARITY_THRESHOLD constant in common.ts (was
duplicated in dedup.ts and multiExtractor.ts)
- Replace console.log/warn with structured logger calls in:
- reasoningBudget.ts
- profile-loader.ts (also removed DEBUG logs)
- PipelineTelemetry.ts (refactored logSummary to use structured logging)
- Create shared LLM filter utility (tools/shared/llm-filter-utils.ts):
- callLLMFilter<T>() abstracts Claude vs OpenRouter differences
- buildThinkingConfig(), buildOpenRouterReasoning() helpers
- truncateDocumentForContext() utility
- Reduces code duplication between filter tools
- Split PipelineView.tsx (871 → 394 lines, ~55% reduction):
- pipelineUtils.ts: Format helpers
- ItemCards.tsx: FilteredItemCard, CommentCard components
- ExtractorCards.tsx: ExtractorCard, DeduplicationCard components
- Fix duplicated ReasoningEffortLevel type in multiExtractor.ts
(now uses ReasoningEffort from common.ts)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
The client index.ts was pulling in Node.js-only modules (async_hooks) through this import chain: index.ts → openrouter.ts → reasoningBudget.ts → logger.ts → jobContext.ts Fix: - Create openrouter-types.ts with client-safe types/constants only - Create reasoningBudget-client.ts with sync functions only - Move ToolConfig/ToolContext types to tools/base/types.ts - Export async API functions from server.ts instead of index.ts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace console.log with logger.debug for OpenRouter API calls - Remove fire emoji debug output from fallacy extractor - Include full response body in OpenRouter error messages for better debugging of rate limits (429) and other errors Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Prevents false positives where the model thinks recent dates are "in the future" due to training cutoff. All system prompts now include the current date as a reference point. Affects: extractor, judge, filters, and review stages. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new "All Evals" tab to Lab UI showing recent user-facing evaluations with their pipeline telemetry (not just validation runs) - Add API endpoint /api/monitor/lab/evaluations to fetch evaluation versions with pipelineTelemetry data - Track items that pass through filters (not just filtered out items): - Add PassedItemRecord type to telemetry - Record passed items in principle-of-charity and supported-elsewhere filters - Display passed items in PipelineView (collapsed by default) - New components: AllEvaluationsList, PassedItemCard, useAllEvaluations hook Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add whitespace-nowrap to prevent text wrapping - Reduce padding and gap for better fit - Shorten 'All Evals' to 'Evals' - Add flex-shrink-0 to icons Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
No description provided.