An AI-powered document evaluation platform that provides intelligent analysis, critiques, and insights on written content. RoastMyPost enables users to get structured feedback from specialized AI agents that examine documents from different perspectives.
- Multi-Agent Evaluations: Deploy various AI agents to analyze documents from different angles
- Interactive Annotations: AI agents provide inline comments and highlights with importance ratings and grades
- Document Versioning: Track changes and evaluations across multiple versions of documents
- Batch Processing: Queue and process multiple evaluations asynchronously with retry logic
- Cost Tracking: Monitor AI API usage and costs per evaluation
- Import from Multiple Sources: Automatically import content from LessWrong, EA Forum, and general web pages
- Export Capabilities: Export evaluations as XML for further processing
- MCP Integration: Fast database access via Model Context Protocol for Claude Code users
- Frontend: Next.js 15.3.2 with App Router, React 19, TypeScript
- Database: PostgreSQL with Prisma ORM v6.13.0
- Authentication: NextAuth.js 5.0.0-beta.28
- UI Components: Tailwind CSS, Heroicons, Slate.js editor
- AI Integration: Anthropic Claude API + OpenRouter
- Background Jobs: Custom async job processing with exponential backoff
- Content Extraction: JSDOM, Turndown, Metascraper suite
- Monorepo: pnpm workspaces with Turborepo
- Node.js 18+
- pnpm 9+ (install with
npm install -g pnpm) - PostgreSQL database
- API keys for Anthropic Claude and/or OpenRouter
- Resend API key for email authentication (optional)
- Clone the repository:
git clone https://github.com/quantified-uncertainty/roast-my-post.git
cd roast-my-post- Install dependencies:
pnpm install- Set up environment variables:
cp .env.example .envEdit .env with your configuration:
DATABASE_URL="postgresql://user:password@localhost:5432/roast_my_post"
NEXTAUTH_URL="http://localhost:3000"
AUTH_SECRET="generate-a-secure-random-string"
ANTHROPIC_API_KEY="your-anthropic-key"
HELICONE_API_KEY="your-helicone-key-optional"
OPENROUTER_API_KEY="your-openrouter-key"
EMAIL_FROM="[email protected]"
AUTH_RESEND_KEY="your-resend-api-key"
Optional: Helicone Integration
To enable monitoring of Anthropic API usage through Helicone:
- Create a Helicone account
- Add your Helicone API key to
HELICONE_API_KEYin.env - All Anthropic API calls will automatically route through Helicone for monitoring and analytics
Optional: Helicone Prompt Caching
To enable prompt caching for improved performance and cost savings:
HELICONE_CACHE_ENABLED=true # Enable prompt caching
HELICONE_CACHE_MAX_AGE=3600 # Cache duration in seconds (1 hour)
HELICONE_CACHE_BUCKET_MAX_SIZE=1000 # Max cache entries per bucketPrompt caching can significantly reduce API costs for repeated similar requests.
- Set up the database and system agents:
pnpm run db:setupThis will push the database schema and synchronize system-managed agents (spelling/grammar checker, math checker, fact checker).
- Start the development server:
pnpm run devVisit http://localhost:3000 to see the application.
pnpm run dev- Start development serverpnpm run build- Build for productionpnpm run typecheck- Run TypeScript type checkingpnpm run lint- Run ESLintpnpm run test:ci- Run tests suitable for CI (no external dependencies)pnpm run test:fast- Run unit and integration testspnpm run test:e2e- Run end-to-end tests (requires API keys)pnpm run db:studio- Open Prisma Studio for database managementpnpm run process-jobs- Process evaluation jobs manually
For package-specific commands:
pnpm --filter @roast/web <command>- Run commands in the web apppnpm --filter @roast/db <command>- Run commands in the database packagepnpm --filter @roast/ai <command>- Run commands in the AI packagepnpm --filter @roast/mcp-server <command>- Run commands in the MCP server
apps/
├── web/ # Next.js web application
│ ├── src/
│ │ ├── app/ # Next.js app router pages
│ │ ├── components/ # React components
│ │ ├── lib/ # Core business logic
│ │ ├── types/ # TypeScript type definitions
│ │ ├── utils/ # Utility functions
│ │ └── scripts/ # CLI scripts for operations
│ └── config/ # App-specific configuration
└── mcp-server/ # Model Context Protocol server
internal-packages/
├── ai/ # Shared AI utilities (@roast/ai)
│ ├── src/
│ │ ├── claude/ # Claude API wrapper
│ │ ├── helicone/ # Cost tracking integration
│ │ ├── analysis-plugins/ # Plugin system for document analysis
│ │ ├── document-analysis/ # Analysis workflows
│ │ ├── tools/ # AI-powered tools (spell check, math check, etc.)
│ │ └── shared/ # Shared types and utilities
│ └── package.json
└── db/ # Shared database package (@roast/db)
├── prisma/
│ ├── schema.prisma # Database schema
│ └── migrations/ # Database migrations
└── src/ # Prisma client exports
dev/ # Development scripts and tools
docs/ # Project documentation
claude/ # Claude Code specific tools and analysis
This project includes an MCP (Model Context Protocol) server that provides instant database access without writing scripts. This is 10-20x faster than creating TypeScript files for data queries.
To set up the MCP server:
pnpm --filter @roast/mcp-server run setupThen restart Claude Code. See /apps/mcp-server/README.md for detailed setup instructions.
AI evaluators that can be configured for various purposes:
- Critical analysis and quality assessment
- Constructive suggestions and improvements
- Context and supplementary information
- Clarification and concept explanation
- Custom evaluation criteria based on specific needs
Content items stored with versioning support. Each document can have multiple versions, and each version can be evaluated by multiple agents.
AI-generated analysis containing:
- Summary and detailed analysis
- Inline comments with highlights
- Importance ratings and grades
- Self-critique from the agent
Asynchronous processing queue for evaluations with:
- Automatic retry logic with exponential backoff
- Cost tracking per job
- Detailed logging for debugging
/api/agents- Agent management/api/documents- Document operations/api/jobs- Job processing/api/monitor- System monitoring (admin only)/api/import- Article import from URLs
The API now follows a clean, intuitive structure that matches web URLs:
Document Operations:
GET /api/docs/{docId}- Get document with all evaluations (public)PUT /api/docs/{docId}- Update document (requires auth + ownership)
Evaluation Operations:
GET /api/docs/{docId}/evals/{agentId}- Get specific evaluation with full details (public)POST /api/docs/{docId}/evals/{agentId}- Create new evaluation (requires auth + ownership)POST /api/docs/{docId}/evals/{agentId}/rerun- Re-run existing evaluation (requires auth + ownership)GET /api/documents/{docId}/evaluations- List all evaluations for document (public)
Security Features:
- Authentication: Required for all write operations (PUT, POST)
- Ownership verification: Document owner-only for modifications
- Input validation: Zod schemas for request body validation
- Security headers: X-Content-Type-Options, X-Frame-Options, X-XSS-Protection
- Consistent error handling: Standardized error responses
Key Benefits:
- Natural identifiers: Use
documentId + agentIdinstead of evaluation IDs - Perfect URL alignment with web interface (
/docs/{docId}/evals/{agentId}) - Complete evaluation data including comments, highlights, and job information
- Consistent security patterns with authentication and validation
The application uses NextAuth.js with support for:
- Email magic links via Resend
- API key authentication for programmatic access
The test suite is organized by dependency requirements:
- Unit tests (
*.test.ts): Fast, no external dependencies - Integration tests (
*.integration.test.ts): Database and internal APIs - E2E tests (
*.e2e.test.ts): External APIs (Firecrawl, LessWrong) - LLM tests (
*.llm.test.ts): AI API calls (expensive)
Run tests with:
pnpm run test:fast # Unit + integration
pnpm run test:ci # CI-safe tests only
pnpm run test:e2e # End-to-end testsAlways use the safe wrapper scripts for database operations:
pnpm run db:push # Safe schema push
pnpm run db:migrate # Safe migration./scripts/backup-database.shThe database includes optimized indexes for:
- Full-text search on document content
- Fast metadata search on titles, authors, platforms
- Efficient evaluation queries
Comprehensive documentation is available in the /docs directory:
For Claude-specific development workflows, see CLAUDE.md.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please ensure:
- All tests pass (
pnpm run test:fast) - Type checking passes (
pnpm run typecheck) - Code is linted (
pnpm run lint)
- Branch from
mainfor any new work. - Open a pull request (PR) to
main:- CI runs build and tests on the PR.
- Merge to
main:- Builds and deploys to staging at https://staging.roastmypost.org.
- Merge to
prod:- Builds and deploys to production at https://roastmypost.org.
- Authentication required for all data-modifying operations
- Rate limiting on API endpoints
- Input validation with Zod schemas
- Admin role for sensitive operations
See Security Documentation for detailed security practices.
Built with excellent open source tools including Next.js, Prisma, Tailwind CSS, and Slate.js.