OpenPipe

Turn expensive prompts into cheap fine-tuned models.

Hosted App - Running Locally - Experiments

Use powerful but expensive LLMs to fine-tune smaller and cheaper models suited to your exact needs. Evaluate model and prompt combinations in the playground. Query your past requests and export optimized training data. Try it out at https://app.openpipe.ai or run it locally.

Features

• Easy integration with OpenAI's SDK in both Python and TypeScript.
  • Python SDK
  • TypeScript SDK
• Fine-tune GPT 3.5, Mistral, and Llama 2 models, host on-platform or download the weights.
  • Model output is OpenAI-compatible.
  • Switching from GPT 4 to a fine-tuned Mistral model only requires changing the model name.
• Swiftly query logs using intuitive built-in filters.
• Import datasets in OpenAI-compatible JSONL files
• Prune large chunks of duplicate text like system prompts.
• Compare output accuracy against base models like gpt-3.5-turbo

Supported Base Models

To fine-tune Mistral and Llama 2 models in the hosted app, join the waitlist.

• mistral-7b
• llama-2-7b
• llama-2-13b
• gpt-3.5-turbo

Documentation

• See docs

Running Locally

1. Install Postgresql.
2. Install NodeJS 20 (earlier versions will very likely work but aren't tested).
3. Install pnpm: npm i -g pnpm
4. Clone this repository: git clone https://github.com/openpipe/openpipe
5. Install the dependencies: cd openpipe && pnpm install
6. Create a .env file (cp .env.example .env) and enter your OPENAI_API_KEY.
7. Update DATABASE_URL if necessary to point to your Postgres instance and run pnpm prisma migrate dev to create the database.
8. Create a GitHub OAuth App, set the callback URL to <your local instance>/api/auth/callback/github, e.g. http://localhost:3000/api/auth/callback/github.
9. Update the GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET values from the Github OAuth app (Note: a PR to make auth optional when running locally would be a great contribution!).
10. Start the app: pnpm dev.
11. Navigate to http://localhost:3000

Testing Locally

1. Copy your .env file to .env.test.
2. Update the DATABASE_URL to have a different database name than your development one
3. Run DATABASE_URL=[your new datatase url] pnpm prisma migrate dev --skip-seed --skip-generate
4. Run pnpm test