AI-search-week-2-tavilly-alternative-research

[kien-ship-it]

Tavily Alternatives: Research & Recommendation

Our Current Setup

Our web-search.ts is cleanly isolated — it exports executeSearch(subQueries) and internally calls callTavily(query). The output is normalized to RawSearchResult[]. This is a great position to be in because swapping the provider only requires replacing the inner callTavily function. The executeSearch orchestration (retry, dedup, logging) stays untouched.

---

The Contenders

Here's what was found across benchmarks, pricing, and practical fit for your news-focused trend search use case:

Provider	Cost per 1K requests	Free Tier	News Support	Latency	Quality (benchmark)	Self-hosted?
Tavily (current)	~$5/1K (credit-based)	1,000 credits/mo	Native topic: "news"	~3s	93.3% accuracy	No
Serper.dev	$0.30/1K	2,500 req/mo	Google News tab	~1.8s	~93.5%	No
Brave Search API	$5/1K (Base AI)	2,000 req/mo (non-commercial)	Has news endpoint	~669ms	Highest agent score (14.89)	No
DuckDuckGo (ddgs)	Free (scraping)	Unlimited (no API key)	searchNews() method	Variable	Unranked, lower quality	No
Exa.ai	~$5-10/1K (credit-based)	$10 free credits	Semantic, not news-specific	<1s simple, 3-8s research	94.9% accuracy	No
SearXNG	Free (self-hosted)	Unlimited	Aggregates multiple engines	Depends on infra	Depends on upstream engines	Yes

Sources: AIMultiple benchmark, HumAI comparison, Scrapfly pricing analysis. Content was rephrased for compliance with licensing restrictions.

---

Detailed Evaluation

1. Serper.dev — Best bang for buck

• 16x cheaper than Tavily ($0.30 vs ~$5 per 1K)
• Scrapes Google Search results, including Google News
• 2,500 free requests/month, then pay-as-you-go
• Fast (~1.8s), simple REST API, returns structured JSON
• No content extraction built-in (just titles, snippets, links) — but your synthesizer LLM handles that anyway
• Integration effort: ~1 hour. Replace callTavily with callSerper. The response shape maps cleanly to RawSearchResult

2. Brave Search API — Best quality

• Topped the AIMultiple agent benchmark, consistently outperformed Tavily
• Fastest latency at ~669ms
• Has a dedicated news search endpoint
• Owns its own index (not scraping Google)
• Free tier is 2,000 req/mo but non-commercial. Paid is $5/1K — same as Tavily
• Integration effort: ~1 hour. Clean REST API, similar response structure

3. DuckDuckGo (via ddgs npm package) — Free but fragile

• Completely free, no API key needed
• The ddgs package uses browser automation (Puppeteer-like) under the hood — it scrapes DDG
• Has a searchNews() method that fits your use case
• Risks: rate limiting, no SLA, no guaranteed uptime, scraping can break anytime
• No relevance scoring in results
• Integration effort: ~2 hours. Need to install ddgs, adapt the response shape, handle browser lifecycle

4. SearXNG (self-hosted) — Free but ops overhead

• Open-source meta-search engine, aggregates Google/Bing/DDG/etc.
• Completely free, no API keys for the meta-engine itself
• Has a JSON API endpoint you can hit from Node.js
• Requires hosting infrastructure (Docker container)
• You already have Docker in your project, so spinning it up is feasible
• Risk: upstream engines can rate-limit or block your instance
• Integration effort: ~4-6 hours. Docker setup + new adapter + testing reliability

5. Exa.ai — Best accuracy, but not cheaper

• Highest accuracy (94.9% on SimpleQA)
• Semantic/neural search — great for research queries
• Not specifically news-focused (no topic: "news" equivalent)
• Pricing is credit-based, roughly comparable to Tavily
• Integration effort: ~1-2 hours

---

Initial Recommendation

Go with Serper.dev as primary, keep Tavily as fallback.

Here's why:

• 16x cheaper for the same (or better) quality
• Google News results are exactly what your trend search needs
• Minimal integration effort — your architecture already isolates the search provider
• The free tier (2,500/mo) covers development and light production use
• If Serper ever has issues, Tavily is already wired up as a fallback

The integration approach suggested is a provider pattern — abstract the search call behind an interface so you can swap providers via env config:

// src/server/trend-search/search-provider.ts

type SearchProvider = "tavily" | "serper" | "brave" | "duckduckgo";

async function callSearchProvider(query: string): Promise<RawSearchResult[]> {
  const provider = env.server.SEARCH_PROVIDER ?? "tavily";
  
  switch (provider) {
    case "serper":  return callSerper(query);
    case "brave":   return callBrave(query);
    case "duckduckgo": return callDuckDuckGo(query);
    case "tavily":
    default:        return callTavily(query);
  }
}

Then executeSearch calls callSearchProvider instead of callTavily directly. Each provider adapter normalizes its response to RawSearchResult[]. Zero changes needed in the pipeline, synthesizer, or Inngest job.

---

What we should avoid

• DuckDuckGo as primary — no SLA, scraping-based, can break without notice. Fine for personal projects, risky for a product.
• SearXNG for now — the ops overhead of self-hosting and maintaining it isn't worth it unless you're doing very high volume (10K+ searches/month). Revisit if costs become a real problem.
• Exa — great accuracy but not cheaper than Tavily and not news-focused. Doesn't solve your cost problem.

---

Should we do it?

Yes, but incrementally. The development debt is minimal (~2-3 hours for the provider abstraction + Serper adapter). Our web-search.ts is already well-structured for this. The cost savings are significant if you scale beyond the free tiers. And having a fallback provider is just good engineering.

[kien-ship-it]

Task overview

• Research Tavilly alternatives
• Integrate into the current trend search tool as a second search provider alternative to Tavilly

[kien-ship-it]

Design Document: Serper Dual-Channel Search Integration

(Recommended but not final, please conduct independent research and use best judgment)

Overview

This module introduces a provider abstraction layer into the existing web search executor (#[[file:src/server/trend-search/web-search.ts]]), adds Serper.dev as a second search provider, and supports four runtime strategies: tavily, serper, fallback, and parallel.

The change is scoped entirely to the web search layer. No changes to the query planner, synthesizer, Inngest pipeline, API routes, DB schema, or types.

                          ┌─────────────────┐
                          │ executeSearch() │
                          │ (orchestrator)  │
                          └────────┬────────┘
                                   │
                          ┌────────▼────────┐
                          │ Provider Strategy│
                          │  (env-driven)   │
                          └────────┬────────┘
                    ┌──────────────┼──────────────┐
                    ▼              ▼               ▼
             ┌──────────┐  ┌──────────┐   ┌──────────────┐
             │  Tavily  │  │  Serper  │   │  Both (merge)│
             │  Adapter │  │  Adapter │   │  + dedup     │
             └──────────┘  └──────────┘   └──────────────┘
                    │              │               │
                    └──────────────┴───────────────┘
                                   │
                          ┌────────▼────────┐
                          │ RawSearchResult[]│
                          └─────────────────┘

Architecture

The refactor splits web-search.ts into focused modules:

src/server/trend-search/
├── types.ts                    # unchanged
├── web-search.ts               # refactored: executeSearch + strategy logic
├── providers/
│   ├── types.ts                # SearchProviderFn type, SearchExecutionResult
│   ├── registry.ts             # provider name → adapter mapping
│   ├── tavily.ts               # callTavily() extracted from current web-search.ts
│   └── serper.ts               # callSerper() — new
├── query-planner.ts            # unchanged
├── synthesizer.ts              # unchanged
├── run.ts                      # minor: log providerUsed
├── db.ts                       # unchanged
└── index.ts                    # unchanged

Components and Interfaces

1. Provider Types (src/server/trend-search/providers/types.ts)

import type { RawSearchResult } from "~/server/trend-search/types";

/** A search provider function: takes a query string, returns normalized results. */
export type SearchProviderFn = (query: string) => Promise<RawSearchResult[]>;

/** The supported provider strategy names. */
export type ProviderStrategy = "tavily" | "serper" | "fallback" | "parallel";

/** Extended result from executeSearch, includes which provider was used. */
export interface SearchExecutionResult {
  results: RawSearchResult[];
  providerUsed: string;
}

2. Tavily Adapter (src/server/trend-search/providers/tavily.ts)

Extracted directly from the current callTavily() in web-search.ts. No logic changes.

export async function callTavily(query: string): Promise<RawSearchResult[]>;

• Reads TAVILY_API_KEY from env
• POSTs to https://api.tavily.com/search with search_depth: "advanced", topic: "news", max_results: 10
• Returns RawSearchResult[] or empty array if key not set
• Throws on non-2xx (triggers retry in orchestrator)

3. Serper Adapter (src/server/trend-search/providers/serper.ts)

New module. Calls the Serper.dev Google News endpoint.

export async function callSerper(query: string): Promise<RawSearchResult[]>;

Implementation:

• Reads SERPER_API_KEY from env
• POSTs to https://google.serper.dev/news with:

  {
    "q": "<query>",
    "num": 10
  }

• Headers: X-API-KEY: <key>, Content-Type: application/json
• Serper news response shape (subset we use):

  interface SerperNewsItem {
    title?: string;
    link?: string;
    snippet?: string;
    date?: string;
    source?: string;
    position?: number;
  }
  interface SerperNewsResponse {
    news?: SerperNewsItem[];
  }

• Normalization to RawSearchResult:
  • url ← link
  • title ← title ?? "Untitled"
  • content ← snippet ?? ""
  • score ← 1 - ((position ?? 0) / totalResults) — positional relevance since Serper doesn't provide a score
  • publishedDate ← date (if present)
• Returns empty array + warning if SERPER_API_KEY not set
• Throws on non-2xx responses

4. Provider Registry (src/server/trend-search/providers/registry.ts)

import type { SearchProviderFn } from "./types";
import { callTavily } from "./tavily";
import { callSerper } from "./serper";

export const providerRegistry: Record<string, SearchProviderFn> = {
  tavily: callTavily,
  serper: callSerper,
};

Simple lookup map. New providers are added here.

5. Refactored executeSearch (src/server/trend-search/web-search.ts)

The public API changes minimally:

export async function executeSearch(
  subQueries: PlannedQuery[],
  strategyOverride?: ProviderStrategy,
): Promise<SearchExecutionResult>;

Strategy resolution:

1. Use strategyOverride if provided
2. Else read SEARCH_PROVIDER from env
3. Else default to "tavily"
4. If strategy requires Serper but SERPER_API_KEY is missing, downgrade to "tavily" with warning

Strategy implementations:

"tavily" / "serper" (single provider)

• Resolve adapter from registry
• Run existing loop: for each sub-query, retry up to 2 times, dedup by URL
• Return { results, providerUsed: "tavily" | "serper" }

"fallback"

• Run all sub-queries against primary (Serper)
• If combined results are empty (all sub-queries failed or returned nothing), log warning and re-run all sub-queries against secondary (Tavily)
• Return { results, providerUsed: "serper" | "tavily (fallback)" }

"parallel"

• Run all sub-queries against both providers concurrently (Promise.all)
• Merge results, dedup by URL — on collision, keep the result with the higher score
• Return { results, providerUsed: "tavily+serper" }

The internal executeWithProvider helper encapsulates the per-provider sub-query loop (retry + dedup), reused across all strategies.

6. Environment Changes (src/env.ts)

Add to serverSchema:

SERPER_API_KEY: optionalString(),
SEARCH_PROVIDER: z.enum(["tavily", "serper", "fallback", "parallel"]).default("tavily").optional(),

Add to parseServerEnv():

SERPER_API_KEY: process.env.SERPER_API_KEY,
SEARCH_PROVIDER: process.env.SEARCH_PROVIDER,

7. Pipeline Integration (src/server/trend-search/run.ts)

Minimal change — runTrendSearch receives SearchExecutionResult instead of RawSearchResult[] and logs providerUsed:

// Step 2: Execute web searches
await options.onStageChange?.("searching");
const { results: rawResults, providerUsed } = await executeSearch(plannedQueries);
console.log(`[trend-search] Search provider used: ${providerUsed}`);

No changes to the return type or public interface of runTrendSearch.

Correctness Properties

Property 13: Provider adapter output normalization

For any valid query string, every Search_Provider adapter SHALL return an array where each element conforms to the RawSearchResult interface (non-empty url, title as string, content as string, score as number).

Validates: Requirements 1.4, 2.3

Property 14: Strategy fallback triggers on total failure

For any set of sub-queries where the primary provider returns zero results for every sub-query, the "fallback" strategy SHALL invoke the secondary provider for all sub-queries.

Validates: Requirements 3.3

Property 15: Parallel strategy deduplication

For any two result sets from different providers containing overlapping URLs, the "parallel" strategy SHALL produce a merged set with no duplicate URLs, keeping the higher-scored result for each URL.

Validates: Requirements 3.4

Property 16: Default strategy backward compatibility

For any execution where SEARCH_PROVIDER is unset and no strategyOverride is provided, executeSearch SHALL invoke only the Tavily adapter.

Validates: Requirements 6.1

Property 17: Missing key graceful downgrade

For any strategy requiring Serper where SERPER_API_KEY is not set, the system SHALL downgrade to "tavily" strategy without throwing.

Validates: Requirements 4.3

Error Handling

Error Scenario	Handling Strategy
SERPER_API_KEY not set, strategy requires Serper	Log warning, downgrade to "tavily" strategy
Serper API returns non-2xx	Throw error (triggers existing retry logic in executeSearch)
Serper API returns 0 news results	Return empty array, log warning, continue with other sub-queries
Serper response missing news array	Return empty array (defensive parsing)
Fallback: primary fails completely	Log warning, re-execute all sub-queries with secondary provider
Parallel: one provider fails	Use results from the other provider (partial success is acceptable)
Both providers fail in parallel	Return empty results (synthesizer handles this with placeholder padding)
Invalid SEARCH_PROVIDER value	Zod validation catches at startup, defaults to "tavily"

Serper.dev API Reference

Endpoint: POST https://google.serper.dev/news

Headers:

• X-API-KEY: <SERPER_API_KEY>
• Content-Type: application/json

Request body:

{
  "q": "AI trends in fashion industry 2026",
  "num": 10,
  "gl": "us",
  "hl": "en"
}

Response shape (news results):

{
  "news": [
    {
      "title": "Fashion Industry Embraces AI for Trend Forecasting",
      "link": "https://example.com/article",
      "snippet": "Major fashion brands are increasingly turning to AI...",
      "date": "2 hours ago",
      "source": "Fashion Weekly",
      "imageUrl": "https://example.com/image.jpg",
      "position": 1
    }
  ],
  "searchParameters": {
    "q": "AI trends in fashion industry 2026",
    "type": "news",
    "num": 10
  }
}

Pricing: ~$0.30 per 1,000 requests. Free tier: 2,500 requests/month.

Comparison with Tavily:

• Tavily: ~$5/1K requests, built-in content extraction, topic: "news" filter, relevance scoring
• Serper: ~$0.30/1K requests, Google News results, positional ranking only, no content extraction (snippets only)

The content quality difference is mitigated by the LLM synthesizer downstream, which processes raw snippets into structured summaries regardless of source.

Testing Strategy

Unit Tests

• Serper adapter: mock fetch, verify request shape (URL, headers, body), verify normalization to RawSearchResult
• Serper adapter: missing API key returns empty array
• Serper adapter: non-2xx throws error
• Strategy resolution: verify each strategy value routes to correct provider(s)
• Fallback: mock primary to fail, verify secondary is called
• Parallel: mock both providers, verify merge and dedup logic

Property-Based Tests

• Property 13: Generate random Serper response shapes, verify all outputs conform to RawSearchResult
• Property 14: Generate random sub-query lists, mock primary to return empty, verify fallback invocation
• Property 15: Generate two random result sets with overlapping URLs, verify dedup keeps higher score
• Property 16: Verify default behavior matches Tavily-only execution
• Property 17: Unset Serper key, verify graceful downgrade for all Serper-dependent strategies

[kien-ship-it]

Suggested Implementation Plan: Serper Dual-Channel Search Integration

Overview

Single-track implementation. All changes are scoped to the web search layer — no DB, Inngest, API route, or synthesizer changes needed. Estimated effort: ~4-6 hours.

Tasks

• 1. Provider abstraction and Tavily extraction

  • 1.1 Create provider types

    • Create src/server/trend-search/providers/types.ts
    • Define SearchProviderFn type: (query: string) => Promise<RawSearchResult[]>
    • Define ProviderStrategy type: "tavily" | "serper" | "fallback" | "parallel"
    • Define SearchExecutionResult interface: { results: RawSearchResult[]; providerUsed: string }
    • Requirements: 1.1, 1.4

  • 1.2 Extract Tavily adapter

    • Create src/server/trend-search/providers/tavily.ts
    • Move callTavily(), TavilyResultItem, TavilySearchResponse, and the Tavily URL constant from web-search.ts into this file
    • Export callTavily as a named export
    • Verify it compiles and the existing web-search.ts can import it
    • No logic changes — pure extraction
    • Requirements: 1.2, 6.3

  • 1.3 Create provider registry

    • Create src/server/trend-search/providers/registry.ts
    • Import callTavily from ./tavily
    • Placeholder import for callSerper (to be implemented in task 2.1)
    • Export providerRegistry: Record<string, SearchProviderFn>
    • Requirements: 1.2

• 2. Serper adapter

  • 2.1 Implement Serper adapter

    • Create src/server/trend-search/providers/serper.ts
    • Implement callSerper(query: string): Promise<RawSearchResult[]>
    • POST to https://google.serper.dev/news with X-API-KEY header and { q, num: 10 } body
    • Define SerperNewsItem and SerperNewsResponse interfaces
    • Normalize response: link → url, snippet → content, positional score calculation
    • Return empty array + warning if SERPER_API_KEY not set
    • Throw on non-2xx responses
    • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5

  • 2.2 Register Serper in provider registry

    • Update src/server/trend-search/providers/registry.ts to import and register callSerper
    • Requirements: 1.2

  • 2.3 Add environment variables

    • Add SERPER_API_KEY: optionalString() to serverSchema in src/env.ts
    • Add SEARCH_PROVIDER as optional enum ("tavily" | "serper" | "fallback" | "parallel") defaulting to "tavily" in serverSchema
    • Add both to the parseServerEnv() values object
    • Add SERPER_API_KEY and SEARCH_PROVIDER entries to .env (commented out, with instructions)
    • Requirements: 4.1, 4.2

• 3. Strategy logic and executeSearch refactor

  • 3.1 Refactor executeSearch with provider strategy support

    • Refactor src/server/trend-search/web-search.ts:
      • Remove callTavily (now in providers/tavily.ts)
      • Keep normalizeUrl and MAX_RETRIES constant
      • Extract inner loop into executeWithProvider(subQueries, providerFn) helper that handles retry + dedup for a single provider
      • Update executeSearch signature: add optional strategyOverride?: ProviderStrategy
      • Return SearchExecutionResult instead of RawSearchResult[]
    • Implement strategy resolution:
      1. strategyOverride → env SEARCH_PROVIDER → "tavily"
      2. If strategy needs Serper and key missing, downgrade to "tavily" with warning
    • Implement "tavily" and "serper" strategies: single provider via executeWithProvider
    • Implement "fallback" strategy: try primary (Serper), if all empty → re-run with secondary (Tavily)
    • Implement "parallel" strategy: Promise.all both providers, merge + dedup (keep higher score)
    • Requirements: 1.3, 1.5, 3.1, 3.2, 3.3, 3.4, 3.5, 4.3

  • 3.2 Update pipeline to use SearchExecutionResult

    • Update src/server/trend-search/run.ts:
      • Destructure { results: rawResults, providerUsed } from executeSearch
      • Log providerUsed
    • Update src/server/inngest/functions/trendSearch.ts if it calls executeSearch directly (verify — it likely calls runTrendSearch which handles this)
    • Requirements: 5.1, 5.2, 5.3

• 4. Testing

  • 4.1 Unit tests for Serper adapter

    • Create __tests__/api/trendSearch/serper-adapter.test.ts
    • Test: valid response normalizes correctly to RawSearchResult[]
    • Test: missing SERPER_API_KEY returns empty array
    • Test: non-2xx response throws error
    • Test: empty news array returns empty results
    • Test: positional score calculation is correct
    • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5

  • 4.2 Unit tests for strategy logic

    • Create __tests__/api/trendSearch/search-strategy.test.ts
    • Test: default strategy (no env) uses Tavily only
    • Test: "serper" strategy uses Serper only
    • Test: "fallback" strategy tries Serper first, falls back to Tavily on total failure
    • Test: "fallback" strategy does NOT fall back when Serper returns results
    • Test: "parallel" strategy calls both providers and merges results
    • Test: "parallel" strategy deduplicates by URL, keeping higher score
    • Test: missing Serper key with "serper" strategy downgrades to "tavily"
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 4.3, 6.1, 6.2

  • [ ]* 4.3 Property-based tests

    • Create __tests__/api/trendSearch/search-provider.pbt.test.ts
    • Property 13: Generate random Serper-shaped responses, verify all outputs conform to RawSearchResult
    • Property 14: Generate random sub-query lists, mock primary to return empty, verify fallback invocation count
    • Property 15: Generate two random result sets with overlapping URLs, verify dedup keeps higher score and no duplicate URLs
    • Property 16: Verify default (no env) matches Tavily-only behavior
    • Property 17: Unset Serper key, verify graceful downgrade for all Serper-dependent strategies
    • Requirements: All

  • 4.4 Verify existing tests still pass

    • Run full test suite, confirm no regressions
    • Existing web-search.pbt.test.ts and inngest-completion.pbt.test.ts should pass without changes
    • Requirements: 6.4, 6.5

• 5. Manual validation

  • 5.1 End-to-end smoke test with Serper

    • Set SERPER_API_KEY and SEARCH_PROVIDER=serper in .env
    • Run scripts/test-trend-search.ts and verify results come back from Google News
    • Compare result quality with Tavily results for the same query
    • Requirements: 2.1, 3.2

  • 5.2 Fallback smoke test

    • Set SEARCH_PROVIDER=fallback with a valid SERPER_API_KEY
    • Run a search, verify Serper is used (check logs)
    • Temporarily invalidate SERPER_API_KEY, run again, verify Tavily fallback is triggered (check logs)
    • Requirements: 3.3

Notes

• Tasks marked with * are optional for MVP
• Task 1 (extraction) is the foundation — do it first and verify nothing breaks
• Task 2 (Serper adapter) and Task 3 (strategy logic) can be developed in sequence
• The src/app/api/agents/documentQ&A/services/tavilySearch.ts file also uses Tavily directly but is out of scope for this spec — it serves a different use case (document Q&A) and can be migrated to the provider pattern later if desired
• The src/lib/tools/trend-search/web-search.ts appears to be a duplicate of src/server/trend-search/web-search.ts — verify which is canonical and clean up if needed