feat: migrate scrape proxy to chittyagent-scrape SDK worker

[chitcommit]

Summary

• Migrates scrape job proxy from ChittyRouter ScrapeAgent (DO) to the new chittyagent-scrape worker in the chittyidentity SDK monorepo
• Adds CHITTYAGENT_SCRAPE_URL env var for direct agent routing
• Falls back to ChittyRouter paths when agent URL is not configured (zero-downtime migration)

Changes

• src/index.ts — Added CHITTYAGENT_SCRAPE_URL to Env type
• src/lib/integrations.ts — Added scrapePost/scrapeGet helpers that try chittyagent-scrape first, fall back to ChittyRouter

Test plan

• tsc --noEmit passes
• vitest 15/15 tests pass
• Deploy with CHITTYAGENT_SCRAPE_URL unset — verify fallback works
• Set CHITTYAGENT_SCRAPE_URL — verify direct agent routing

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added support for new scrape agent service integration with legacy fallback capability.
  • Enhanced scrape job submission with optional job ID parameter.

• Updates

  • Modified scrape job status retrieval and listing functionality.
  • Updated scrape queue processing and deadletter handling behavior.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

1. @coderabbitai review
2. @copilot review
3. @codex review
4. @claude review
   Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 822e7e44-4be2-4e11-b640-c613d684cc44

📥 Commits

Reviewing files that changed from the base of the PR and between 13fbf75 and 8837dfe.

📒 Files selected for processing (2)

• src/index.ts
• src/lib/integrations.ts

---

📝 Walkthrough

Walkthrough

This PR adds support for a new chittyagent-scrape integration by introducing a configuration environment variable and refactoring scrape-related proxy methods in the integrations module. Methods are updated to call new agent endpoints with modified request shapes and parameters, with fallback to legacy router paths.

Changes

Cohort / File(s)	Summary
Environment Configuration src/index.ts	Added optional CHITTYAGENT_SCRAPE_URL environment binding to the Env type.
Scrape Agent Integration src/lib/integrations.ts	Added scrapePost and scrapeGet helper functions that route to agent endpoints when available with fallback to legacy router paths. Updated public methods: enqueueScrapeJob now accepts optional jobId parameter and posts to /api/v1/enqueue; getScrapeJobStatus, retryScrapeJob, and getScrapeDeadLetters now have unused parameters; listScrapeJobs now supports chittyId and offset filters; processScrapeQueue and getScrapeStatus route through new agent endpoints. Some methods fall back to no-op implementations.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant routerClient
    participant Agent as Scrape Agent
    participant Legacy as Legacy Router
    
    Client->>routerClient: enqueueScrapeJob(jobType, target, opts)
    
    alt CHITTYAGENT_SCRAPE_URL is set
        routerClient->>Agent: POST /api/v1/enqueue<br/>(with mapped fields)
        Agent-->>routerClient: Response
        alt Response OK
            routerClient-->>Client: Return result
        else Response Error
            routerClient->>Legacy: POST /agents/scrape/enqueue<br/>(fallback)
            Legacy-->>routerClient: Response
            routerClient-->>Client: Return result
        end
    else CHITTYAGENT_SCRAPE_URL not set
        routerClient->>Legacy: POST /agents/scrape/enqueue
        Legacy-->>routerClient: Response
        routerClient-->>Client: Return result
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• feat: convert to thin proxy (scrape → router, prompts → connect) #49: Modifies the same scrape-related routerClient integrations, adding and rewiring scrape methods with new request shapes and agent endpoint routing.

Poem

🐰 A scraper's hop toward the agent's nest,
With endpoints new and fallback at its best,
Through router and API, the requests now dance,
Legacy paths still welcome, just in case! ✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/scrape-agent-sdk-migration

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}