Add skill: firecrawl

skills/firecrawl/README.md

@@ -0,0 +1,48 @@
+# firecrawl
+
+Web scraping, searching, and crawling CLI for AI agents. Use when you need to: (1) scrape web pages for clean markdown content, (2) perform web searches with optional page scraping, or (3) discover and map all URLs on a site.
+
+Returns clean markdown optimized for LLM context windows, handles JavaScript rendering, bypasses common blocks, and provides structured data.
+
+## Installation
+
+```bash
+npm install -g firecrawl-cli
+```
+
+Then authenticate:
+
+```bash
+firecrawl login --browser
+```
+
+## Features
+
+- **Search** - Web, image, and news search with optional scraping of results
+- **Scrape** - Single page content extraction with multiple output formats (Markdown, HTML, links, screenshots)
+- **Map** - Discover and list all URLs on a site with filtering and subdomain support
+- **Parallel execution** - Run multiple scrapes concurrently up to your concurrency limit
+- **LLM-optimized output** - Clean markdown output designed for AI agent context windows
+
+## Triggers
+
+- Any URL or webpage
+- Web, image, and news search
+- Research, deep research, investigation
+- Reading pages, docs, articles, sites, documentation
+- Content extraction, link discovery, site mapping, crawling
+
+## Tags
+
+`web-scraping`, `search`, `crawling`, `content-extraction`, `markdown`
+
+## Author
+
+firecrawl
+
+## Links
+
+- [Firecrawl](https://firecrawl.dev)
+- [GitHub](https://github.com/firecrawl)
+- [npm](https://www.npmjs.com/package/firecrawl-cli)
+- [Documentation](https://docs.firecrawl.dev)

skills/firecrawl/SKILL.md

@@ -0,0 +1,270 @@
+---
+name: firecrawl
+description: "Web scraping, searching, and crawling CLI for AI agents. Use when you need to: (1) scrape web pages for clean markdown content, (2) perform web searches with optional page scraping, or (3) discover and map all URLs on a site."
+version: 1.0.0
+category: search-data
+---
+
+# Firecrawl
+
+Web scraping, searching, and crawling CLI for AI agents. Returns clean markdown optimized for LLM context windows, handles JavaScript rendering, bypasses common blocks, and provides structured data.
+
+## Installation
+
+```bash
+npm install -g firecrawl-cli
+```
+
+Check status, auth, and rate limits:
+
+```bash
+firecrawl --status
+```
+
+Output when ready:
+
+```text
+  🔥 firecrawl cli v1.0.2
+
+  ● Authenticated via FIRECRAWL_API_KEY
+  Concurrency: 0/100 jobs (parallel scrape limit)
+  Credits: 500,000 remaining
+```
+
+- **Concurrency**: Max parallel jobs. Run parallel operations close to this limit but not above.
+- **Credits**: Remaining API credits. Each scrape/crawl consumes credits.
+
+## Authentication
+
+If not authenticated, run:
+
+```bash
+firecrawl login --browser
+```
+
+The `--browser` flag automatically opens the browser for authentication without prompting. This is the recommended method for agents.
+
+### Manual API Key
+
+Alternatively, authenticate with an API key from [firecrawl.dev](https://firecrawl.dev):
+
+```bash
+firecrawl login --api-key "<your-key>"
+```
+
+Or set the environment variable:
+
+```bash
+export FIRECRAWL_API_KEY="<your-key>"
+```
+
+## Organization
+
+Create a `.firecrawl/` folder in the working directory to store results. Always use `-o` to write directly to file (avoids flooding context):
+
+```bash
+# Search the web
+firecrawl search "your query" -o .firecrawl/search-{query}.json
+
+# Search with scraping enabled
+firecrawl search "your query" --scrape -o .firecrawl/search-{query}-scraped.json
+
+# Scrape a page
+firecrawl scrape https://example.com -o .firecrawl/{site}-{path}.md
+```
+
+**Always quote URLs** - shell interprets `?` and `&` as special characters.
+
+## Commands
+
+### Search - Web search with optional scraping
+
+```bash
+# Basic search (human-readable output)
+firecrawl search "your query" -o .firecrawl/search-query.txt
+
+# JSON output (recommended for parsing)
+firecrawl search "your query" -o .firecrawl/search-query.json --json
+
+# Limit results
+firecrawl search "AI news" --limit 10 -o .firecrawl/search-ai-news.json --json
+
+# Search specific sources
+firecrawl search "tech startups" --sources news -o .firecrawl/search-news.json --json
+firecrawl search "landscapes" --sources images -o .firecrawl/search-images.json --json
+firecrawl search "machine learning" --sources web,news,images -o .firecrawl/search-ml.json --json
+
+# Filter by category (GitHub repos, research papers, PDFs)
+firecrawl search "web scraping python" --categories github -o .firecrawl/search-github.json --json
+firecrawl search "transformer architecture" --categories research -o .firecrawl/search-research.json --json
+
+# Time-based search
+firecrawl search "AI announcements" --tbs qdr:d -o .firecrawl/search-today.json --json  # Past day
+firecrawl search "tech news" --tbs qdr:w -o .firecrawl/search-week.json --json          # Past week
+firecrawl search "yearly review" --tbs qdr:y -o .firecrawl/search-year.json --json      # Past year
+
+# Location-based search
+firecrawl search "restaurants" --location "San Francisco,California,United States" -o .firecrawl/search-sf.json --json
+firecrawl search "local news" --country DE -o .firecrawl/search-germany.json --json
+
+# Search AND scrape content from results
+firecrawl search "firecrawl tutorials" --scrape -o .firecrawl/search-scraped.json --json
+firecrawl search "API docs" --scrape --scrape-formats markdown,links -o .firecrawl/search-docs.json --json
+```
+
+**Search Options:**
+
+- `--limit <n>` - Maximum results (default: 5, max: 100)
+- `--sources <sources>` - Comma-separated: web, images, news (default: web)
+- `--categories <categories>` - Comma-separated: github, research, pdf
+- `--tbs <value>` - Time filter: qdr:h (hour), qdr:d (day), qdr:w (week), qdr:m (month), qdr:y (year)
+- `--location <location>` - Geo-targeting (e.g., "Germany")
+- `--country <code>` - ISO country code (default: US)
+- `--scrape` - Enable scraping of search results
+- `--scrape-formats <formats>` - Scrape formats when --scrape enabled (default: markdown)
+- `-o, --output <path>` - Save to file
+
+### Scrape - Single page content extraction
+
+```bash
+# Basic scrape (markdown output)
+firecrawl scrape https://example.com -o .firecrawl/example.md
+
+# Get raw HTML
+firecrawl scrape https://example.com --html -o .firecrawl/example.html
+
+# Multiple formats (JSON output)
+firecrawl scrape https://example.com --format markdown,links -o .firecrawl/example.json
+
+# Main content only (removes nav, footer, ads)
+firecrawl scrape https://example.com --only-main-content -o .firecrawl/example.md
+
+# Wait for JS to render
+firecrawl scrape https://spa-app.com --wait-for 3000 -o .firecrawl/spa.md
+
+# Extract links only
+firecrawl scrape https://example.com --format links -o .firecrawl/links.json
+
+# Include/exclude specific HTML tags
+firecrawl scrape https://example.com --include-tags article,main -o .firecrawl/article.md
+firecrawl scrape https://example.com --exclude-tags nav,aside,.ad -o .firecrawl/clean.md
+```
+
+**Scrape Options:**
+
+- `-f, --format <formats>` - Output format(s): markdown, html, rawHtml, links, screenshot, json
+- `-H, --html` - Shortcut for `--format html`
+- `--only-main-content` - Extract main content only
+- `--wait-for <ms>` - Wait before scraping (for JS content)
+- `--include-tags <tags>` - Only include specific HTML tags
+- `--exclude-tags <tags>` - Exclude specific HTML tags
+- `-o, --output <path>` - Save to file
+
+### Map - Discover all URLs on a site
+
+```bash
+# List all URLs (one per line)
+firecrawl map https://example.com -o .firecrawl/urls.txt
+
+# Output as JSON
+firecrawl map https://example.com --json -o .firecrawl/urls.json
+
+# Search for specific URLs
+firecrawl map https://example.com --search "blog" -o .firecrawl/blog-urls.txt
+
+# Limit results
+firecrawl map https://example.com --limit 500 -o .firecrawl/urls.txt
+
+# Include subdomains
+firecrawl map https://example.com --include-subdomains -o .firecrawl/all-urls.txt
+```
+
+**Map Options:**
+
+- `--limit <n>` - Maximum URLs to discover
+- `--search <query>` - Filter URLs by search query
+- `--sitemap <mode>` - include, skip, or only
+- `--include-subdomains` - Include subdomains
+- `--json` - Output as JSON
+- `-o, --output <path>` - Save to file
+
+## Reading Scraped Files
+
+Firecrawl output files can be large (1000+ lines). Use grep, head, or incremental reads instead of reading entire files at once:
+
+```bash
+# Check file size and preview structure
+wc -l .firecrawl/file.md && head -50 .firecrawl/file.md
+
+# Use grep to find specific content
+grep -n "keyword" .firecrawl/file.md
+grep -A 10 "## Section" .firecrawl/file.md
+```
+
+## Format Behavior
+
+- **Single format**: Outputs raw content (Markdown text, HTML, etc.)
+- **Multiple formats**: Outputs JSON with all requested data
+
+```bash
+# Raw markdown output
+firecrawl scrape https://example.com --format markdown -o .firecrawl/page.md
+
+# JSON output with multiple formats
+firecrawl scrape https://example.com --format markdown,links -o .firecrawl/page.json
+```
+
+## Combining with Other Tools
+
+```bash
+# Extract URLs from search results
+jq -r '.data.web[].url' .firecrawl/search-query.json
+
+# Get titles from search results
+jq -r '.data.web[] | "\(.title): \(.url)"' .firecrawl/search-query.json
+
+# Process news results
+jq -r '.data.news[] | "[\(.date)] \(.title)"' .firecrawl/search-news.json
+
+# Count URLs from map
+firecrawl map https://example.com | wc -l
+```
+
+## Parallelization
+
+Always run multiple scrapes in parallel, never sequentially. Check `firecrawl --status` for concurrency limit, then use `&` and `wait`:
+
+```bash
+# Parallel scraping
+firecrawl scrape https://site1.com -o .firecrawl/1.md &
+firecrawl scrape https://site2.com -o .firecrawl/2.md &
+firecrawl scrape https://site3.com -o .firecrawl/3.md &
+wait
+```
+
+For many URLs, use xargs with `-P` for parallel execution:
+
+```bash
+cat urls.txt | xargs -P 10 -I {} sh -c 'firecrawl scrape "{}" -o ".firecrawl/$(echo {} | md5).md"'
+```
+
+## Troubleshooting
+
+### Command not found
+
+If `firecrawl` command is not found after installation:
+
+1. Make sure npm global bin is in PATH
+2. Try: `npx firecrawl-cli --version`
+3. Or reinstall: `npm install -g firecrawl-cli`
+
+### Permission errors
+
+If you get permission errors during installation:
+
+```bash
+# Fix npm permissions (recommended)
+mkdir ~/.npm-global
+npm config set prefix '~/.npm-global'
+export PATH=~/.npm-global/bin:$PATH
+```