twitter_crawler

A minimal crawler that scans Twitter/X for crypto projects, collects all related tweets, extracts their stats, and tracks them over time.

How it works

• Opens the Twitter/X website via Puppeteer (no Twitter API required).
• Searches for tweets related to specified crypto projects.
• Extracts tweet metadata/stats and persists results as JSON, then stores them in a PostgreSQL database.

No official Twitter/X API is used — the site is read directly in a headless browser session.

Prerequisites

• Node.js & npm
• Ability to run Puppeteer (Chromium will be handled by Puppeteer)
• PostgreSQL available (the app writes parsed JSON to a Postgres DB)

Environment variables

Create a .env (or use your platform’s env configuration) and set:

# Created by Vercel CLI
CRON_SECRET=...           # Secret string to trigger cron
TWITTER_USER=...          # Twitter/X username for login
TWITTER_PASSWORD=...      # Twitter/X password for login

Installation

npm install

Run locally

Use either the Node server or Vercel’s dev server:

# start with Node
npm run start

# or, if you use Vercel locally
vercel dev

The crawler will:

1. Launch a Puppeteer session,
2. Log in to Twitter/X using the provided credentials,
3. Collect tweets & stats for the target crypto projects,
4. Store parsed JSON and write the data to PostgreSQL.

Deployment

• The repository contains a vercel.json.
• Set the environment variables (CRON_SECRET, TWITTER_USER, TWITTER_PASSWORD) in your deployment environment.
• If you trigger scheduled jobs, protect any cron entrypoint with CRON_SECRET.

Data & Storage

• Data format: JSON (intermediate)
• Persistence: Written to a PostgreSQL database

Security & Responsibility

• Keep TWITTER_USER and TWITTER_PASSWORD out of version control and rotate them if exposed.
• Use this tool responsibly. Ensure your usage complies with Twitter/X Terms of Service and any applicable laws/regulations in your jurisdiction.