StreamSight — Real-time News Chat (local)

StreamSight is a local Flask-based prototype that demonstrates:

• Real-time streaming ingestion of a fake NewsAPI-style dataset
• Per-item normalization and semantic upsert (vector indexing)
• Event streaming (SSE) to a web UI for live updates
• A ChatGPT-style UI backed by RAG + OpenRouter chat fallbacks

This README explains how to run the app locally, where files live, and the key API endpoints exposed by the server.

Prerequisites

• Python 3.10+ (recommended)
• A virtual environment for dependencies
• An OpenRouter API key set in the environment variable OPENROUTER_API_KEY if you want LLM responses

Repository layout

• server.py — Flask application and SSE stream endpoint
• chat_store.py — lightweight JSON-backed chat/session store
• controller/ — OpenRouter client + RAG controller logic
• model/ — ingestion, normalization, indexing, upsert utilities
• data/ — raw_news.json (input) and normalized_news.json (output)
• view/ui/ — frontend assets: index.html, styles.css, script.js
• requirements.txt — Python dependencies

Install

On Windows (PowerShell):

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On macOS / Linux (bash):

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Environment

Create a .env file in the project root or set the env var manually:

OPENROUTER_API_KEY=your_openrouter_key_here

Run the server

Start the Flask server (this also starts the background ingestion worker):

python server.py

Open your browser to: http://127.0.0.1:5000/ to access the UI.

Important behavior

• Ingestion: model/ingestion.py yields raw news items one-by-one using a randomized delay between 0 and 180 seconds (0–3 minutes). Each item is normalized and semantically upserted immediately.
• Event Stream: The server exposes /stream (SSE). The frontend connects to this endpoint to receive new items in real time. The news column displays incoming items one-by-one with newest at the top.
• Chat: Chat requests use RAG via existing controller logic; if no documents are available, the system falls back to the OpenRouter chat endpoint. Chat sessions are persisted to data/chats.json via chat_store.py.

Key endpoints

• GET / — Serves the frontend UI
• GET /stream — Server-Sent Events stream of newly ingested items
• POST /query — Internal retrieval API used by the UI
• POST /gemini — Proxy to the OpenRouter chat / generate-answer flow
• POST /new-chat — Create a new chat session
• GET /chats — List chats
• GET /chat/<chat_id> — Get chat history
• POST /chat — Send a user message to a chat session (store + generate answer)

Notes

• The UI is located in view/ui/ and uses a dark theme with a subtle animated background. The chat message container is transparent and the right-side news column displays live updates.
• No external database is required — storage is file-backed or in-memory.
• Do not commit your .env or API keys to source control.

Troubleshooting

• If the server fails on startup complaining about missing OPENROUTER_API_KEY, set the env var or add .env.
• If you need to reset stored chats, remove data/chats.json.

Enjoy StreamSight — ask in chat about the latest ingested items. News Intelligence Backend (Pathway-style)

This repository is a hackathon-focused backend core that demonstrates:

• Streaming ingestion of NewsAPI-style raw data
• Normalization into a stable schema
• Semantic upsert (dedup + update) using embeddings
• Vector indexing and hybrid RAG-style retrieval

Structure

• model/ - pipeline primitives and processing logic
• controller/ - retrieval controller
• data/ - raw_news.json (fake NewsAPI-like data) and normalized_news.json (output)
• run.py - pipeline entrypoint
• requirements.txt - required packages

Quick start

1. Create a virtualenv and install dependencies:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

2. Run the pipeline:

python run.py

After running, data/normalized_news.json will be populated with the semantic upserted normalized news items. The controller exposes a hybrid_retrieve function which performs vector search + keyword filtering.

Notes

• A lightweight model/pathway_shim.py is included to emulate a tiny streaming API similar to Pathway so the code is portable for the hackathon environment.
• Embeddings use scikit-learn TF-IDF (no external LLMs or API keys).
• The upsert logic uses vector similarity (NearestNeighbors) to decide whether to update an existing record or insert a new one.