AstroPh Summarizer

A modular system for fetching, filtering, and summarizing recent astrophysics papers from arXiv using RAG (Retrieval-Augmented Generation) and LLM-based filtering.

Features

1. Scrapes recent papers from arXiv astro-ph
2. Extracts paper metadata (ID, title, abstract, URL, authors)
3. Stores papers in JSON format
4. Uses FAISS + embeddings (Ollama or OpenAI) for RAG-based filtering
5. Filters papers using LLM (OpenAI or Ollama with pydantic_ai) based on user-defined research interests
6. Generates structured summaries (1-3 sentences) for relevant papers
7. Posts summaries to Slack channels

Project Structure

astroph/
├── src/
│   ├── astroph/
│   │   ├── __init__.py
│   │   ├── models.py           # Pydantic data models
│   │   ├── arxiv_scraper.py    # ArXiv scraping
│   │   ├── embeddings.py       # FAISS + embedding service
│   │   ├── llm_filter.py       # LLM filtering with pydantic_ai
│   │   ├── slack_notifier.py   # Slack posting
│   │   └── pipeline.py         # Main orchestration
│   └── config.py               # Configuration management
├── tests/                      # Unit tests for each module
│   ├── test_arxiv_scraper.py
│   ├── test_embeddings.py
│   ├── test_llm_filter.py
│   ├── test_slack_notifier.py
│   └── test_pipeline.py
├── main.py                     # Entry point
├── requirements.txt
└── .env                        # Environment variables

Installation

1. Clone the repository
2. Create a virtual environment:

   python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Copy .env.example to .env and configure:

   cp .env.example .env
   # Edit .env with your credentials

Configuration

Create a .env file with the following variables:

# Slack Configuration
SLACK_TOKEN=xoxb-your-slack-bot-token
SLACK_CHANNEL=C0A4V13PF4L

# OpenAI Configuration (if using OpenAI)
OPENAI_API_KEY=sk-your-openai-api-key

# Embedding Provider: "ollama" or "openai"
EMBEDDING_PROVIDER=ollama
EMBEDDING_MODEL=nomic-embed-text

# LLM Provider: "openai" or "ollama"
LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini

# Ollama Configuration (if using Ollama)
OLLAMA_BASE_URL=http://localhost:11434

Configuration Options

• EMBEDDING_PROVIDER: ollama or openai
• LLM_PROVIDER: openai or ollama
• SLACK_TOKEN: Your Slack bot token
• OPENAI_API_KEY: Your OpenAI API key

Usage

Basic Usage

Run the pipeline with your research interests:

python main.py --field "gravitational wave astronomy" --max-papers 10

Advanced Options

# With keywords
python main.py \
  --field "exoplanet detection" \
  --keywords "transit" "radial velocity" \
  --max-papers 15

# Save results without posting to Slack
python main.py \
  --field "dark matter" \
  --max-papers 20 \
  --save-json results.json \
  --no-slack

# Save and reuse RAG index
# First run - build and save index
python main.py \
  --field "cosmology" \
  --save-index index.faiss papers.json

# Later runs - load saved index
python main.py \
  --field "dark energy" \
  --load-index index.faiss papers.json

Command-Line Arguments

• --field: Research field or topic (required)
• --keywords: Optional keywords to focus on
• --max-papers: Maximum papers to return (default: 10)
• --no-rag: Disable RAG filtering (process all papers with LLM)
• --no-slack: Don't post results to Slack
• --save-json: Path to save results as JSON
• --save-index: Save RAG index for reuse
• --load-index: Load previously saved RAG index

Data Flow

1. ArxivScraper: Fetches recent papers → List[Paper]
2. EmbeddingService: Creates embeddings → FAISS Index
3. EmbeddingService.search(): RAG filtering → Top K candidates
4. LLMFilter: Analyzes candidates → List[FilteredPaper]
5. SlackNotifier: Posts summaries → Slack channel

Example Workflow

from src.astroph.pipeline import PaperPipeline
from src.astroph.models import UserQuery

pipeline = PaperPipeline(
    embedding_provider="ollama",
    llm_provider="openai",
    llm_model="gpt-4o-mini"
)

user_query = UserQuery(
    field="gravitational wave astronomy",
    keywords=["LIGO", "black holes"],
    max_papers=10
)

results = pipeline.run(
    user_query=user_query,
    use_rag=True,
    post_to_slack=True,
    save_to_json="results.json"
)

Common Issues

Ollama not running

# Start Ollama
ollama serve

# Pull nomic-embed model
ollama pull nomic-embed-text

Slack permission errors

Ensure your bot has:

• chat:write scope
• Access to the target channel

OpenAI rate limits

Use smaller --max-papers values or switch to Ollama.

Requirements

• Python 3.10+
• Ollama (optional, for local embeddings/LLM)
• OpenAI API key (optional, for OpenAI embeddings/LLM)
• Slack bot token (for posting to Slack)

License

MIT License

Copyright (c) 2025

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.