JSL Visual Document Analyzer

A comprehensive Streamlit-based PDF analysis tool with advanced AI capabilities

🚀 Quick Start • 📖 Documentation • ⚙️ Configuration • 🤝 Contributing

🚀 Overview

The JSL Visual Document Analyzer is a powerful, AI-driven document processing application that transforms PDF documents into structured, searchable content. Built with Streamlit and powered by multiple advanced processing backends, it offers unparalleled flexibility and accuracy in document analysis.

Why Choose JSL Visual Document Analyzer?

• ⚡ Multi-Engine Processing: Choose the best processor for your specific needs
• 🎯 High Accuracy: Advanced AI models for superior text extraction
• 💰 Cost Effective: CPU-optimized option requires no API keys
• 🔧 Developer Friendly: Clean APIs and extensible architecture
• 📊 Enterprise Ready: Robust caching, error handling, and monitoring

✨ Key Features

🎯 Multiple Processing Engines

Engine	Best For	API Required	Speed	Accuracy
CPU Optimized	General documents, cost-effective	❌ No	⚡ Fast	⭐⭐⭐
GPU Optimized (VLM)	Complex layouts, highest quality	✅ OpenRouter	🐌 Slower	⭐⭐⭐⭐⭐
PDF to Markdown	Clean formatting, academic docs	✅ OpenAI	🚀 Medium	⭐⭐⭐⭐
Schema Support	Structured data, forms, invoices	✅ OpenAI/OpenRouter	🚀 Medium	⭐⭐⭐⭐

🧠 AI-Powered Capabilities

• Smart Text Extraction: Maintains document structure and formatting
• Advanced Table Detection: Automatically identifies and extracts tables in markdown format
• Layout Understanding: Preserves document hierarchy and organization
• Multi-Modal Analysis: Combines visual and textual understanding for superior results
• OCR Processing: Advanced optical character recognition with high accuracy
• Image Processing: High-quality page image generation and analysis

🔧 Advanced Features

• Intelligent Caching: Automatic result caching with TTL management (saves time and API costs)
• Real-time Progress: Live processing status with ETA estimates
• Multiple Export Formats: Markdown, Text, HTML, and more
• Print Optimization: Built-in print-ready document generation
• Batch Processing: Handle multiple documents efficiently
• Error Recovery: Robust error handling and recovery mechanisms

🚀 Quick Start

System Requirements

• Python: 3.8 or higher
• Memory: 4GB RAM minimum (8GB recommended for large documents)
• Storage: 1GB free space for cache and temporary files
• Internet: Required for API-based processors

Installation

1. Clone the repository:

git clone <repository-url>
cd jsl-visual-doc-analyzer

2. Create virtual environment (recommended):

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Set up environment variables: Create a .env file in the project root:

# OpenRouter API (for GPU processing)
OPENROUTER_API_KEY=your_openrouter_api_key_here
OPENROUTER_MODEL=meta-llama/llama-3.2-90b-vision-instruct

# OpenAI API (for PDF to Markdown Converter and Schema Support)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_DEFAULT_MODEL=gpt-4o

# Optional: Cache settings
CACHE_TTL_HOURS=2
MAX_FILE_SIZE_MB=25

5. Run the application:

streamlit run app.py

6. Access the application: Open your browser and navigate to http://localhost:8501

📖 How to Use

Step 1: Choose Your Processing Engine

Select the optimal processor for your use case:

🖥️ CPU Optimized (Recommended for beginners)

✅ No API key required
✅ Fast processing
✅ Good for general documents
✅ Cost-effective
❌ Limited accuracy for complex layouts

🚀 GPU Optimized (VLM)

✅ Highest accuracy
✅ Excellent for complex documents
✅ Advanced visual understanding
❌ Requires OpenRouter API key
❌ Slower processing
❌ API costs apply

📝 PDF to Markdown Converter

✅ Clean markdown output
✅ Great for academic documents
✅ Preserves formatting
❌ Requires OpenAI API key
❌ API costs apply

🎯 Schema Support

✅ Structured data extraction
✅ Custom JSON schemas
✅ Perfect for forms/invoices
❌ Requires API key
❌ More complex setup

Step 2: Upload Your Document

Choose your preferred input method:

Method 1: File Upload

1. Click "Upload Files" in the sidebar
2. Drag and drop your PDF (max 25MB)
3. Processing starts automatically

Method 2: Storage Files

1. Place PDF files in data/input/ folder
2. Click "Storage" in the sidebar
3. Select file and click "Process"

Step 3: Monitor Processing

• Progress Bar: Shows real-time processing status
• ETA Estimates: Approximate completion time
• Status Messages: Detailed processing steps
• Error Handling: Clear error messages if issues occur

Step 4: Review Results

Navigate through four comprehensive tabs:

🎨 AI Preview

• Beautifully formatted document display
• Ready for reading and sharing
• Download as markdown
• Print-optimized layout

📝 Extracted Content

• Editable text area with extracted content
• Copy and modify as needed
• Full document text

🔍 Document Analysis

• Page-by-page breakdown
• Processing metrics
• Quality assessment
• Table detection results

📄 Extracted Text

• Plain text output
• Character and word counts
• Raw text for further processing

Step 5: Export and Share

Available actions from any tab:

• Download: Multiple formats (MD, TXT, HTML)
• Print: Optimized for paper output
• Copy: Direct clipboard integration
• Cache: Automatic saving for future access

⚙️ Configuration

Environment Variables Reference

Variable	Description	Default	Required For
OPENROUTER_API_KEY	OpenRouter API authentication	-	GPU Optimized
OPENROUTER_MODEL	VLM model identifier	llama-3.2-90b-vision-instruct	GPU Optimized
OPENAI_API_KEY	OpenAI API authentication	-	PDF to Markdown, Schema
OPENAI_DEFAULT_MODEL	OpenAI model identifier	gpt-4o	PDF to Markdown, Schema
CACHE_TTL_HOURS	Cache expiration time	2	All processors
MAX_FILE_SIZE_MB	Maximum upload size	25	All processors

Performance Tuning

For Large Documents (>10MB):

• Use CPU Optimized processor
• Enable aggressive caching
• Process during off-peak hours (for API-based processors)

For High Accuracy Requirements:

• Use GPU Optimized (VLM)
• Enable detailed analysis
• Review results in Document Analysis tab

For Cost Optimization:

• Start with CPU Optimized
• Use caching aggressively
• Monitor API usage in your provider dashboard

🏗️ Project Architecture

jsl-visual-doc-analyzer/
├── app.py                 # Main Streamlit application entry point
├── src/                   # Core application logic
│   ├── config.py         # Configuration and environment management
│   ├── models.py         # Pydantic data models and schemas
│   ├── processors.py     # Document processing backends
│   └── utils.py          # Utility functions and helpers
├── data/                  # Data storage and cache
│   ├── input/            # Input PDFs (for storage mode)
│   └── output/           # Processed results and cache
├── markpdfdown/          # PDF to Markdown processor module
├── sparrow/              # Schema Support processor module
├── requirements.txt      # Python dependencies
├── .env.example         # Environment variables template
└── README.md            # This documentation

🚨 Troubleshooting

Common Issues and Solutions

Installation Issues

# Problem: Dependencies fail to install
# Solution: Upgrade pip and use virtual environment
pip install --upgrade pip
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Runtime Errors

Error	Cause	Solution
"No processors available"	Missing dependencies	Run pip install -r requirements.txt
"File too large"	File exceeds 25MB limit	Compress PDF or split document
"API key invalid"	Incorrect/expired API key	Verify API key in the .env file
"Processing timeout"	Large file/slow network	Use CPU processor or smaller files

Performance Issues

• Slow processing: Try CPU Optimized processor first
• High memory usage: Close other applications, use smaller files
• API rate limits: Enable caching, spread out processing

Getting Help

1. Check logs: Look for error messages in the Streamlit interface
2. Verify setup: Ensure all requirements are installed
3. Test with small files: Validate setup with simple documents
4. Check API status: Verify your API provider's service status

🔒 Security and Privacy

• Local Processing: CPU Optimized mode processes documents entirely locally
• API Security: API keys are stored securely in environment variables
• Data Retention: Cached files are automatically cleaned based on TTL settings
• No Persistent Storage: Documents are not permanently stored on our servers

🤝 Contributing

We welcome contributions from the community!

How to Contribute

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit your changes: git commit -m 'Add amazing feature'
4. Push to the branch: git push origin feature/amazing-feature
5. Open a Pull Request

Development Setup

# Clone your fork
git clone https://github.com/yourusername/jsl-visual-doc-analyzer.git
cd jsl-visual-doc-analyzer

# Install development dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt  # If available

# Run tests
python -m pytest tests/  # If tests are available

# Start development server
streamlit run app.py

Contribution Guidelines

• Follow Python PEP 8 style guidelines
• Add tests for new features
• Update documentation for new functionality
• Ensure backwards compatibility

📊 Roadmap

Upcoming Features

• Batch Processing: Process multiple files simultaneously
• API Endpoints: REST API for programmatic access
• Cloud Integration: AWS S3, Google Drive, Dropbox support
• Advanced OCR: Support for handwritten text recognition
• Custom Models: Integration with custom-trained models
• Collaborative Features: Multi-user document review

Version History

• v3.0: Multi-processor architecture, improved caching
• v2.0: GPU optimization, Schema support
• v1.0: Initial release with CPU processing

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🆘 Support

Getting Support

• 📖 Documentation: Start with this README
• 🐛 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📧 Email: support@johnsnowlabs.com

Enterprise Support

For enterprise customers, we offer:

• Priority technical support
• Custom integrations
• Training and consultation
• SLA-backed response times

Contact us at enterprise@johnsnowlabs.com

🏢 About John Snow Labs

John Snow Labs is the AI company for healthcare and life sciences, providing state-of-the-art AI platforms and solutions. We specialize in natural language processing, computer vision, and machine learning technologies.

Learn more: johnsnowlabs.com

---

⭐ Star this repository if you find it helpful!

Made with ❤️ by the John Snow Labs team

Report Bug • Request Feature • Documentation