Jarvis — Voice-Controlled PC Assistant

A fully offline-capable, voice-controlled PC assistant for Windows. Clap to activate, speak commands, and control your computer hands-free.

Features

• Clap activation — clap to wake, say "jarvis" to dismiss
• Conversation mode — stays active for 15s, then auto-sleeps
• Offline speech — Vosk small English model (~40MB, auto-downloaded on first use)
• Ollama LLM — understands natural language commands using a local LLM
• Screen interaction — click buttons, type into search bars, read text on screen via Windows UI Automation + Tesseract OCR
• App search — type queries directly into Chrome, Spotify, Discord, VS Code, and more via app-specific hotkeys
• System control — volume, media playback, window management, power (shutdown/restart/sleep/lock)
• Timers — voice-set countdown timers with Windows toast notifications
• Web search — DuckDuckGo-powered factual answers aloud
• Hand gesture control — MediaPipe-based mouse control via hand gestures
• Security gate — blocked/dangerous commands require voice confirmation
• CLI mode — text-based interaction for testing

Requirements

• Windows 10/11
• Python 3.10+
• Ollama — install from ollama.com and run ollama serve
• Tesseract OCR — for screen text reading. Install via winget: winget install -e --id UB-Mannheim.TesseractOCR

Quick Start

# Install dependencies
pip install -r requirements.txt

# Pull the default LLM model
ollama pull llama3.2

# Run in voice mode (clap to activate)
python main.py

# Or run in CLI mode for testing
python main.py --cli

Commands

Category	Examples
System	"open calculator", "open chrome", "close notepad"
Search	"search for cats", "search python on youtube"
Screen	"click on the search bar", "what is on my screen", "read the screen"
Web	"what is the weather", "who is the president"
Media	"play", "next song", "volume up", "mute"
Window	"show desktop", "maximize window", "close this window"
Power	"lock my computer", "shutdown", "restart"
Timer	"set a timer for 5 minutes"
Clipboard	"copy", "paste", "what is on my clipboard"
Info	"what time is it", "system info", "battery", "what is running"
Hands	"turn on hand control", "turn off hand control"

Auto-Start

python main.py --install     # Add to Windows Startup
python main.py --uninstall   # Remove from Startup

Project Structure

Jarvis/
├── main.py                  # Entry point, CLI parsing, Jarvis class
├── requirements.txt
├── config/
│   ├── settings.py          # All configuration constants
│   └── logger.py            # Logging setup
├── core/
│   ├── brain.py             # Ollama LLM integration
│   ├── listener.py          # Clap detection + Vosk/SpeechRecognition
│   └── speaker.py           # PowerShell TTS with queue
└── modules/
    ├── app_control.py       # App-specific search hotkeys
    ├── automator.py         # PyAutoGUI keyboard/mouse wrapper
    ├── hand_control.py      # MediaPipe hand gesture control
    ├── ocr.py               # UIA + Tesseract OCR
    ├── security.py          # Command classification (blocked/dangerous/safe)
    ├── system.py            # Process management, power, system info
    ├── vision.py            # Screen interaction (locate, click, read, describe)
    └── web_search.py        # DuckDuckGo search