On-device LLM server and companion chat application for Android. Thinai wraps llama.cpp (via a vendored fork of fllama) and exposes a local Ollama- and OpenAI-compatible HTTP API on 127.0.0.1:11434.
Once Thinai is running with a downloaded model, any local client or companion app (such as Thoogai AF) can point to http://127.0.0.1:11434 for fully offline, private on-device inference.
- On-Device Inference: Run GGUF models locally on your Android phone using llama.cpp with no cloud dependency.
- Dual API Compatibility: Exposes both Ollama (
/api/*) and OpenAI (/v1/*) HTTP endpoints. - Vector Embeddings: Dedicated on-device embedding endpoints (
/api/embed,/v1/embeddings) for local RAG and semantic search. - Built-in Chat Interface: Clean mobile UI with model downloader, context slider, and generation temperature settings.
- On-Device Web Search: Optional web search capability to answer questions needing current information, citing sources while keeping chats local.
- Multimodal Attachments: Take photos via the system camera, attach images, or analyze documents directly in chat.
- Background Service: Android foreground service ensures the local HTTP server continues running when the app is backgrounded.
- Hardware Acceleration: Experimental GPU offload (Vulkan / OpenCL) with built-in CPU vs. GPU benchmark, and automatic fallback to the CPU when a device's GPU driver misbehaves.
GET /healthz: Liveness check,{"status":"ok"}. Needs no token, so uptime monitors can use it through the public URL.
GET /api/tags: List downloaded modelsGET /api/ps: Currently loaded model statusPOST /api/show: Model metadata and context lengthPOST /api/generate: Text generation (NDJSON stream)POST /api/chat: Chat completion (NDJSON stream)POST /api/embed: Embeddings (input= string or array)POST /api/embeddings: Legacy single-promptembeddings
OpenAI-Compatible (Full Reference)
GET /v1/models: List available modelsPOST /v1/chat/completions: Chat completion (SSE stream, tool calling,stopsequences, usage metrics)POST /v1/embeddings: Vector embeddings withencoding_format(float|base64) anddimensionsshortening
Point any standard OpenAI SDK or HTTP client at http://127.0.0.1:11434/v1 with any placeholder API key.
Turn on Public URL in the Server tab to reach the API from anywhere through a free Cloudflare quick tunnel. It needs no account or sign-up. Each start gets a random https://<words>.trycloudflare.com URL. Rotate URL opens a new one, and the old URL stops working at once.
Every request through the public URL (and over the LAN) needs a bearer token. Create named tokens under API tokens, one per client. The list shows when each was created, and deleting a token cuts off that client immediately, without restarting the server.
curl https://<words>.trycloudflare.com/api/tags -H "Authorization: Bearer <token>"The tunnel is run by the open-source cloudflared (Apache-2.0). The Gradle build downloads it at a pinned version, checks its SHA-256, and bundles it as lib/arm64-v8a/libcloudflared.so, so it isn't committed to git. cloudflared can't resolve DNS on Android (Go finds no /etc/resolv.conf), so lib/server/quick_tunnel.dart does both lookups in Dart: it requests the tunnel from api.trycloudflare.com and passes the resolved edge IPs with --edge.
Embeddings run fully on-device. Download an embedding model from the Models tab ("Embedding models" section), then query it:
curl http://127.0.0.1:11434/api/embed \
-d '{"model":"nomic-embed-text-v1.5-q5_k_m","input":["hello","world"]}'curl http://127.0.0.1:11434/v1/embeddings \
-d '{"model":"nomic-embed-text-v1.5-q5_k_m","input":"hello"}'- Use an embedding model, not a chat model: Chat models lack pooling layers. Calling an embedding endpoint with a chat model returns HTTP
400rather than degraded token-level vectors. - Vectors are L2-normalized; cosine similarity is a plain dot product.
dimensionstruncates and re-normalizes (Matryoshka representation).- Embeddings use a separate llama context from chat, so inference calls do not evict each other.
Upstream fllama exposes no embedding FFI, so third_party/fllama is a vendored fork that provides it:
| Layer | File |
|---|---|
| Native bridge | third_party/fllama/src/fllama_embed.{h,cpp} |
| Dart FFI | third_party/fllama/lib/io/fllama_io_embed.dart |
| Engine | lib/llm/llm_engine.dart (embedBatch) |
| Routes | lib/server/routes/{ollama,openai}_routes.dart |
fllama_embed.cpp posts SERVER_TASK_TYPE_EMBEDDING tasks to llama.cpp's server_context: the same mechanism llama-server uses for its /embedding endpoint. third_party/fllama is committed in-tree so a clean checkout and CI can both build without an external private remote.
- Flutter SDK: 3.41+ (Dart
^3.11.0) - Android SDK: API level 24 (Android 7.0 Nougat) or higher
- Android NDK & CMake: CMake
3.22.1installed via Android SDK Manager - Device: Android device with 64-bit ARM architecture (
arm64-v8a) recommended for adequate RAM and LLM performance. - iOS (early): Xcode 16.4+ with CocoaPods, an Apple ID for signing, and an iPhone on iOS 15 or later. Tested on an iPhone 12.
git clone https://github.com/Abhinivesh2729/Thinai.git
cd Thinai
flutter pub getConnect an Android device or launch an emulator:
flutter runNote: The first build compiles the native
llama.cppC++ engine from source and takes several minutes.
On iOS, open ios/Runner.xcworkspace once and pick your team under Signing & Capabilities, then run a release build (the mode tested so far):
flutter run --releaseThe API server runs only while Thinai is in the foreground on iOS. There is no equivalent of Android's foreground service.
Open the Server tab in the app to start the HTTP listener and select an installed model.
Run static analysis:
flutter analyzeFormat code:
dart format --set-exit-if-changed .Run unit and integration tests (excluding native on-device tests):
flutter test --exclude-tags nativeNote: Tests marked
@Tags(['native'])require an on-device environment with downloaded model weights.
We welcome community contributions! Please read our contributing guide and code of conduct before getting started:
- See CONTRIBUTING.md for branch guidelines, workflows, and PR expectations.
- See CODE_OF_CONDUCT.md for community standards.
Please report vulnerabilities responsibly. Do NOT report security issues via public GitHub issues.
- See SECURITY.md for reporting instructions via private security advisories.
This project is licensed under the MIT License — see the LICENSE file for details.
The vendored third_party/fllama package and its bundled llama.cpp sources are licensed under their respective licenses (see third_party/fllama/LICENSE).