README.TXT

FINORA - COMPLETE BEGINNER GUIDE
================================

Last updated: 21 August 2026

This guide explains the project from the first browser click to the last database
write. You do not need backend experience to follow it. Read the QUICK START first
if you only want to run the app. Read the remaining sections to understand how it
works internally.


1. WHAT FINORA IS
-----------------

Finora is a full-stack document intelligence application. A user can upload a
PDF, DOC, DOCX, or TXT file, keep the original file in private object storage,
process its contents, search source-linked passages, review extracted fields, ask
questions with citations, edit an extraction template, and view workspace analytics.

The browser is not the source of truth. Users, workspaces, documents, processing
states, fields, schemas, activities, and assistant history are persisted by the
backend. The interface does not contain fake documents, fake AI answers, or fake
processing timers.


2. QUICK START IN VS CODE ON WINDOWS
------------------------------------

Prerequisites:

1. Install Node.js 22.13 or newer.
2. Install Visual Studio Code.
3. Open VS Code.
4. Choose File -> Open Folder.
5. Select:

   C:\Users\anubh\Desktop\Talk2PDF

6. In VS Code choose Terminal -> New Terminal.
7. Run these commands one at a time:

   npm install
   Copy-Item .env.example .env.local
   npm run db:migrate:local
   npm run dev

8. Wait until the terminal prints:

   Local: http://localhost:3000/

9. Open http://localhost:3000 in a browser.
10. Click "Open workspace" or go directly to:

   http://localhost:3000/workspace

Keep the terminal open. The development server stops when the terminal closes or
when you press Ctrl+C.


3. ENABLE FREE-TIER PDF AI PROCESSING
-------------------------------------

TXT files work locally without an AI key. They are really stored, split into
passages, indexed, and searchable. PDF understanding and the cited AI assistant can
use Google's Gemini API free tier. Google applies free usage limits, so this means
free within its current quota, not unlimited use forever.

Create a Gemini API key at:

   https://aistudio.google.com/api-keys

Sign in with Google, create an API key in a free project, and do not enable billing
unless you personally decide to upgrade later. Then open .env.local and set:

   AI_PROVIDER=gemini
   GEMINI_API_KEY=your_private_gemini_key
   GEMINI_MODEL=gemini-3.1-flash-lite
   GEMINI_EMBEDDING_MODEL=gemini-embedding-001

Then stop and restart npm run dev.

Important rules:

- Never add NEXT_PUBLIC_ in front of GEMINI_API_KEY. NEXT_PUBLIC values can be sent
  to the browser.
- Never paste the key into a chat, screenshot, source file, or Git commit. Put it
  only after GEMINI_API_KEY= in the private .env.local file.
- Never commit .env.local. The .gitignore already excludes it.
- If the key is missing, Finora deliberately reports a configuration error. It
  does not invent processing results or AI answers.
- Gemini mode supports PDF and TXT input. Convert DOC or DOCX files to PDF before
  uploading them. The optional paid OpenAI provider remains available for those
  formats if you configure AI_PROVIDER=openai and OPENAI_API_KEY.
- Google's free-tier terms say submitted content may be used to improve Google
  products. Do not use the free tier for confidential documents unless that policy
  is acceptable to you.


4. TECHNOLOGY USED
------------------

Frontend:

- React 19 renders the interactive interface.
- TypeScript checks the shapes of data and catches mistakes before runtime.
- vinext provides the Next.js-compatible application router on Vite.
- TanStack Query loads server data, caches it, handles errors, and refreshes it.
- Zustand stores small device/UI choices such as the active section and selected
  document. It does not replace the server database.
- React Hook Form and Zod validate editable forms.
- Framer Motion provides restrained interface animation.
- Lucide supplies accessible interface icons.

Backend and storage:

- Route handlers in app/api are the HTTP backend.
- A Cloudflare Worker runs the full application.
- D1 (SQLite) stores relational application records.
- R2 stores the original uploaded document bytes.
- Drizzle ORM creates typed SQL queries and migrations.
- unpdf supplies a serverless PDF.js build that extracts text separately from every
  PDF page inside Node.js and Cloudflare Workers.
- Gemini Generate Content reads PDF/TXT documents and produces strict,
  source-faithful structured output on the free tier.
- Gemini Embeddings creates vectors for semantic ranking on the free tier.
- OpenAI Responses and Embeddings remain an optional paid alternative.

The frontend and backend are in one deployable project. Browser requests use /api,
so no separate Express server or CORS configuration is required.


5. IMPORTANT FOLDERS AND FILES
------------------------------

app/
  Page routes and API routes. app/page.tsx is the landing page.
  app/workspace/page.tsx is the signed-in product shell.

app/api/
  The real backend HTTP endpoints. Every protected endpoint creates an authenticated
  request context before reading or changing workspace data.

components/
  Shared visual building blocks: top bar, sidebar, status, upload dialog, command
  palette, and empty states.

features/
  Product areas grouped by purpose: dashboard, documents, search, assistant,
  schemas, analytics, settings, and the landing tour.

services/api-client.ts
  The one browser API gateway. UI components call this file instead of constructing
  unrelated fetch requests in many places.

stores/workspace-store.ts
  Short-lived interface state such as the current navigation section, theme,
  selected document, page number, and open dialogs.

types/index.ts
  Shared browser-side TypeScript contracts for documents, snapshots, fields,
  results, citations, and analytics.

server/auth.ts
  Turns trusted hosting headers or the local development identity into a user,
  workspace, membership, and role. It also checks document ownership and write roles.

server/ai.ts
  Analyzes compact document metadata, creates rate-aware grouped embeddings, and
  answers only from page-linked source passages through Gemini or OpenAI.

server/document-content.ts
  Uses unpdf to read every original PDF page locally, measures readable-page
  coverage, rejects image-only scans that need OCR, and produces searchable chunks.

server/text-chunking.ts
  Normalizes extracted text and splits long pages into overlapping passages so a
  fact crossing a chunk boundary is not silently lost.

server/documents.ts
  Runs actual document processing, persists pages/chunks/fields, changes job state,
  records audit activity, and saves actionable failures.

server/search.ts
  Loads only chunks from the active workspace, applies semantic similarity when
  embeddings exist, and uses honest keyword ranking when they do not.

server/runtime.ts
  Reads D1, R2, AI model configuration, and local identity from the runtime.

next.config.ts
  Raises vinext's multipart/Server Action request envelope from its 1 MB default to
  100 MB. The upload API still applies the stricter product rules: 25 MB per file
  and no more than 10 files. Without this framework setting, a 1.3 MB book can be
  rejected before Finora's upload route gets a chance to validate or store it.

server/serializers.ts
  Converts raw database rows into stable, human-friendly API responses.

db/schema.ts
  The complete relational data model.

drizzle/
  Generated SQL migrations. These files create the database in a repeatable way.

worker/index.ts
  The production Cloudflare Worker entry point. It serves application routes and
  image optimization.

.openai/hosting.json
  Declares the hosted project plus its DB and DOCUMENTS bindings.

wrangler.jsonc
  Defines matching local D1/R2 resources and the migration directory.

.env.example
  A safe template of environment variable names. It contains no secret value.

aboutme.txt
  A dated implementation and verification log, updated as backend work progresses.


6. DATABASE TABLES
------------------

users
  One row per authenticated person: ID, email, display name, timestamps.

workspaces
  A separate data boundary for a team. created_by records the creator.

memberships
  Connects users to workspaces and assigns owner, admin, reviewer, member, or viewer.

documents
  File metadata: workspace, uploader, R2 key, MIME type, size, detected type,
  processing state, pages, confidence, favorite flag, and failure information.

processing_jobs
  Real processing state: stage, progress, attempts, start/end times, and errors.

document_pages
  Readable text grouped by original page number.

document_chunks
  Smaller searchable passages with workspace, document, page, section, content,
  and an optional JSON-encoded embedding.

extraction_schemas
  Workspace extraction-template headers such as Invoice and its version.

schema_fields
  Ordered field definitions: name, type, required flag, and description.

extraction_fields
  Values found in one document, including confidence, page, provenance, and verifier.

activities
  Audit-friendly workspace events for uploads, processing, AI, and human review.

assistant_queries
  Successfully generated questions, answers, and their citations.

Foreign keys prevent orphan records. Workspace indexes make the actual application
queries efficient. Unique indexes stop duplicate memberships, page numbers, jobs,
schemas, and extracted field keys.


7. AUTHENTICATION AND AUTHORIZATION
-----------------------------------

In hosted production, the platform supplies trusted headers:

- oai-authenticated-user-id
- oai-authenticated-user-email
- oai-authenticated-user-full-name

The backend reads these headers; the browser is not allowed to choose a workspace
ID or user role. On the first request, the backend upserts the user. If that user has
no membership, it creates "My Workspace", owner membership, and a default invoice
schema.

For local development, .env.local provides LOCAL_DEV_USER_EMAIL and
LOCAL_DEV_USER_NAME. This avoids building a fake login screen while still exercising
the same persisted user/workspace/membership checks.

Every document-specific route verifies both the document ID and active workspace.
Viewer roles cannot upload, edit fields, change schemas, or retry processing.


8. COMPLETE UPLOAD AND PROCESSING FLOW
--------------------------------------

1. The user opens Upload documents.
2. The browser validates extension, size (25 MB maximum), and a maximum of 10 files.
3. services/api-client.ts sends multipart/form-data to the upload API. The 100 MB
   framework envelope lets ordinary document requests reach the application; the
   API then enforces the documented per-file and file-count rules.
4. The backend authenticates the request and checks write permission.
5. The backend validates again. Browser validation is helpful but never trusted.
6. A random document ID and workspace-scoped storage key are created.
7. Original bytes are written to the DOCUMENTS R2 bucket.
8. A document row, processing-job row, and upload activity are inserted in D1.
9. The processing service reads the bytes back from R2.
10. For PDF, unpdf reads every original page using its Cloudflare-compatible PDF.js
    engine. TXT is split by form-feed page boundaries. This extraction does not ask
    Gemini to summarize the file and therefore cannot silently shrink a whole book
    into a few AI-selected snippets.
11. Each original page is normalized and saved, including blank pages. Pages with
    text are divided into passages of about 1,600 characters with 240 characters of
    overlap. The overlap preserves sentences and facts near passage boundaries.
12. The backend checks extraction coverage. A mostly image-only/scanned PDF is not
    marked complete with missing content; it returns instructions to run OCR first.
13. Gemini or OpenAI receives a small spread of page samples only for classification,
    a concise description, and useful fields. That metadata response is never used
    as the full search index.
14. Gemini embeddings combine up to five neighboring passages under a safe character
    limit. One group vector is attached back to each member passage. This covers a
    book within far fewer free-tier embedding operations. If quota is unavailable,
    full keyword search remains usable and processing still stores every page.
15. Page rows are inserted in batches of 20 and chunk rows in batches of 10 so D1's
    SQL parameter limit is not exceeded by long books.
16. Retry clears old processing output and rebuilds the page/chunk/field set in small
    batches. A failed rebuild remains visibly Failed and can be retried safely.
17. Coverage below 80 percent or low-confidence fields puts the document in Review.
    Coverage below 50 percent fails with OCR guidance instead of false success.
18. The activity log records total pages, readable pages, coverage, characters,
    chunks, embedded chunks, and fields without storing document text in logs.
19. TanStack Query refreshes the workspace so the persisted result appears.

If processing throws, the original file remains stored, document/job rows become
failed, the exact safe message is saved, and the UI offers Retry processing. Retry
uses the same authorized pipeline and increments the attempt count.


9. SEARCH FLOW
--------------

1. The user enters at least two characters.
2. The backend restricts candidate chunks to the active workspace, selected document,
   and optional type. It evaluates every stored chunk; there is no old 300-chunk
   cutoff that could exclude later chapters.
3. With embeddings, it compares the query with grouped passage vectors by cosine
   similarity and combines semantic and lexical relevance.
4. Without embeddings, it counts honest query-term matches across the complete
   document. Common question words are removed and simple word endings are reduced
   so "hire", "hired", and similar forms rank more consistently. Free-tier quota
   failure never deletes the extracted page index.
5. Results include document ID/name, original page, section, passage, type, and score.
6. Clicking a result selects that document and opens its source page.


10. AI ASSISTANT AND CITATIONS
------------------------------

1. The question is validated (3 to 2,000 characters).
2. If a document is selected, workspace access to it is verified.
3. The backend classifies the question as focused, whole-document overview, or
   ending analysis. A focused question retrieves up to 12 relevant passages from
   every indexed page.
4. A broad prompt containing terms such as summarize, synopsis, overview, whole
   book, or main themes loads the complete ordered book index (within the provider's
   safe context budget) so the answer represents the whole work. Ending phrases
   such as "explain the ending," "final twist," "epilogue," "last chapter," and
   "what really happened" also load the complete book instead of relying on a few
   keyword matches.
5. Only permitted page-linked passages and the question are sent to the AI service.
   If a focused query has no lexical or semantic match at all, the assistant falls
   back to the complete selected-document context instead of failing prematurely.
   It also performs that full-context second pass when the focused answer contains
   no valid citation, protecting paraphrased questions from a weak first retrieval.
6. Strict output requires an answer and source indexes. By default, focused answers
   target 250 to 600 words, full summaries target 700 to 1,200 words, and ending
   explanations target 600 to 1,000 words. An explicit request for a brief answer
   overrides those targets.
7. Ending answers must walk through the decisive events, reveal or twist, motives,
   consequences, conflicting evidence, and any deliberate ambiguity. Confirmed
   source facts must be separated from interpretation.
8. If a cited answer is still below its mode's minimum depth, the backend makes one
   controlled rewrite request asking for a fuller explanation without repetition.
   Unsupported questions may remain short because the system must not pad a refusal
   with invented details.
9. The prompt may correct an incorrectly named role, but it may never invent an
   unsupported relationship. The provider has a 6,144-token output budget so it is
   not accidentally forced into a tiny response.
10. Invalid indexes are removed and citations are capped at 6 for focused answers or
    12 for overviews/endings so the interface remains useful.
11. The successful answer and citations are saved to assistant_queries.
12. Paragraph breaks from detailed answers are preserved on screen. Each source
    button opens the exact stored document page.

When the selected provider key is absent, the endpoint returns HTTP 503 with setup
guidance. There is no local fabricated answer fallback.


11. EXTRACTION AND SCHEMA EDITING
---------------------------------

Opening a document calls its detail endpoint, so the extracted fields always belong
to that selected document. Editing a value changes its provenance to modified.
Saving performs validated upserts keyed by document plus field key and creates an
activity record.

The Schema page edits an ordinary visual form. Saving validates 1 to 50 fields,
replaces the ordered Invoice field set, increments the schema version, and records
the activity. Returned IDs are the same IDs stored in D1.


12. ANALYTICS
-------------

Analytics are calculated from persisted workspace rows:

- processed documents
- completed/review success percentage
- average processing duration
- documents needing review
- average extraction confidence
- original storage bytes
- daily upload throughput

No chart number is hardcoded in the frontend.


13. API REFERENCE
-----------------

GET  /api/health
  Service readiness and whether AI is configured.

GET  /api/workspaces/current/snapshot
  Current user/workspace, documents, activity, pipeline, latest fields, and schema.

POST /api/workspaces/current/documents
  Validated multipart upload, R2 storage, processing, and persisted result.

GET  /api/workspaces/current/documents/:id
  One authorized document plus its own extracted fields.

GET  /api/workspaces/current/documents/:id/file
  Original R2 object with inline content headers and private no-store caching.

POST /api/workspaces/current/documents/:id/retry
  Authorized retry of a failed or reprocessable document.

PUT  /api/workspaces/current/documents/:id/extraction
  Validate and save reviewed/modified extraction fields.

GET  /api/workspaces/current/search?q=...
  Workspace-scoped semantic/keyword passage results.

POST /api/workspaces/current/assistant
  Source-constrained cited answer.

PUT  /api/workspaces/current/schemas/invoice
  Validate and save the Invoice extraction field template.

GET  /api/workspaces/current/analytics
  Live workspace metrics.

Errors use JSON with message and code. Unknown internal details are not leaked.


14. LOCAL DATA AND MIGRATIONS
-----------------------------

Local D1 and R2 emulator state is under .wrangler/. It is ignored by Git.

To apply new migrations after schema changes:

   npm run db:generate
   npm run db:migrate:local

Do not manually edit generated migration snapshots. Change db/schema.ts first and
generate the migration.

Deleting .wrangler removes local development data. That is destructive and cannot
be undone, so stop the server and back up anything important before doing it.


15. QUALITY CHECKS
------------------

Run:

   npx tsc --noEmit
   npm run lint
   npm test

npx tsc checks TypeScript. npm run lint checks React, unused values, and code
quality. npm test makes a production build, checks important rendered HTML, creates
a two-page PDF in memory, verifies that both pages are extracted separately, and
checks that long overlapping chunks retain their beginning and ending.

Manual primary journey:

1. Open /workspace.
2. Upload tests/fixtures/sample-agreement.txt.
3. Confirm Completed appears.
4. Open the document and confirm the original text loads.
5. Search for "termination" and open the result.
6. Add/edit an extracted field and save it.
7. Open Analytics and confirm persisted counts.
8. With GEMINI_API_KEY configured, ask a question and open its citation.
9. Without the key, confirm the assistant gives an actionable configuration error.


16. PRODUCTION HOSTING
----------------------

For a complete beginner-friendly launch sequence, access decisions, secret setup,
private-first deployment, smoke testing, security checks, monitoring, rollback, and
scaling plan, read HOSTING_ROADMAP.TXT in this project.

The project is prepared for OpenAI Sites/Cloudflare hosting through
.openai/hosting.json. Production needs:

- the hosted D1 binding named DB
- the hosted R2 binding named DOCUMENTS
- the D1 migration applied to the production database
- GEMINI_API_KEY stored as a server secret when free-tier AI features are wanted
- authenticated hosting enabled so trusted user headers are supplied

Do not upload/deploy source code or create paid resources merely by running the
local app. Publishing is a separate external action and should be done only after
the owner approves the target project and secret configuration.


17. CURRENT ARCHITECTURE BOUNDARIES
-----------------------------------

This repository originally had no PostgreSQL, Redis, BullMQ, Python service, or
WebSocket server to preserve. The working implementation uses its existing native
Sites/Cloudflare runtime: D1, R2, route handlers, and server-side AI-provider calls.

Processing currently completes inside the upload/retry request. Its stages are real
persisted job states, but it is not a detached queue worker. NEXT_PUBLIC_SOCKET_URL
is optional support for an external authenticated Socket.IO event source; the local
product refreshes server state after mutations and does not simulate socket events.

For very large production workloads, the deliberate next scaling step is a managed
queue/consumer that calls the same processing service and publishes authenticated
workspace events. D1 chunk embeddings are suitable for this portfolio-scale product;
a dedicated vector index is the next step for millions of passages.


18. TROUBLESHOOTING
-------------------

"npm is not recognized"
  Install Node.js, restart VS Code, and open a new terminal.

"No connection could be made to localhost:3000"
  Run npm run dev and keep that terminal open.

"no such table"
  Run npm run db:migrate:local, then restart npm run dev.

"AI document processing is not configured"
  Create a Gemini key, add GEMINI_API_KEY to .env.local, and restart the server.
  TXT search still works without the key.

"Gemini free-tier quota is exhausted"
  Wait for Google's quota window to reset, reduce requests, or choose a paid plan.
  Finora does not silently charge money or switch providers. PDF pages remain
  fully stored and keyword-searchable even when an embedding window is unavailable.

"Only N of M PDF pages contain extractable text"
  The PDF is probably a scan made from page images. Run OCR in Adobe Acrobat,
  Microsoft Lens, Google Drive, or another trusted OCR tool, export a searchable
  PDF, and upload it again. Finora refuses false completion when most page text
  cannot actually be read.

"Gemini mode currently supports PDF and TXT"
  Open the DOC/DOCX file in Word or Google Docs, export/download it as PDF, and
  upload that PDF.

"The document was not found or you do not have access"
  The selected ID is absent or belongs to another workspace. Return to Documents
  and open a file visible in the current workspace.

Upload rejected
  Use PDF, DOC, DOCX, or TXT; keep each file at or below 25 MB and each batch at or
  below 10 files. The whole multipart request must also stay below the 100 MB
  framework envelope, so split a very large batch into multiple uploads.

"Payload Too Large" or HTTP 413
  Restart npm run dev so the current next.config.ts is loaded. If the error remains,
  upload fewer files in one batch. The browser now displays this server reason
  instead of replacing it with a generic upload failure.

Changes do not appear
  Refresh once. In development, also check the terminal for a compilation/runtime
  error and restart npm run dev after changing environment values.


19. SECURITY CHECKLIST
----------------------

- Server secrets never use NEXT_PUBLIC prefixes.
- Protected routes authenticate before database or object reads.
- Document access is always workspace-scoped.
- Write operations enforce roles.
- File extension, count, and size are validated on server and client.
- R2 file responses are private and no-store.
- AI receives files from the server, not an exposed browser key.
- AI output is schema-constrained and citations are bounded to retrieved sources.
- Successful and failed important actions produce persisted activity records.
- No fake success fallback hides missing storage, database, authentication, or AI.


20. ONE-SENTENCE MENTAL MODEL
-----------------------------

The React interface sends authorized requests to the same Cloudflare application;
the backend stores file bytes in R2, structured truth in D1, uses the configured
Gemini or OpenAI provider only when its private key is present, and returns
source-linked persisted results for the interface to render.