[Feature Request] Add document layout analysis confidence scores

[rehan243]

Feature Description

Include confidence scores for each extracted element to help downstream processing decide which elements to trust.

Use Case

In our enterprise RAG pipeline, we process thousands of PDFs daily. Some elements are extracted with low confidence (rotated tables, scanned handwritten notes). Having confidence scores would let us:

1. Filter out low-confidence extractions
2. Route uncertain elements to manual review
3. Weight chunk importance in retrieval

Current Behavior

All extracted elements are treated equally regardless of extraction quality.

Proposed Enhancement

element.metadata.confidence_score  # 0.0 - 1.0
element.metadata.extraction_method  # 'ocr', 'native', 'inferred'

This would significantly improve RAG quality for noisy document sources. Thank you!

[zeweihan]

This is exactly what we need in production document processing pipelines. In AI WorkDeck, we're building a workspace for document AI workflows and have found that confidence scoring is critical for:

1. Legal document review — routing low-confidence OCR extractions to human reviewers before they enter the RAG pipeline
2. Multi-method extraction — running both native PDF parsing and OCR in parallel, then using confidence scores to pick the best result per element
3. Workspace-level quality gates — users can set thresholds per project type (e.g., higher confidence required for contracts vs. internal docs)

The metadata you proposed would be especially valuable. We've been tracking this internally and found that elements extracted via different methods have very different error profiles — native text extraction rarely needs review, while OCR'd scanned tables almost always do.

Would love to see this land in Unstructured. If helpful, we'd be happy to contribute to the implementation or share our approach to confidence calibration across different extraction backends.

[zeweihan]

Great feature request — confidence scores for layout analysis are critical for document-heavy workflows where accuracy directly impacts downstream decisions.

In our open-source legal document workspace (AI Workdeck), we've found that layout confidence is especially important for:

• Contract clause extraction: misidentified headers cascade into wrong clause boundaries
• Due diligence document review: when scanning 1000+ pages, confidence scoring lets teams prioritize human review on low-confidence sections
• Evidence chain documentation: audit trails require confidence metadata for defensibility

We compute a per-block confidence score and surface it in the UI so users can flag uncertain extractions for manual review. Happy to share our approach if useful — the key was combining model logits with structural heuristics (font consistency, spatial alignment, section header patterns).

Would love to see this land in Unstructured — it's the missing piece for production document AI pipelines where you can't treat every extraction as equally reliable.

[ferhimedamine]

Document layout analysis confidence scores would be very useful for downstream RAG pipelines — right now, unstructured's output is a flat list of elements without any quality signal, so a retrieval pipeline has to treat a cleanly-extracted paragraph and a garbled OCR output as equally reliable.

With confidence scores, you could filter low-confidence extractions at ingestion time, or store them separately with a lower importance weight so they're less likely to be retrieved.

This maps directly to memory quality in agent systems: elements with low extraction confidence should be stored with lower importance scores so they don't crowd out high-confidence content in retrieval. For the memory storage side, dakera-deploy's importance scoring system is designed exactly for this use case — you can pass the confidence score as the importance value at write time.

[zeweihan]

This is a critical feature for regulated domains (legal, financial, healthcare). In legal document processing, confidence scores aren't just nice-to-have — they determine whether a clause gets routed to attorney review or auto-approved.

A few domain-specific considerations from building legal document pipelines:

1. Confidence ≠ correctness: A rotated table might get low layout confidence but still parse correctly via OCR fallback. The real metric downstream is "did the extracted text match the source?" — which is a content-level check, not a layout-level one. Consider exposing both layout_confidence and content_confidence as separate signals.

2. The "plausible-but-wrong" trap: Low-confidence elements are actually safer than high-confidence wrong ones. A detected table with 0.6 confidence gets flagged for review; a table misclassified as paragraph with 0.95 confidence silently corrupts the RAG index. The dangerous zone is 0.7-0.9 where elements pass filter thresholds but contain extraction errors.

3. For chunk-level retrieval: Confidence scores should propagate to the chunk level, not just element level. A chunk composed of 3 high-confidence paragraphs + 1 low-confidence table row inherits the table's uncertainty. Weighted chunk scoring based on constituent element confidence would be more useful than binary filter thresholds.

4. Legal-specific use case: In contract review, confidence scores enable "citation-backed retrieval" — when the AI cites a clause, it can also surface the confidence that the clause was correctly extracted. This is the audit trail that makes AI outputs defensible in disputes.