Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/document-understanding.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,31 @@ spending and scheduling limits, durable manifests, job orchestration, vector
storage, and search serving. Docbank does not compute, store, or serve
embeddings for its own vault.

## Convert CSV locally for PDF OCR

Go applications can use `document/csvpdf` to convert an explicitly declared `text/csv` source to a bounded PDF before using the existing PDF OCR path. Conversion runs locally and closes the supplied `ocr.Source.Content` on every path. It verifies the original byte count and SHA-256, parses CSV records, labels every cell, and counts the generated PDF pages before returning any result. It performs no upload or credential lookup.

```go
policy, err := csvpdf.NewPolicy(csvpdf.DefaultLimits())
if err != nil {
return err
}
converted, err := csvpdf.Convert(ctx, originalCSV, policy)
if err != nil {
return err
}
receipt := converted.Receipt()
pdfSource, err := converted.Source()
if err != nil {
return err
}
// Retain receipt with the source and pass pdfSource to the authorized PDF processor.
```

Defaults are also hard ceilings. A conversion accepts at most 1 MiB of source bytes, 10 MiB of PDF bytes, 10,000 records, 50,000 cells, 64 KiB per cell, and 100 generated pages. `NewPolicy` accepts positive tighter limits. The embedded Go Mono font supports a subset of Latin, Greek, Cyrillic and common punctuation or symbols. Conversion rejects missing glyphs, characters outside the Unicode basic multilingual plane, combining marks, shaping scripts, formatting controls, and control characters except cell newlines. Use precomposed accented characters such as `é`. CSV quoting, blank lines and CRLF normalization follow Go's `encoding/csv`; empty cells, variable-width records and quoted newlines remain represented. Long cells continue on subsequent generated pages. CSV formulas remain literal text.

Retain both original and generated hashes and byte counts, the converter version, the conversion policy fingerprint, and the receipt's generated-page to original-record/cell spans. All span identifiers are one-based. A record can span multiple generated pages. PDF OCR evidence refers to generated pages, so applications must join that evidence through the receipt to cite original CSV cells. The receipt does not authorize upload. The application must separately verify its PDF capability manifest and consent, choose PDF byte and page limits that accommodate the conversion, and retain the conversion policy with that consent. This Go API does not add CSV OCR to the daemon or CLI.

## Package boundary

The dependency direction is deliberate:
Expand Down
13 changes: 13 additions & 0 deletions docs/internal/csv-pdf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# CSV to PDF conversion

`document/csvpdf.Convert` owns local conversion from an `ocr.Source` declared as `text/csv` to independently identified PDF bytes. It imports neither vault storage nor a provider client. Applications compose `Result.Source` with the existing PDF OCR path and retain the conversion receipt separately from provider evidence and upload consent.

An immutable `Policy` captures six positive limits bounded by fixed hard ceilings. Its canonical JSON fingerprint includes those limits, the converter version, fixed layout identity and embedded font hash. Changes to parsing, layout, font or the pinned PDF writer require a converter version change. The zero policy cannot authorize conversion.

Conversion reads at most the declared source length plus one byte, bounded by the source ceiling, and recomputes SHA-256 before parsing. Go's CSV reader owns quoting, escaped quotes, blank-line handling and CRLF normalization. Parsing accepts variable-width records and bounds record count, cell count and per-cell bytes. Source byte limits also bound parser allocations before those finer limits can be checked. Conversion closes its input on every path. Cancellation stops between reads and during parsing and layout; an arbitrary blocking reader remains the caller's responsibility. A close failure discards a would-be result.

Layout uses embedded Go Mono at 10 points on A4 pages, with 80 characters per content line and 48 lines per page. Each cell has a record/cell label followed by its value, including empty values and explicit empty lines. Labels consume layout lines. The renderer accepts only font-supported basic multilingual plane characters in Latin, Greek, Cyrillic or Common scripts, rejecting marks, formatting and control characters except newlines. This avoids silent glyph replacement or incorrect shaping. A fixed page heading identifies generated pages. Page spans record every page containing a cell label or value, including continuation pages.

The pinned fpdf writer uses a private embedded-font copy, explicit compression, fixed timestamps, sorted catalogs and disabled automatic page breaks. Layout checks the page limit before PDF creation. The output writer enforces the PDF byte limit, then `media.CountPDFPages` independently checks the generated object graph and its count against the planned pages. The PDF library builds an internal buffer before writing; source and page hard ceilings bound that work, while the output byte limit bounds retained output. Applications must bound simultaneous conversions to control process memory.

Only a complete counted PDF yields a `Result`. Its receipt binds original and generated SHA-256 hashes, both byte counts, generated page count, policy fingerprint, converter version and page/record/cell spans. `PDF`, `Receipt` and `Source` return independent copies, so callers cannot mutate the authoritative bytes or mapping. Generated PDF pages never become an inferred count of original CSV records. A receipt supplies provenance, while the existing PDF manifest and application consent supply upload authority.
123 changes: 123 additions & 0 deletions document/csvpdf/convert.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
package csvpdf

import (
"bytes"
"context"
"encoding/csv"
"errors"
"fmt"
"io"
"unicode/utf8"

"go.kenn.io/docbank/document/media"
"go.kenn.io/docbank/document/ocr"
)

// Convert consumes and closes source.Content on every path, without network I/O.
// Cancellation is checked between reads; arbitrary blocking readers cannot be interrupted.
func Convert(ctx context.Context, source ocr.Source, policy Policy) (result *Result, err error) {
if source.Content != nil {
defer func() {
if closeErr := source.Content.Close(); closeErr != nil {
result = nil
err = errors.Join(err, errors.New("close CSV source failed"))
}
}()
}
if err := ctx.Err(); err != nil {
return nil, err
}
if policy.fingerprint == "" {
return nil, errors.New("CSV PDF policy is invalid; use NewPolicy")
}
if err := source.Validate(); err != nil {
return nil, err
}
if source.MediaType != "text/csv" {
return nil, errors.New("CSV source requires text/csv")
}
if source.Size > policy.limits.MaxSourceBytes {
return nil, errors.New("CSV source exceeds byte limit")
}
content, err := io.ReadAll(io.LimitReader(contextReader{ctx, source.Content}, source.Size+1))
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return nil, ctxErr
}
return nil, errors.New("read CSV source failed")
}
if int64(len(content)) != source.Size || digest(content) != source.SHA256 {
return nil, errors.New("CSV source does not match declared size and SHA-256")
}
if !utf8.Valid(content) {
return nil, errors.New("CSV source is not UTF-8")
}
records, err := parse(ctx, content, policy.limits)
if err != nil {
return nil, err
}
pdf, spans, pages, err := render(ctx, records, policy.limits)
if err != nil {
return nil, err
}
count, err := media.CountPDFPages(pdf)
if err != nil {
return nil, fmt.Errorf("verify generated PDF: %w", err)
}
if count != int64(pages) || count > int64(policy.limits.MaxPages) {
return nil, errors.New("generated PDF page count does not match layout")
}
if err := ctx.Err(); err != nil {
return nil, err
}
return &Result{pdf: pdf, receipt: Receipt{SourceSHA256: source.SHA256, SourceBytes: source.Size, PDFSHA256: digest(pdf), PDFBytes: int64(len(pdf)), Pages: pages, PolicyFingerprint: policy.fingerprint, ConverterVersion: ConverterVersion, Spans: spans}}, nil
}

type contextReader struct {
ctx context.Context
reader io.Reader
}

func (r contextReader) Read(p []byte) (int, error) {
if err := r.ctx.Err(); err != nil {
return 0, err
}
n, err := r.reader.Read(p)
if ctxErr := r.ctx.Err(); ctxErr != nil {
return n, ctxErr
}
return n, err
}

func parse(ctx context.Context, content []byte, limits Limits) ([][]string, error) {
reader := csv.NewReader(bytes.NewReader(content))
reader.FieldsPerRecord = -1
var records [][]string
cells := 0
for {
if err := ctx.Err(); err != nil {
return nil, err
}
record, err := reader.Read()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, errors.New("CSV source has invalid record syntax")
}
if len(records) == limits.MaxRecords || len(record) > limits.MaxCells-cells {
return nil, errors.New("CSV source exceeds record or cell limit")
}
for _, cell := range record {
if len(cell) > limits.MaxCellBytes {
return nil, errors.New("CSV source exceeds cell byte limit")
}
}
cells += len(record)
records = append(records, record)
}
if len(records) == 0 {
return nil, errors.New("CSV source has no records")
}
return records, nil
}
Loading