diff --git a/docs/document-understanding.md b/docs/document-understanding.md index aa9b1125..5e58e530 100644 --- a/docs/document-understanding.md +++ b/docs/document-understanding.md @@ -265,6 +265,31 @@ spending and scheduling limits, durable manifests, job orchestration, vector storage, and search serving. Docbank does not compute, store, or serve embeddings for its own vault. +## Convert CSV locally for PDF OCR + +Go applications can use `document/csvpdf` to convert an explicitly declared `text/csv` source to a bounded PDF before using the existing PDF OCR path. Conversion runs locally and closes the supplied `ocr.Source.Content` on every path. It verifies the original byte count and SHA-256, parses CSV records, labels every cell, and counts the generated PDF pages before returning any result. It performs no upload or credential lookup. + +```go +policy, err := csvpdf.NewPolicy(csvpdf.DefaultLimits()) +if err != nil { + return err +} +converted, err := csvpdf.Convert(ctx, originalCSV, policy) +if err != nil { + return err +} +receipt := converted.Receipt() +pdfSource, err := converted.Source() +if err != nil { + return err +} +// Retain receipt with the source and pass pdfSource to the authorized PDF processor. +``` + +Defaults are also hard ceilings. A conversion accepts at most 1 MiB of source bytes, 10 MiB of PDF bytes, 10,000 records, 50,000 cells, 64 KiB per cell, and 100 generated pages. `NewPolicy` accepts positive tighter limits. The embedded Go Mono font supports a subset of Latin, Greek, Cyrillic and common punctuation or symbols. Conversion rejects missing glyphs, characters outside the Unicode basic multilingual plane, combining marks, shaping scripts, formatting controls, and control characters except cell newlines. Use precomposed accented characters such as `é`. CSV quoting, blank lines and CRLF normalization follow Go's `encoding/csv`; empty cells, variable-width records and quoted newlines remain represented. Long cells continue on subsequent generated pages. CSV formulas remain literal text. + +Retain both original and generated hashes and byte counts, the converter version, the conversion policy fingerprint, and the receipt's generated-page to original-record/cell spans. All span identifiers are one-based. A record can span multiple generated pages. PDF OCR evidence refers to generated pages, so applications must join that evidence through the receipt to cite original CSV cells. The receipt does not authorize upload. The application must separately verify its PDF capability manifest and consent, choose PDF byte and page limits that accommodate the conversion, and retain the conversion policy with that consent. This Go API does not add CSV OCR to the daemon or CLI. + ## Package boundary The dependency direction is deliberate: diff --git a/docs/internal/csv-pdf.md b/docs/internal/csv-pdf.md new file mode 100644 index 00000000..fb8aa64f --- /dev/null +++ b/docs/internal/csv-pdf.md @@ -0,0 +1,13 @@ +# CSV to PDF conversion + +`document/csvpdf.Convert` owns local conversion from an `ocr.Source` declared as `text/csv` to independently identified PDF bytes. It imports neither vault storage nor a provider client. Applications compose `Result.Source` with the existing PDF OCR path and retain the conversion receipt separately from provider evidence and upload consent. + +An immutable `Policy` captures six positive limits bounded by fixed hard ceilings. Its canonical JSON fingerprint includes those limits, the converter version, fixed layout identity and embedded font hash. Changes to parsing, layout, font or the pinned PDF writer require a converter version change. The zero policy cannot authorize conversion. + +Conversion reads at most the declared source length plus one byte, bounded by the source ceiling, and recomputes SHA-256 before parsing. Go's CSV reader owns quoting, escaped quotes, blank-line handling and CRLF normalization. Parsing accepts variable-width records and bounds record count, cell count and per-cell bytes. Source byte limits also bound parser allocations before those finer limits can be checked. Conversion closes its input on every path. Cancellation stops between reads and during parsing and layout; an arbitrary blocking reader remains the caller's responsibility. A close failure discards a would-be result. + +Layout uses embedded Go Mono at 10 points on A4 pages, with 80 characters per content line and 48 lines per page. Each cell has a record/cell label followed by its value, including empty values and explicit empty lines. Labels consume layout lines. The renderer accepts only font-supported basic multilingual plane characters in Latin, Greek, Cyrillic or Common scripts, rejecting marks, formatting and control characters except newlines. This avoids silent glyph replacement or incorrect shaping. A fixed page heading identifies generated pages. Page spans record every page containing a cell label or value, including continuation pages. + +The pinned fpdf writer uses a private embedded-font copy, explicit compression, fixed timestamps, sorted catalogs and disabled automatic page breaks. Layout checks the page limit before PDF creation. The output writer enforces the PDF byte limit, then `media.CountPDFPages` independently checks the generated object graph and its count against the planned pages. The PDF library builds an internal buffer before writing; source and page hard ceilings bound that work, while the output byte limit bounds retained output. Applications must bound simultaneous conversions to control process memory. + +Only a complete counted PDF yields a `Result`. Its receipt binds original and generated SHA-256 hashes, both byte counts, generated page count, policy fingerprint, converter version and page/record/cell spans. `PDF`, `Receipt` and `Source` return independent copies, so callers cannot mutate the authoritative bytes or mapping. Generated PDF pages never become an inferred count of original CSV records. A receipt supplies provenance, while the existing PDF manifest and application consent supply upload authority. diff --git a/document/csvpdf/convert.go b/document/csvpdf/convert.go new file mode 100644 index 00000000..78639e44 --- /dev/null +++ b/document/csvpdf/convert.go @@ -0,0 +1,123 @@ +package csvpdf + +import ( + "bytes" + "context" + "encoding/csv" + "errors" + "fmt" + "io" + "unicode/utf8" + + "go.kenn.io/docbank/document/media" + "go.kenn.io/docbank/document/ocr" +) + +// Convert consumes and closes source.Content on every path, without network I/O. +// Cancellation is checked between reads; arbitrary blocking readers cannot be interrupted. +func Convert(ctx context.Context, source ocr.Source, policy Policy) (result *Result, err error) { + if source.Content != nil { + defer func() { + if closeErr := source.Content.Close(); closeErr != nil { + result = nil + err = errors.Join(err, errors.New("close CSV source failed")) + } + }() + } + if err := ctx.Err(); err != nil { + return nil, err + } + if policy.fingerprint == "" { + return nil, errors.New("CSV PDF policy is invalid; use NewPolicy") + } + if err := source.Validate(); err != nil { + return nil, err + } + if source.MediaType != "text/csv" { + return nil, errors.New("CSV source requires text/csv") + } + if source.Size > policy.limits.MaxSourceBytes { + return nil, errors.New("CSV source exceeds byte limit") + } + content, err := io.ReadAll(io.LimitReader(contextReader{ctx, source.Content}, source.Size+1)) + if err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return nil, ctxErr + } + return nil, errors.New("read CSV source failed") + } + if int64(len(content)) != source.Size || digest(content) != source.SHA256 { + return nil, errors.New("CSV source does not match declared size and SHA-256") + } + if !utf8.Valid(content) { + return nil, errors.New("CSV source is not UTF-8") + } + records, err := parse(ctx, content, policy.limits) + if err != nil { + return nil, err + } + pdf, spans, pages, err := render(ctx, records, policy.limits) + if err != nil { + return nil, err + } + count, err := media.CountPDFPages(pdf) + if err != nil { + return nil, fmt.Errorf("verify generated PDF: %w", err) + } + if count != int64(pages) || count > int64(policy.limits.MaxPages) { + return nil, errors.New("generated PDF page count does not match layout") + } + if err := ctx.Err(); err != nil { + return nil, err + } + return &Result{pdf: pdf, receipt: Receipt{SourceSHA256: source.SHA256, SourceBytes: source.Size, PDFSHA256: digest(pdf), PDFBytes: int64(len(pdf)), Pages: pages, PolicyFingerprint: policy.fingerprint, ConverterVersion: ConverterVersion, Spans: spans}}, nil +} + +type contextReader struct { + ctx context.Context + reader io.Reader +} + +func (r contextReader) Read(p []byte) (int, error) { + if err := r.ctx.Err(); err != nil { + return 0, err + } + n, err := r.reader.Read(p) + if ctxErr := r.ctx.Err(); ctxErr != nil { + return n, ctxErr + } + return n, err +} + +func parse(ctx context.Context, content []byte, limits Limits) ([][]string, error) { + reader := csv.NewReader(bytes.NewReader(content)) + reader.FieldsPerRecord = -1 + var records [][]string + cells := 0 + for { + if err := ctx.Err(); err != nil { + return nil, err + } + record, err := reader.Read() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return nil, errors.New("CSV source has invalid record syntax") + } + if len(records) == limits.MaxRecords || len(record) > limits.MaxCells-cells { + return nil, errors.New("CSV source exceeds record or cell limit") + } + for _, cell := range record { + if len(cell) > limits.MaxCellBytes { + return nil, errors.New("CSV source exceeds cell byte limit") + } + } + cells += len(record) + records = append(records, record) + } + if len(records) == 0 { + return nil, errors.New("CSV source has no records") + } + return records, nil +} diff --git a/document/csvpdf/convert_test.go b/document/csvpdf/convert_test.go new file mode 100644 index 00000000..aa856672 --- /dev/null +++ b/document/csvpdf/convert_test.go @@ -0,0 +1,323 @@ +package csvpdf + +import ( + "context" + "errors" + "fmt" + "io" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" + "go.kenn.io/docbank/document/media" + "go.kenn.io/docbank/document/ocr" +) + +type trackedReader struct { + io.Reader + + closed bool + closeErr error +} + +func (r *trackedReader) Close() error { r.closed = true; return r.closeErr } + +func sourceFor(t *testing.T, content string) (ocr.Source, *trackedReader) { + t.Helper() + reader := &trackedReader{Reader: strings.NewReader(content)} + source, err := ocr.NewSource(reader, "text/csv", int64(len(content)), digest([]byte(content))) + require.NoError(t, err) + return source, reader +} + +func policyFor(t *testing.T, limits Limits) Policy { + t.Helper() + policy, err := NewPolicy(limits) + require.NoError(t, err) + return policy +} + +func TestConvertDeterministicProvenanceAndCopies(t *testing.T) { + content := "name,value,empty\r\n\"café κόσμος Привет\",\"one\ntwo\",\r\nshort\r\n" + policy := policyFor(t, DefaultLimits()) + source, reader := sourceFor(t, content) + result, err := Convert(t.Context(), source, policy) + require.NoError(t, err) + require.True(t, reader.closed) + source, _ = sourceFor(t, content) + again, err := Convert(t.Context(), source, policy) + require.NoError(t, err) + require.Equal(t, result.PDF(), again.PDF()) + receipt := result.Receipt() + require.Equal(t, digest([]byte(content)), receipt.SourceSHA256) + require.Equal(t, digest(result.PDF()), receipt.PDFSHA256) + require.NotEqual(t, receipt.SourceSHA256, receipt.PDFSHA256) + require.Equal(t, policy.Fingerprint(), receipt.PolicyFingerprint) + require.Equal(t, int64(len(content)), receipt.SourceBytes) + require.Equal(t, int64(len(result.PDF())), receipt.PDFBytes) + require.Equal(t, []Span{{1, 1, 1}, {1, 1, 2}, {1, 1, 3}, {1, 2, 1}, {1, 2, 2}, {1, 2, 3}, {1, 3, 1}}, receipt.Spans) + pdfCopy := result.PDF() + pdfCopy[0] = 0 + receipt.Spans[0].Page = 99 + receipt.PDFSHA256 = "changed" + require.Equal(t, again.PDF(), result.PDF()) + require.Equal(t, again.Receipt(), result.Receipt()) + first, err := result.Source() + require.NoError(t, err) + second, err := result.Source() + require.NoError(t, err) + defer func() { require.NoError(t, first.Content.Close()); require.NoError(t, second.Content.Close()) }() + _, err = first.Content.Read(make([]byte, 10)) + require.NoError(t, err) + all, err := io.ReadAll(second.Content) + require.NoError(t, err) + require.Equal(t, result.PDF(), all) + require.Equal(t, "application/pdf", second.MediaType) + third, err := result.Source() + require.NoError(t, err) + _, err = io.Copy(mutatingWriter{}, third.Content) + require.NoError(t, err) + require.NoError(t, third.Content.Close()) + require.Equal(t, again.PDF(), result.PDF()) + require.Equal(t, again.Receipt(), result.Receipt()) +} + +type mutatingWriter struct{} + +func (mutatingWriter) Write(p []byte) (int, error) { + clear(p) + return len(p), nil +} + +func TestPolicyBoundsAndIdentity(t *testing.T) { + setters := []func(*Limits, int64){ + func(l *Limits, v int64) { l.MaxSourceBytes = v }, func(l *Limits, v int64) { l.MaxPDFBytes = v }, + func(l *Limits, v int64) { l.MaxRecords = int(v) }, func(l *Limits, v int64) { l.MaxCells = int(v) }, + func(l *Limits, v int64) { l.MaxCellBytes = int(v) }, func(l *Limits, v int64) { l.MaxPages = int(v) }, + } + defaults := DefaultLimits() + maxima := []int64{defaults.MaxSourceBytes, defaults.MaxPDFBytes, int64(defaults.MaxRecords), int64(defaults.MaxCells), int64(defaults.MaxCellBytes), int64(defaults.MaxPages)} + base := policyFor(t, defaults) + for index, set := range setters { + for _, value := range []int64{-1, 0, maxima[index] + 1} { + t.Run(fmt.Sprintf("limit_%d/value_%d", index, value), func(t *testing.T) { + limits := defaults + set(&limits, value) + _, err := NewPolicy(limits) + require.Error(t, err) + }) + } + limits := defaults + set(&limits, maxima[index]-1) + changed := policyFor(t, limits) + require.NotEqual(t, base.Fingerprint(), changed.Fingerprint()) + require.Equal(t, changed.Fingerprint(), policyFor(t, limits).Fingerprint()) + } + require.Empty(t, (Policy{}).Fingerprint()) +} + +func TestConcurrentConversionsRemainDeterministic(t *testing.T) { + policy := policyFor(t, DefaultLimits()) + content := "café,κόσμος,Привет\n" + strings.Repeat("x,y,z\n", 40) + source, _ := sourceFor(t, content) + want, err := Convert(t.Context(), source, policy) + require.NoError(t, err) + for index := range 8 { + t.Run(fmt.Sprintf("conversion_%d", index), func(t *testing.T) { + t.Parallel() + source, _ := sourceFor(t, content) + got, err := Convert(t.Context(), source, policy) + require.NoError(t, err) + require.Equal(t, want.PDF(), got.PDF()) + require.Equal(t, want.Receipt(), got.Receipt()) + }) + } +} + +func TestConvertRejectsInvalidSourceAndText(t *testing.T) { + for _, content := range []string{"\n", "\"unclosed", "x\"y", "\xff", "你好", "مرحبا", "a\u0301", "x\tY", "x\rY", "x\u202eY", "x\u0000Y", "😀", "\U00010000"} { + t.Run(digest([]byte(content))[:8], func(t *testing.T) { + source, reader := sourceFor(t, content) + result, err := Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.Error(t, err) + require.Nil(t, result) + require.True(t, reader.closed) + }) + } + for _, change := range []func(*ocr.Source){ + func(s *ocr.Source) { s.Size++ }, func(s *ocr.Source) { s.Size-- }, func(s *ocr.Source) { s.SHA256 = strings.Repeat("0", 64) }, + func(s *ocr.Source) { s.MediaType = "application/pdf" }, func(s *ocr.Source) { s.MediaType = "Text/CSV" }, func(s *ocr.Source) { s.Size = 0 }, + } { + source, reader := sourceFor(t, "x,y\n") + change(&source) + result, err := Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.Error(t, err) + require.Nil(t, result) + require.True(t, reader.closed) + } +} + +func TestConvertBoundaries(t *testing.T) { + for _, test := range []struct { + name, content string + tighten func(*Limits) + }{ + {"source", "ab\n", func(l *Limits) { l.MaxSourceBytes = 3 }}, + {"records", "a\nb\n", func(l *Limits) { l.MaxRecords = 2 }}, + {"cells", "a,b\n", func(l *Limits) { l.MaxCells = 2 }}, + {"cell bytes", "éx\n", func(l *Limits) { l.MaxCellBytes = 3 }}, + } { + t.Run(test.name, func(t *testing.T) { + limits := DefaultLimits() + test.tighten(&limits) + source, _ := sourceFor(t, test.content) + _, err := Convert(t.Context(), source, policyFor(t, limits)) + require.NoError(t, err) + switch test.name { + case "source": + limits.MaxSourceBytes-- + case "records": + limits.MaxRecords-- + case "cells": + limits.MaxCells-- + case "cell bytes": + limits.MaxCellBytes-- + } + source, reader := sourceFor(t, test.content) + result, err := Convert(t.Context(), source, policyFor(t, limits)) + require.Error(t, err) + require.Nil(t, result) + require.True(t, reader.closed) + }) + } + limits := DefaultLimits() + limits.MaxPages = 1 + content := "\"" + strings.Repeat("x\n", 46) + "x\"\n" + source, _ := sourceFor(t, content) + one, err := Convert(t.Context(), source, policyFor(t, limits)) + require.NoError(t, err) + require.Equal(t, 1, one.Receipt().Pages) + source, _ = sourceFor(t, "\""+strings.Repeat("x\n", 47)+"x\"\n") + result, err := Convert(t.Context(), source, policyFor(t, limits)) + require.Error(t, err) + require.Nil(t, result) + limits = DefaultLimits() + limits.MaxPDFBytes = int64(len(one.PDF())) + source, _ = sourceFor(t, content) + _, err = Convert(t.Context(), source, policyFor(t, limits)) + require.NoError(t, err) + limits.MaxPDFBytes-- + source, _ = sourceFor(t, content) + result, err = Convert(t.Context(), source, policyFor(t, limits)) + require.Error(t, err) + require.Nil(t, result) +} + +func TestConvertLongCellMapping(t *testing.T) { + source, _ := sourceFor(t, strings.Repeat("x", columns*linesPerPage)+"\n") + result, err := Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.NoError(t, err) + require.Equal(t, []Span{{1, 1, 1}, {2, 1, 1}}, result.Receipt().Spans) + pages, err := media.CountPDFPages(result.PDF()) + require.NoError(t, err) + require.Equal(t, int64(2), pages) + lines, err := layout(t.Context(), [][]string{{"first\n\nlast\n", ""}}, DefaultLimits()) + require.NoError(t, err) + var texts []string + for _, line := range lines { + texts = append(texts, line.text) + } + require.Equal(t, []string{"Record 1, cell 1", "first", "", "last", "", "Record 1, cell 2", ""}, texts) +} + +func TestConvertCancellationClosureAndZeroValues(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + cancel() + source, reader := sourceFor(t, "x\n") + result, err := Convert(ctx, source, policyFor(t, DefaultLimits())) + require.ErrorIs(t, err, context.Canceled) + require.Nil(t, result) + require.True(t, reader.closed) + source, reader = sourceFor(t, "x\n") + result, err = Convert(t.Context(), source, Policy{}) + require.Error(t, err) + require.Nil(t, result) + require.True(t, reader.closed) + source, reader = sourceFor(t, "x\n") + reader.closeErr = errors.New("synthetic close failure") + result, err = Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.EqualError(t, err, "close CSV source failed") + require.Nil(t, result) + for _, zero := range []*Result{nil, {}} { + require.Empty(t, zero.PDF()) + require.Empty(t, zero.Receipt()) + _, err := zero.Source() + require.Error(t, err) + } + _, err = Convert(t.Context(), ocr.Source{}, policyFor(t, DefaultLimits())) + require.Error(t, err) + ctx, cancel = context.WithCancel(t.Context()) + reader = &trackedReader{Reader: cancelReader{cancel: cancel}} + source = ocr.Source{Content: reader, MediaType: "text/csv", Size: 1, SHA256: digest([]byte("x"))} + result, err = Convert(ctx, source, policyFor(t, DefaultLimits())) + require.ErrorIs(t, err, context.Canceled) + require.Nil(t, result) + require.True(t, reader.closed) +} + +func TestConvertSanitizesSourceErrors(t *testing.T) { + const sentinel = "synthetic-cell-value-93817" + for _, test := range []struct { + name string + readFailure, closeFailure bool + }{ + {"read", true, false}, {"close", false, true}, {"read and close", true, true}, + } { + t.Run(test.name, func(t *testing.T) { + source, reader := sourceFor(t, sentinel) + if test.readFailure { + reader.Reader = failingReader{err: errors.New("read failed for " + sentinel)} + } + if test.closeFailure { + reader.closeErr = errors.New("close failed for " + sentinel) + } + result, err := Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.Error(t, err) + require.NotContains(t, err.Error(), sentinel) + require.Nil(t, result) + require.True(t, reader.closed) + }) + } + for _, deadline := range []bool{false, true} { + t.Run(fmt.Sprintf("context deadline %t", deadline), func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + want := context.Canceled + if deadline { + cancel() + ctx, cancel = context.WithDeadline(t.Context(), time.Unix(0, 0)) + want = context.DeadlineExceeded + } + defer cancel() + source, reader := sourceFor(t, "x") + reader.Reader = cancelReader{cancel: cancel} + reader.closeErr = errors.New("close failed for " + sentinel) + result, err := Convert(ctx, source, policyFor(t, DefaultLimits())) + require.ErrorIs(t, err, want) + require.NotContains(t, err.Error(), sentinel) + require.Nil(t, result) + require.True(t, reader.closed) + }) + } +} + +type failingReader struct{ err error } + +func (r failingReader) Read([]byte) (int, error) { return 0, r.err } + +type cancelReader struct{ cancel context.CancelFunc } + +func (r cancelReader) Read(p []byte) (int, error) { + r.cancel() + return copy(p, "x"), nil +} diff --git a/document/csvpdf/integration_test.go b/document/csvpdf/integration_test.go new file mode 100644 index 00000000..b00248e8 --- /dev/null +++ b/document/csvpdf/integration_test.go @@ -0,0 +1,73 @@ +package csvpdf + +import ( + "bytes" + "encoding/base64" + "encoding/json/v2" + "io" + "net/http" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" + "go.kenn.io/docbank/document" + "go.kenn.io/docbank/document/mistral" + "go.kenn.io/docbank/document/mistral/mistraltest" + "go.kenn.io/kit/safefileio" +) + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +func TestConvertedSourceUsesExistingPDFProcessing(t *testing.T) { + source, _ := sourceFor(t, "name,value\ncafé,κόσμος Привет\n") + converted, err := Convert(t.Context(), source, policyFor(t, DefaultLimits())) + require.NoError(t, err) + normalize, err := document.NewNormalizePolicy(100_000) + require.NoError(t, err) + policy, err := mistral.NewPolicy(mistral.PolicyConfig{Region: mistral.RegionEU, Model: mistral.DefaultModel, Retention: mistral.RetentionZDR, Training: mistral.TrainingOptedOut, MaxDocumentBytes: 1 << 20, MaxResponseBytes: 1 << 20, MaxUnits: 3, NormalizePolicy: normalize}) + require.NoError(t, err) + manifest, err := mistraltest.SyntheticManifest(policy, true) + require.NoError(t, err) + authorization, err := policy.Authorize(manifest, "pdf") + require.NoError(t, err) + directory := filepath.Join(t.TempDir(), "spool") + require.NoError(t, safefileio.EnsurePrivateDir(directory)) + generated, err := converted.Source() + require.NoError(t, err) + prepared, err := mistral.Prepare(t.Context(), generated.Content, policy, mistral.PrepareOptions{Directory: directory, DeclaredMediaType: generated.MediaType, ExpectedSize: generated.Size, ExpectedSHA256: generated.SHA256, MaxSpoolBytes: 1 << 20, MinFreeBytes: 1}) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, prepared.Release()) }) + require.Equal(t, converted.Receipt().PDFSHA256, prepared.SHA256()) + requests := 0 + client, err := mistral.NewClient(policy, mistral.ClientConfig{APIKey: "synthetic-key", HTTPClient: &http.Client{Transport: roundTripFunc(func(request *http.Request) (*http.Response, error) { + requests++ + body, readErr := io.ReadAll(request.Body) + require.NoError(t, readErr) + var payload struct { + Model string `json:"model"` + Pages string `json:"pages"` + Document struct { + Type string `json:"type"` + URL string `json:"document_url"` + } `json:"document"` + } + require.NoError(t, json.Unmarshal(body, &payload)) + require.Equal(t, mistral.DefaultModel, payload.Model) + require.Equal(t, "0-2", payload.Pages) + require.Equal(t, "document_url", payload.Document.Type) + require.True(t, strings.HasPrefix(payload.Document.URL, "data:application/pdf;base64,")) + pdf, decodeErr := base64.StdEncoding.DecodeString(strings.TrimPrefix(payload.Document.URL, "data:application/pdf;base64,")) + require.NoError(t, decodeErr) + require.True(t, bytes.Equal(converted.PDF(), pdf)) + return &http.Response{StatusCode: http.StatusOK, Header: http.Header{"Content-Type": []string{"application/json"}}, Body: io.NopCloser(strings.NewReader(`{"model":"mistral-ocr-4-0","pages":[{"index":0,"markdown":"café κόσμος Привет"}],"usage_info":{"pages_processed":1}}`))}, nil + })}}) + require.NoError(t, err) + result, err := client.Process(t.Context(), prepared, authorization) + require.NoError(t, err) + require.Equal(t, 1, requests) + require.Equal(t, 1, result.UnitsProcessed) + require.NotZero(t, result.Document) +} diff --git a/document/csvpdf/policy.go b/document/csvpdf/policy.go new file mode 100644 index 00000000..73abaf73 --- /dev/null +++ b/document/csvpdf/policy.go @@ -0,0 +1,58 @@ +// Package csvpdf converts byte-verified CSV sources to bounded PDFs locally. +package csvpdf + +import ( + "crypto/sha256" + "errors" + "fmt" + + "go.kenn.io/docbank/internal/canonical" + "golang.org/x/image/font/gofont/gomono" +) + +// ConverterVersion identifies the parser, layout, font and PDF writer contract. +const ConverterVersion = "csvpdf-v1-fpdf-0.9.0" + +// Limits bounds one conversion. NewPolicy permits tightening the defaults only. +type Limits struct { + MaxSourceBytes int64 `json:"max_source_bytes"` + MaxPDFBytes int64 `json:"max_pdf_bytes"` + MaxRecords int `json:"max_records"` + MaxCells int `json:"max_cells"` + MaxCellBytes int `json:"max_cell_bytes"` + MaxPages int `json:"max_pages"` +} + +// DefaultLimits returns the hard ceilings for a conversion. +func DefaultLimits() Limits { + return Limits{MaxSourceBytes: 1 << 20, MaxPDFBytes: 10 << 20, MaxRecords: 10_000, MaxCells: 50_000, MaxCellBytes: 64 << 10, MaxPages: 100} +} + +// Policy captures immutable conversion limits. Its zero value is invalid. +type Policy struct { + limits Limits + fingerprint string +} + +// NewPolicy validates limits and fingerprints the complete conversion contract. +func NewPolicy(limits Limits) (Policy, error) { + ceiling := DefaultLimits() + if limits.MaxSourceBytes <= 0 || limits.MaxSourceBytes > ceiling.MaxSourceBytes || limits.MaxPDFBytes <= 0 || limits.MaxPDFBytes > ceiling.MaxPDFBytes || limits.MaxRecords <= 0 || limits.MaxRecords > ceiling.MaxRecords || limits.MaxCells <= 0 || limits.MaxCells > ceiling.MaxCells || limits.MaxCellBytes <= 0 || limits.MaxCellBytes > ceiling.MaxCellBytes || limits.MaxPages <= 0 || limits.MaxPages > ceiling.MaxPages { + return Policy{}, errors.New("CSV PDF limits must be positive and within DefaultLimits") + } + encoded, err := canonical.Marshal(struct { + Version string `json:"version"` + Layout string `json:"layout"` + FontSHA256 string `json:"font_sha256"` + Limits Limits `json:"limits"` + }{ConverterVersion, "a4-10pt-mono-80-columns-48-lines-v1", digest(gomono.TTF), limits}) + if err != nil { + return Policy{}, err + } + return Policy{limits: limits, fingerprint: digest(encoded)}, nil +} + +// Fingerprint returns the conversion identity; an invalid policy returns empty. +func (p Policy) Fingerprint() string { return p.fingerprint } + +func digest(data []byte) string { return fmt.Sprintf("%x", sha256.Sum256(data)) } diff --git a/document/csvpdf/render.go b/document/csvpdf/render.go new file mode 100644 index 00000000..36d3e055 --- /dev/null +++ b/document/csvpdf/render.go @@ -0,0 +1,136 @@ +package csvpdf + +import ( + "bytes" + "context" + "errors" + "fmt" + "strings" + "time" + "unicode" + + "github.com/go-pdf/fpdf" + "golang.org/x/image/font/gofont/gomono" + "golang.org/x/image/font/sfnt" +) + +const ( + columns = 80 + linesPerPage = 48 +) + +type layoutLine struct { + text string + record, cell int +} + +func layout(ctx context.Context, records [][]string, limits Limits) ([]layoutLine, error) { + font, err := sfnt.Parse(gomono.TTF) + if err != nil { + return nil, fmt.Errorf("parse CSV PDF font: %w", err) + } + var buffer sfnt.Buffer + var lines []layoutLine + appendLine := func(text string, record, cell int) error { + if len(lines) == limits.MaxPages*linesPerPage { + return errors.New("CSV PDF exceeds page limit") + } + lines = append(lines, layoutLine{text, record, cell}) + return nil + } + for recordIndex, record := range records { + for cellIndex, cell := range record { + if err := ctx.Err(); err != nil { + return nil, err + } + for _, char := range cell { + if char == '\n' { + continue + } + allowedScript := unicode.Is(unicode.Latin, char) || unicode.Is(unicode.Greek, char) || unicode.Is(unicode.Cyrillic, char) || unicode.Is(unicode.Common, char) + if char > 0xffff || unicode.IsControl(char) || unicode.Is(unicode.Cf, char) || unicode.IsMark(char) || !allowedScript { + return nil, errors.New("CSV cell contains unsupported text") + } + glyph, err := font.GlyphIndex(&buffer, char) + if err != nil || glyph == 0 { + return nil, errors.New("CSV cell contains a character absent from the embedded font") + } + } + if err := appendLine(fmt.Sprintf("Record %d, cell %d", recordIndex+1, cellIndex+1), recordIndex+1, cellIndex+1); err != nil { + return nil, err + } + for valueLine := range strings.SplitSeq(cell, "\n") { + runes := []rune(valueLine) + for len(runes) > columns { + if err := appendLine(string(runes[:columns]), recordIndex+1, cellIndex+1); err != nil { + return nil, err + } + runes = runes[columns:] + } + if err := appendLine(string(runes), recordIndex+1, cellIndex+1); err != nil { + return nil, err + } + } + } + } + return lines, nil +} + +func render(ctx context.Context, records [][]string, limits Limits) ([]byte, []Span, int, error) { + lines, err := layout(ctx, records, limits) + if err != nil { + return nil, nil, 0, err + } + pdf := fpdf.New("P", "mm", "A4", "") + pdf.SetCatalogSort(true) + pdf.SetCompression(true) + pdf.SetCreationDate(time.Unix(0, 0).UTC()) + pdf.SetModificationDate(time.Unix(0, 0).UTC()) + pdf.SetAutoPageBreak(false, 0) + pdf.AddUTF8FontFromBytes("GoMono", "", bytes.Clone(gomono.TTF)) + pdf.SetFont("GoMono", "", 10) + var spans []Span + pages := 0 + for index, line := range lines { + if err := ctx.Err(); err != nil { + return nil, nil, 0, err + } + row := index % linesPerPage + if row == 0 { + pdf.AddPage() + pages++ + pdf.Text(20, 14, fmt.Sprintf("CSV conversion, generated page %d", pages)) + } + pdf.Text(20, 24+float64(row)*5, line.text) + span := Span{Page: pages, Record: line.record, Cell: line.cell} + if len(spans) == 0 || spans[len(spans)-1] != span { + spans = append(spans, span) + } + } + output := boundedWriter{ctx: ctx, maximum: limits.MaxPDFBytes} + if err := pdf.Output(&output); err != nil { + return nil, nil, 0, fmt.Errorf("write CSV PDF: %w", err) + } + return output.Bytes(), spans, pages, nil +} + +type boundedWriter struct { + bytes.Buffer + + ctx context.Context + maximum int64 +} + +func (w *boundedWriter) Write(p []byte) (int, error) { + if err := w.ctx.Err(); err != nil { + return 0, err + } + if int64(len(p)) > w.maximum-int64(w.Len()) { + return 0, errors.New("CSV PDF exceeds byte limit") + } + n, err := w.Buffer.Write(p) + if err != nil { + return n, fmt.Errorf("buffer CSV PDF: %w", err) + } + return n, nil +} diff --git a/document/csvpdf/result.go b/document/csvpdf/result.go new file mode 100644 index 00000000..464d3c00 --- /dev/null +++ b/document/csvpdf/result.go @@ -0,0 +1,61 @@ +package csvpdf + +import ( + "bytes" + "errors" + "io" + "slices" + + "go.kenn.io/docbank/document/ocr" +) + +// Span associates a generated PDF page with a one-based CSV record and cell. +type Span struct { + Page int `json:"page"` + Record int `json:"record"` + Cell int `json:"cell"` +} + +// Receipt records conversion provenance, not permission to upload either file. +type Receipt struct { + SourceSHA256 string `json:"source_sha256"` + SourceBytes int64 `json:"source_bytes"` + PDFSHA256 string `json:"pdf_sha256"` + PDFBytes int64 `json:"pdf_bytes"` + Pages int `json:"pages"` + PolicyFingerprint string `json:"policy_fingerprint"` + ConverterVersion string `json:"converter_version"` + Spans []Span `json:"spans"` +} + +// Result owns verified PDF bytes and their conversion receipt. +type Result struct { + pdf []byte + receipt Receipt +} + +// PDF returns a copy of the generated PDF, or nil for a nil or zero result. +func (r *Result) PDF() []byte { + if r == nil { + return nil + } + return bytes.Clone(r.pdf) +} + +// Receipt returns a deep copy of the provenance, or zero for a nil result. +func (r *Result) Receipt() Receipt { + if r == nil { + return Receipt{} + } + receipt := r.receipt + receipt.Spans = slices.Clone(receipt.Spans) + return receipt +} + +// Source returns a fresh stream over the generated PDF for the existing OCR API. +func (r *Result) Source() (ocr.Source, error) { + if r == nil || len(r.pdf) == 0 { + return ocr.Source{}, errors.New("CSV PDF result is invalid") + } + return ocr.NewSource(io.NopCloser(bytes.NewReader(bytes.Clone(r.pdf))), "application/pdf", r.receipt.PDFBytes, r.receipt.PDFSHA256) +} diff --git a/go.mod b/go.mod index 4c8cb13b..4827e4ec 100644 --- a/go.mod +++ b/go.mod @@ -14,6 +14,7 @@ require ( github.com/charmbracelet/x/ansi v0.11.7 github.com/coder/websocket v1.8.15 github.com/danielgtaylor/huma/v2 v2.38.0 + github.com/go-pdf/fpdf v0.9.0 github.com/gofrs/flock v0.13.0 github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 github.com/mattn/go-sqlite3 v1.14.47 diff --git a/go.sum b/go.sum index 46bc4df0..e0467b8c 100644 --- a/go.sum +++ b/go.sum @@ -80,6 +80,8 @@ github.com/fxamacker/cbor/v2 v2.9.1 h1:2rWm8B193Ll4VdjsJY28jxs70IdDsHRWgQYAI80+r github.com/fxamacker/cbor/v2 v2.9.1/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= +github.com/go-pdf/fpdf v0.9.0 h1:PPvSaUuo1iMi9KkaAn90NuKi+P4gwMedWPHhj8YlJQw= +github.com/go-pdf/fpdf v0.9.0/go.mod h1:oO8N111TkmKb9D7VvWGLvLJlaZUQVPM+6V42pp3iV4Y= github.com/gofrs/flock v0.13.0 h1:95JolYOvGMqeH31+FC7D2+uULf6mG61mEZ/A8dRYMzw= github.com/gofrs/flock v0.13.0/go.mod h1:jxeyy9R1auM5S6JYDBhDt+E2TCo7DkratH4Pgi8P+Z0= github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=