feat(csv-import): drop-anywhere import — dedupe, account detection, saved profiles#1786
Open
canalesb93 wants to merge 7 commits into
Open
feat(csv-import): drop-anywhere import — dedupe, account detection, saved profiles#1786canalesb93 wants to merge 7 commits into
canalesb93 wants to merge 7 commits into
Conversation
…tent_hash, fingerprint
Groundwork for CSV import v2 (drag-anywhere, dedupe, account detection, profiles):
- Extract pure name-similarity scorer into tag-free internal/textmatch so the
CSV classifier can reuse the exact sync-matcher logic without importing sync
(cycle-free). Matcher delegates; behavior unchanged (existing tests pass).
- Migrations (additive, shared-DB safe):
- transactions.content_hash (account-independent dedup) + partial
transactions_dedup_lookup index on (account_id, date, amount).
- csv_import_profiles (saved recipes keyed by header fingerprint).
- csv_import_sessions + csv_import_rows (durable staging so preview == apply).
- sqlc queries for the three tables (rows via :copyfrom for 50k-row inserts)
plus UpsertTransactionV2 returning (xmax = 0) AS inserted for reliable
insert-vs-update counts.
- internal/provider/csv/fingerprint.go: HeaderFingerprint, ExtractMask
(card column + filename), FilenameTokens, InstitutionTokens. Unit tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atcher - internal/service/csv_classify.go: ParseCSVRows (mapping → parsed rows, errors preserved for inline-fix) + ClassifyCSVRows. Loads the resolved account's live transactions in the date window (provider-agnostic — dedupes vs Plaid/Teller too) and classifies each row as new / exact_dup (provider id or content_hash) / probable_dup (same amount, date within tolerance, name similar via textmatch) / error. content_hash + stable provider_txn_id derived per row. - internal/service/csv_account_match.go: weighted, explainable account ranker — saved-profile default account (auto), mask/last4 (+40), transaction overlap (≤+40), institution name (+15), filename hint (+5). Preselect only when the top is strong AND a clear leader, or a profile matched. - csv dedup helpers: GenerateContentHash (account-independent) + GenerateExternalIDWithOccurrence (force-import disambiguator). - sqlc: ListAccountTransactionsForDedup, ListTransactionKeysInRange. - Integration tests: all-new, exact dup by provider id, cross-provider probable dup, nearby-date/different-name stays new, parse error; overlap ranking, mask+overlap preselect, profile preselect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/service/csv_import_v2.go — the durable staging flow: - CreateImportSession: parse file, detect template/mapping/date-format/currency, apply a saved profile (auto mapping + auto-resolve to its account), run account detection, persist the session + rows. Auto-classifies when an account is confidently chosen; otherwise stages rows as needs_account. - ResolveImportAccount: bind an existing account or mint a new CSV connection+ account, reclassify every row, status → previewed. - ListImportRows: server-paginated rows + per-classification summary counts. - EditImportRow: inline-edit a row and reclassify just it. - BulkImportOp: include/exclude by classification, set-category, remap (re-parse + reclassify all). - ApplyImportSession: one tx — upsert the exact included set via UpsertTransactionV2 (reliable insert/update counts), occurrence-disambiguate genuine same-key duplicates, stamp session currency, write a sync_log, upsert the source profile, finalize (snapshot result, drop blob). Idempotent: a re-drop into the same account upserts to no-ops. New row column category_id (migration) for user-chosen Breadbox category applied at import; CountIncludedCSVImportRows query. Integration tests: full lifecycle, profile-driven re-drop (dedup 2 + add 1), double-import idempotency, account-change-before-confirm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/admin/csv_import_v2.go + routes under /-/csv/v2/* (session-cookie auth,
admin-only, so no OpenAPI surface / drift impact):
- POST /-/csv/v2/sessions multipart analyze → ImportAnalysis
- POST /-/csv/v2/sessions/{id}/resolve bind/create account → session
- GET /-/csv/v2/sessions/{id}/rows paginated rows + summary + session
- PATCH /-/csv/v2/sessions/{id}/rows/{rid} inline edit
- POST /-/csv/v2/sessions/{id}/bulk include/exclude/set-category/remap
- POST /-/csv/v2/sessions/{id}/apply apply → CSVImportResult
30MB upload cap; single-household-member default user; error envelope mapping
(404 for unknown session, 422 otherwise).
Integration test drives analyze → resolve(create-new) → rows → apply through a
real chi router and asserts 2 new transactions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…odal UI for the v2 flow, daisy-first, matching the Workflows-page quality bar: - base.html: $store.csvImport store + document-level drag-and-drop (dragenter depth counter, file-only overlay, [data-dropzone] opt-out, multi-file → first + toast). Full-window dashed drop overlay. Mounts components.ImportModal in the shared global-overlay region (renders on every page, hidden until a drop). - components.ImportModal: large daisy <dialog class="modal"> flow — analyzing / account picker (ranked suggestions w/ confidence + reason chips + saved-profile badge + create-new) / preview (classification filter join-tabs, summary counts, server-paginated table table-sm with include checkboxes, classification badges, inline row edit, bulk include/exclude) / success. - static/js/admin/components/csv_import_v2.js: the csvImport Alpine factory — watches the store, drives analyze→resolve→rows→edit/bulk→apply against the /-/csv/v2/* endpoints (CSRF auto-injected), toasts on apply. - Registered ImportModal in the renderComponent bridge. - Sandboxified: SectionCSVImport in /design (patterns) — drop overlay, account suggestions, classification badges, preview table, success. Screenshot-verified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- SweepExpiredImportSessions + hourly cron in serve.go: reaps un-applied
sessions past their 24h TTL (CASCADE drops staged rows + the raw file blob).
- Profile management: ListCSVProfiles / RenameCSVProfile / DeleteCSVProfile
(internal/service/csv_profiles.go) + admin endpoints GET/PATCH/DELETE
/-/csv/v2/profiles[/{id}]. A user's rename survives future imports; times_used
bumps on re-import. Integration test covers list/rename/redrop/delete.
- docs/csv-import.md: canonical doc for the v2 flow (staging, account detection,
dedup, profiles, currency, schema).
- Makefile sqlc/sqlc-tag now clean + build-tag internal/db/copyfrom.go (new
:copyfrom artifact) so the lite build excludes the db package correctly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Driven by two real exports (Chase checking, Amex): - ParseFile sets FieldsPerRecord = -1. Chase checking CSVs append a trailing comma to every data row (8 fields vs a 7-column header); Go's csv.Reader rejected the whole file. Column mapping is index-based + bounds-checked, so ragged trailing fields are harmless. (Amex's embedded newlines in quoted fields already parsed correctly.) - ExtractMask now also recognizes the "<institution><last4>" filename shape (Chase0198, Amex1009) — letters immediately followed by 4 digits — without mistaking a date stamp (export_20260607) for a mask. Regression tests: ragged-row parse + Chase Checking template match; the new filename mask cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CSV import v2 — drop-anywhere, dedupe, account detection, saved profiles
Turns CSV import from a nice-to-have wizard into a first-class, confidence-inspiring path. Drop a CSV on any page → a global hover overlay → a polished flow that detects which account the file belongs to, dedupes against what's already there (including Plaid/Teller rows), shows a preview of exactly what will change, lets you make targeted edits, and applies only on confirm. Re-dropping a later export of the same account only adds the genuinely new rows.
How it works
[data-dropzone]opt-out) opens a full-screen daisy modal: analyze → ranked account picker → deduped preview → apply.provider_transaction_idor account-independentcontent_hash) / probable-dup (same amount, date ±1d, similar name via the shared sync-matcher similarity) / error. Apply upserts viaUpsertTransactionV2and is idempotent — re-importing the same file is a no-op./-/csv/v2/profiles.Architecture
Durable staging (
csv_import_sessions+csv_import_rows) so the applied result equals the preview. All additive migrations (shared-DB safe). Admin-only routes under/-/csv/v2/*(no OpenAPI surface). The legacy wizard at/admin/connections/import-csvis left intact in parallel.Built in 6 self-contained waves (each independently builds + vets + tests green):
content_hash, fingerprint helpers/design/c/csv-import)Testing
internal/textmatch,internal/provider/csv(fingerprint/mask).go build ./...(default +-tags lite+-tags headless),go vet ./..., and theservice/admin/db/apiintegration suites all pass.Docs:
docs/csv-import.md.🤖 Generated with Claude Code