diff --git a/docs/mcp-tools-reference.md b/docs/mcp-tools-reference.md index 70294ea5..6e9461e5 100644 --- a/docs/mcp-tools-reference.md +++ b/docs/mcp-tools-reference.md @@ -302,20 +302,20 @@ The tools below are thin API skins over the DSL — the DSL doc is the source of ### create_transaction_rule (Write) -Create a rule that fires during sync. Actions compose (`set_category` + `add_tag` + `add_comment` in a single rule are all valid). +Create a rule that fires during sync. Actions compose (`set_category` + `add_tag` + `set_metadata` in a single rule are all valid). | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Human-readable rule name | -| `conditions` | object | Condition tree. Omit or `{}` for match-all. Supports `and` / `or` / `not` nesting up to depth 10. | -| `actions` | array | Typed actions: `set_category`, `add_tag`, `remove_tag`, `add_comment`. Either this or `category_slug` is required. | +| `conditions` | object | Condition tree. Omit or `{}` for match-all. Supports `and` / `or` / `not` nesting up to depth 10. Leaf fields include `metadata.` to read a key from the free-form metadata blob (ops: `eq`/`neq`/`contains`/`not_contains`/`matches`/`in`/`gt`/`gte`/`lt`/`lte`/`exists`/`not_exists`; an absent key matches only `not_exists`). | +| `actions` | array | Typed actions: `set_category`, `add_tag`, `remove_tag`, `add_comment`, `set_metadata` (`metadata_key` + `metadata_value`, any JSON), `remove_metadata` (`metadata_key`), `assign_series`. Either this or `category_slug` is required. | | `category_slug` | string | Shorthand for `actions=[{type:set_category,category_slug:...}]` | | `trigger` | string | `on_create` (default) / `on_change` / `always`. `on_update` accepted as legacy alias. | | `stage` | string | **Preferred.** Semantic pipeline stage: `baseline` / `standard` / `refinement` / `override`. Resolves to priority `0 / 10 / 50 / 100`. | | `priority` | int | Raw pipeline-stage integer, 0–1000. Use for fine-grained slotting within a stage. If both `stage` and `priority` are supplied, `priority` wins. Defaults to `10` (standard) if neither is provided. | | `enabled` | bool | Default true | | `expires_in` | string | Optional duration (e.g., `24h`, `30d`, `1w`) | -| `apply_retroactively` | bool | Also back-fill matching existing transactions (materializes `set_category` / `add_tag` / `remove_tag`; `add_comment` is sync-only) | +| `apply_retroactively` | bool | Also back-fill matching existing transactions (materializes `set_category` / `add_tag` / `remove_tag` / `set_metadata` / `remove_metadata` / `assign_series`; `add_comment` is sync-only) | ### list_transaction_rules (Read) diff --git a/docs/rule-dsl.md b/docs/rule-dsl.md index 42c47671..e37e53c8 100644 --- a/docs/rule-dsl.md +++ b/docs/rule-dsl.md @@ -73,6 +73,7 @@ Combinators nest. Max depth: **10**. Empty / zero-value condition (`{}`) means * | `tags` | tags | List of current transaction tag slugs (special, see below) | | `series` | string | `short_id` of the recurring series the transaction belongs to (empty when unassigned) | | `in_series` | bool | Whether the transaction is linked to any recurring series | +| `metadata.` | metadata | One key from the transaction's free-form metadata blob — e.g. `metadata.tax_deductible` reads `metadata["tax_deductible"]`. See "Metadata conditions" below. | > **Raw vs assigned category.** `provider_category_primary` / `provider_category_detailed` are the provider's classification — they don't change when Breadbox, a rule, or the user reassigns. Use `category` when you want to react to the *current* category, including mid-pass rule updates (see "Rule chaining" below). @@ -89,6 +90,9 @@ Combinators nest. Max depth: **10**. Empty / zero-value condition (`{}`) means * | **bool** | `eq`, `neq` | `true` / `false` | | **tags** | `contains`, `not_contains` | `value` is a single tag slug; case-insensitive | | | `in` | `value` is an array of slugs; matches if any slug is present | +| **metadata** | `eq`, `neq`, `contains`, `not_contains`, `matches`, `in` | string-style ops on the stringified value (`contains`/`matches` case-insensitive / RE2) | +| | `gt`, `gte`, `lt`, `lte` | numeric comparison; both sides must parse as numbers | +| | `exists`, `not_exists` | key-presence test; `value` is ignored | Unknown field or unknown op → condition evaluates to false (the rule simply won't match). Invalid regex or wrong value type → **rejected at write time**. @@ -115,6 +119,30 @@ The `tags` slice starts from tags already persisted on the row (loaded during sy Mutations are scoped to the resolver run — the caller's `TransactionContext` (and the incoming tag slice) are not modified. +### Metadata conditions + +Transactions carry a free-form JSONB `metadata` blob — arbitrary key/value enrichment your household cares about that isn't a first-class field (`tax_deductible`, `trip`, `reimbursable_by`, …). A rule reads one key per leaf via the dotted field `metadata.`: + +```json +{ "field": "metadata.tax_deductible", "op": "eq", "value": true } +{ "field": "metadata.trip", "op": "eq", "value": "japan-2026" } +{ "field": "metadata.reimburse_amount", "op": "gte", "value": 100 } +{ "field": "metadata.notes", "op": "contains", "value": "warranty" } +{ "field": "metadata.project_code", "op": "exists" } +``` + +The key (everything after `metadata.`) must be a valid metadata key — non-empty, ≤128 chars. Metadata keys are **case-sensitive** (they're JSONB keys), unlike tag slugs. + +**Semantics:** + +- `exists` / `not_exists` test key *presence* and ignore `value`. +- **Every other operator requires the key to be present.** An absent key matches only `not_exists` — so `metadata.foo neq "x"` does **not** match a transaction that simply has no `foo` key (use `not_exists`, or an `or` of the two, if you want "missing OR different"). This avoids a `neq`/`not_contains` rule silently matching every transaction that lacks the key. +- `eq` / `neq` pick their comparison from the **expected value's type**: a boolean expected compares as bool, a numeric expected compares numerically, otherwise the stored value is stringified and compared case-insensitively. So `value: true` matches stored `true` (and the string `"true"`); `value: 100` matches stored `100` or `"100"`. +- `contains` / `not_contains` / `matches` / `in` always operate on the **stringified** stored value (numbers → their decimal form, booleans → `true`/`false`, objects/arrays → their JSON encoding), so a `matches` can pattern-match inside a structured value. +- `gt` / `gte` / `lt` / `lte` require both the stored value and the expected value to parse as numbers; otherwise the leaf is false. + +Metadata conditions evaluate identically at sync time, in `preview_rule`, and during retroactive apply. They also chain: a `set_metadata` action by an earlier-stage rule is visible to a later-stage rule's `metadata.` condition within the same pass (see "Rule chaining" above and the `set_metadata` action below). + ## Actions Actions describe what a matching rule does to the transaction. An action array must have at least one element. @@ -183,6 +211,29 @@ Behavior: This is the declarative counterpart to the `assign_series` MCP tool: author the rule once and every future matching charge auto-joins the series with zero agent runs. +### `set_metadata` + +```json +{ "type": "set_metadata", "metadata_key": "tax_deductible", "metadata_value": true } +``` + +Upserts **one key** in the transaction's free-form metadata blob, leaving every other key untouched. `metadata_value` may be any JSON value (string, number, boolean, object, array). This is the declarative counterpart to the `set_transaction_metadata` MCP tool. + +- `metadata_key` must be non-empty and ≤128 chars; `metadata_value` must serialize to ≤4 KiB (the same per-value cap the scoped metadata ops enforce). Both checked at write time. +- Repeatable — a rule may set several keys at once. **Last-writer-wins per key** across the pipeline (a higher-priority rule's `set_metadata` for the same key overrides a lower one). +- Chains: the write is mirrored into the live context, so a later-stage rule's `metadata.` condition observes it within the same pass. +- Materializes inside the sync transaction (and on retroactive apply) via a single `metadata = (metadata - removed) || set` merge. **No dedicated timeline annotation** is emitted (same as `assign_series`); the rule's `hit_count` records the firing. + +### `remove_metadata` + +```json +{ "type": "remove_metadata", "metadata_key": "needs_receipt" } +``` + +Deletes **one key** from the metadata blob. No-op if the key isn't present. Repeatable. + +- **Net-diff with `set_metadata`** in a single pass: if an earlier-stage rule sets a key and a later-stage rule removes the same key, they cancel — neither hits the DB. A remove-then-set ends as a set. Mirrors `add_tag` / `remove_tag` net-diff semantics. + ### Combining actions A rule can carry multiple actions of different types. Override (`category_override <> 'none'`) suppresses only the `set_category` part — `add_tag` and `add_comment` still fire. @@ -199,7 +250,7 @@ A rule can carry multiple actions of different types. Override (`category_overri #### Which combinations make sense -Only `set_category` is singleton per rule — repeating it is rejected at write time. `add_tag`, `remove_tag`, and `add_comment` can appear multiple times in one rule (e.g. add two tags at once, or add one and remove another). The admin UI disables a second `set_category` dropdown option once one is picked; tag and comment rows are freely repeatable. +Only `set_category` is singleton per rule — repeating it is rejected at write time. `add_tag`, `remove_tag`, `add_comment`, `set_metadata`, and `remove_metadata` can appear multiple times in one rule (e.g. add two tags at once, or write two metadata keys). The admin UI disables a second `set_category` dropdown option once one is picked; tag and comment rows are freely repeatable. Useful combinations: @@ -280,6 +331,8 @@ The rule engine has two entry points. They share condition evaluation and priori | `set_category` | Applied (respects override) | Applied (respects override) | | `add_tag` | Applied | Applied | | `remove_tag` | Applied | Applied | +| `set_metadata` | Applied | Applied | +| `remove_metadata` | Applied | Applied | | `add_comment` | Applied | **Not applied** (by design) | | `hit_count` | +1 per condition match | +1 per condition match | | `rule_applied` annotation | Written | Written (with `applied_by = "retroactive"`) | diff --git a/internal/mcp/tools.go b/internal/mcp/tools.go index eb0c6469..59cdf470 100644 --- a/internal/mcp/tools.go +++ b/internal/mcp/tools.go @@ -314,28 +314,28 @@ func (s *MCPServer) handleTransactionSummary(_ context.Context, _ *mcpsdk.CallTo // --- Transaction Rules --- type createTransactionRuleInput struct { - Name string `json:"name" jsonschema:"required,Name for this rule (human-readable description)"` - Conditions map[string]any `json:"conditions,omitempty" jsonschema:"JSON condition tree. Omit or pass {} to match every transaction. Leaf: {\"field\":\"...\",\"op\":\"...\",\"value\":...}. Combinators: {\"and\":[...]}, {\"or\":[...]}, {\"not\":{...}} (nest freely, max depth 10). Fields: provider_name provider_merchant_name amount provider_category_primary provider_category_detailed category(assigned slug, live-updated by earlier-stage rules) pending provider account_id account_name user_id user_name tags. Ops: string/category=eq|neq|contains|not_contains|matches(RE2)|in; numeric=eq|neq|gt|gte|lt|lte; bool=eq|neq; tags=contains|not_contains|in. Nested example: {\"or\":[{\"and\":[{\"field\":\"provider_merchant_name\",\"op\":\"contains\",\"value\":\"starbucks\"},{\"field\":\"amount\",\"op\":\"gte\",\"value\":5}]},{\"field\":\"tags\",\"op\":\"contains\",\"value\":\"coffee\"}]}. Full spec: docs/rule-dsl.md."` - Actions []map[string]string `json:"actions,omitempty" jsonschema:"Array of typed actions. {\"type\":\"set_category\",\"category_slug\":\"...\"} | {\"type\":\"add_tag\",\"tag_slug\":\"...\"} | {\"type\":\"remove_tag\",\"tag_slug\":\"...\"} | {\"type\":\"add_comment\",\"content\":\"...\"}. Actions compose: a rule can set a category AND add a tag AND add a comment in the same match. add_comment fires only at sync time (not on retroactive apply). remove_tag net-diffs against add_tag within the same sync pass. If omitted, use category_slug instead."` - CategorySlug string `json:"category_slug,omitempty" jsonschema:"Shorthand for actions: [{\"type\":\"set_category\",\"category_slug\":\"\"}]. Either actions or category_slug is required."` - Trigger string `json:"trigger,omitempty" jsonschema:"When the rule fires during sync: 'on_create' (default — first-synced transactions) | 'on_change' (existing transactions that changed on re-sync) | 'always' (both). 'on_update' is accepted as a legacy alias for 'on_change'. Retroactive apply ignores trigger."` - Stage string `json:"stage,omitempty" jsonschema:"Semantic pipeline stage — preferred over raw priority for agent-authored rules. One of: 'baseline' (runs first, broad defaults), 'standard' (default), 'refinement' (reacts to earlier stages), 'override' (runs last, wins set_category). Resolves to priority 0/10/50/100. If both stage and priority are supplied, priority wins. Leave both unset for 'standard'."` - Priority int `json:"priority,omitempty" jsonschema:"Raw pipeline-stage integer, 0..1000. Lower runs first. Prefer 'stage' for shared vocabulary. Canonical values: 0=baseline, 10=standard (default), 50=refinement, 100=override. Higher-priority rules observe earlier-stage rules' tag/category mutations via conditions, and win set_category under last-writer semantics."` - ExpiresIn string `json:"expires_in,omitempty" jsonschema:"Optional expiry duration: 24h, 30d, 1w. Rule auto-disables after this period."` - ApplyRetroactively bool `json:"apply_retroactively,omitempty" jsonschema:"If true, immediately apply this rule to existing transactions after creation. Materializes set_category / add_tag / remove_tag; skips add_comment (sync-only). Hit count reflects every condition match, matching sync-time semantics."` + Name string `json:"name" jsonschema:"required,Name for this rule (human-readable description)"` + Conditions map[string]any `json:"conditions,omitempty" jsonschema:"JSON condition tree. Omit or pass {} to match every transaction. Leaf: {\"field\":\"...\",\"op\":\"...\",\"value\":...}. Combinators: {\"and\":[...]}, {\"or\":[...]}, {\"not\":{...}} (nest freely, max depth 10). Fields: provider_name provider_merchant_name amount provider_category_primary provider_category_detailed category(assigned slug, live-updated by earlier-stage rules) pending provider account_id account_name user_id user_name tags series in_series, plus metadata. to read a key from the free-form metadata blob (e.g. metadata.tax_deductible). Ops: string/category=eq|neq|contains|not_contains|matches(RE2)|in; numeric=eq|neq|gt|gte|lt|lte; bool=eq|neq; tags=contains|not_contains|in; metadata.=eq|neq|contains|not_contains|matches|in|gt|gte|lt|lte|exists|not_exists (an absent key matches only not_exists; eq/neq comparison type follows the value's type). Nested example: {\"or\":[{\"and\":[{\"field\":\"provider_merchant_name\",\"op\":\"contains\",\"value\":\"starbucks\"},{\"field\":\"amount\",\"op\":\"gte\",\"value\":5}]},{\"field\":\"metadata.reimbursable\",\"op\":\"eq\",\"value\":true}]}. Full spec: docs/rule-dsl.md."` + Actions []map[string]any `json:"actions,omitempty" jsonschema:"Array of typed actions. {\"type\":\"set_category\",\"category_slug\":\"...\"} | {\"type\":\"add_tag\",\"tag_slug\":\"...\"} | {\"type\":\"remove_tag\",\"tag_slug\":\"...\"} | {\"type\":\"add_comment\",\"content\":\"...\"} | {\"type\":\"set_metadata\",\"metadata_key\":\"...\",\"metadata_value\":} | {\"type\":\"remove_metadata\",\"metadata_key\":\"...\"} | {\"type\":\"assign_series\",\"series_short_id\":\"...\"} or {\"type\":\"assign_series\",\"merchant_key\":\"...\",\"create_if_missing\":true}. Actions compose: a rule can set a category AND add a tag AND write metadata in the same match. set_metadata upserts one key (value may be any JSON type); remove_metadata deletes one key; both repeat freely. add_comment fires only at sync time (not on retroactive apply). remove_tag/remove_metadata net-diff against add_tag/set_metadata within the same sync pass. If omitted, use category_slug instead."` + CategorySlug string `json:"category_slug,omitempty" jsonschema:"Shorthand for actions: [{\"type\":\"set_category\",\"category_slug\":\"\"}]. Either actions or category_slug is required."` + Trigger string `json:"trigger,omitempty" jsonschema:"When the rule fires during sync: 'on_create' (default — first-synced transactions) | 'on_change' (existing transactions that changed on re-sync) | 'always' (both). 'on_update' is accepted as a legacy alias for 'on_change'. Retroactive apply ignores trigger."` + Stage string `json:"stage,omitempty" jsonschema:"Semantic pipeline stage — preferred over raw priority for agent-authored rules. One of: 'baseline' (runs first, broad defaults), 'standard' (default), 'refinement' (reacts to earlier stages), 'override' (runs last, wins set_category). Resolves to priority 0/10/50/100. If both stage and priority are supplied, priority wins. Leave both unset for 'standard'."` + Priority int `json:"priority,omitempty" jsonschema:"Raw pipeline-stage integer, 0..1000. Lower runs first. Prefer 'stage' for shared vocabulary. Canonical values: 0=baseline, 10=standard (default), 50=refinement, 100=override. Higher-priority rules observe earlier-stage rules' tag/category mutations via conditions, and win set_category under last-writer semantics."` + ExpiresIn string `json:"expires_in,omitempty" jsonschema:"Optional expiry duration: 24h, 30d, 1w. Rule auto-disables after this period."` + ApplyRetroactively bool `json:"apply_retroactively,omitempty" jsonschema:"If true, immediately apply this rule to existing transactions after creation. Materializes set_category / add_tag / remove_tag / set_metadata / remove_metadata / assign_series; skips add_comment (sync-only). Hit count reflects every condition match, matching sync-time semantics."` } type updateTransactionRuleInput struct { - ID string `json:"id" jsonschema:"required,UUID of the rule to update"` - Name *string `json:"name,omitempty" jsonschema:"New name for the rule. Omit to leave unchanged."` - Conditions map[string]any `json:"conditions,omitempty" jsonschema:"New condition tree (same format as create). Pass {} to explicitly change to match-all. Omit entirely to leave conditions unchanged."` - Actions *[]map[string]string `json:"actions,omitempty" jsonschema:"Replace the entire actions array with typed actions: {\"type\":\"set_category|add_tag|remove_tag|add_comment\", ...}. Pass an empty array to reject (rules must have at least one action). Omit to leave actions unchanged."` - CategorySlug *string `json:"category_slug,omitempty" jsonschema:"Shorthand: replace only the set_category action. Other action types on the rule are preserved. Omit to leave unchanged."` - Trigger *string `json:"trigger,omitempty" jsonschema:"New trigger: on_create, on_change, or always. 'on_update' accepted as alias for on_change. Omit to leave unchanged."` - Stage *string `json:"stage,omitempty" jsonschema:"New semantic pipeline stage: baseline | standard | refinement | override (resolves to priority 0/10/50/100). Preferred alias for agent-authored updates. If both stage and priority are supplied, priority wins. Omit to leave unchanged."` - Priority *int `json:"priority,omitempty" jsonschema:"New raw priority (pipeline stage). Prefer 'stage' for shared vocabulary. Omit to leave unchanged."` - Enabled *bool `json:"enabled,omitempty" jsonschema:"Enable or disable the rule. Disabled rules are excluded from sync + retroactive apply."` - ExpiresAt *string `json:"expires_at,omitempty" jsonschema:"New expiry timestamp (RFC3339) or empty string to clear expiry entirely. Omit to leave unchanged."` + ID string `json:"id" jsonschema:"required,UUID of the rule to update"` + Name *string `json:"name,omitempty" jsonschema:"New name for the rule. Omit to leave unchanged."` + Conditions map[string]any `json:"conditions,omitempty" jsonschema:"New condition tree (same format as create). Pass {} to explicitly change to match-all. Omit entirely to leave conditions unchanged."` + Actions *[]map[string]any `json:"actions,omitempty" jsonschema:"Replace the entire actions array with typed actions: {\"type\":\"set_category|add_tag|remove_tag|add_comment|set_metadata|remove_metadata|assign_series\", ...}. set_metadata takes metadata_key + metadata_value (any JSON); remove_metadata takes metadata_key. Pass an empty array to reject (rules must have at least one action). Omit to leave actions unchanged."` + CategorySlug *string `json:"category_slug,omitempty" jsonschema:"Shorthand: replace only the set_category action. Other action types on the rule are preserved. Omit to leave unchanged."` + Trigger *string `json:"trigger,omitempty" jsonschema:"New trigger: on_create, on_change, or always. 'on_update' accepted as alias for on_change. Omit to leave unchanged."` + Stage *string `json:"stage,omitempty" jsonschema:"New semantic pipeline stage: baseline | standard | refinement | override (resolves to priority 0/10/50/100). Preferred alias for agent-authored updates. If both stage and priority are supplied, priority wins. Omit to leave unchanged."` + Priority *int `json:"priority,omitempty" jsonschema:"New raw priority (pipeline stage). Prefer 'stage' for shared vocabulary. Omit to leave unchanged."` + Enabled *bool `json:"enabled,omitempty" jsonschema:"Enable or disable the rule. Disabled rules are excluded from sync + retroactive apply."` + ExpiresAt *string `json:"expires_at,omitempty" jsonschema:"New expiry timestamp (RFC3339) or empty string to clear expiry entirely. Omit to leave unchanged."` } type deleteTransactionRuleInput struct { @@ -347,14 +347,14 @@ type batchCreateRulesInput struct { } type batchRuleItem struct { - Name string `json:"name" jsonschema:"required,Human-readable rule name"` - Actions []map[string]string `json:"actions,omitempty" jsonschema:"Actions array (typed — same format as create_transaction_rule)"` - CategorySlug string `json:"category_slug,omitempty" jsonschema:"Shorthand for set_category action. Either actions or category_slug required."` - Conditions map[string]any `json:"conditions,omitempty" jsonschema:"Condition tree as JSON object. Omit or {} for match-all."` - Trigger string `json:"trigger,omitempty" jsonschema:"on_create (default), on_change, or always. 'on_update' accepted as alias for on_change."` - Stage string `json:"stage,omitempty" jsonschema:"Semantic pipeline stage: baseline | standard (default) | refinement | override — resolves to priority 0/10/50/100. Prefer over raw priority for cross-agent consistency. If both are supplied, priority wins."` - Priority int `json:"priority,omitempty" jsonschema:"Raw priority integer. Prefer 'stage' for shared vocabulary. Defaults to 10 (standard)."` - ExpiresIn string `json:"expires_in,omitempty" jsonschema:"Optional expiry duration"` + Name string `json:"name" jsonschema:"required,Human-readable rule name"` + Actions []map[string]any `json:"actions,omitempty" jsonschema:"Actions array (typed — same format as create_transaction_rule, incl. set_metadata / remove_metadata)"` + CategorySlug string `json:"category_slug,omitempty" jsonschema:"Shorthand for set_category action. Either actions or category_slug required."` + Conditions map[string]any `json:"conditions,omitempty" jsonschema:"Condition tree as JSON object. Omit or {} for match-all."` + Trigger string `json:"trigger,omitempty" jsonschema:"on_create (default), on_change, or always. 'on_update' accepted as alias for on_change."` + Stage string `json:"stage,omitempty" jsonschema:"Semantic pipeline stage: baseline | standard (default) | refinement | override — resolves to priority 0/10/50/100. Prefer over raw priority for cross-agent consistency. If both are supplied, priority wins."` + Priority int `json:"priority,omitempty" jsonschema:"Raw priority integer. Prefer 'stage' for shared vocabulary. Defaults to 10 (standard)."` + ExpiresIn string `json:"expires_in,omitempty" jsonschema:"Optional expiry duration"` } type applyRulesInput struct { @@ -670,26 +670,33 @@ func (s *MCPServer) handleSubmitReport(reqCtx context.Context, _ *mcpsdk.CallToo // parseConditions converts a map[string]any (from MCP input) to a service.Condition. // convertMCPActions converts MCP action maps to service RuleAction slice. // -// Accepts the typed shape {type, category_slug|tag_slug|content}. Legacy -// shape {field:"category", value:""} is auto-translated to -// {type:"set_category", category_slug:""} for back-compat. -func convertMCPActions(actions []map[string]string) []service.RuleAction { +// Accepts the typed shape {type, category_slug|tag_slug|content|series_*| +// metadata_*}. metadata_value keeps its native JSON type (string, number, bool, +// object, array). Legacy shape {field:"category", value:""} is +// auto-translated to {type:"set_category", category_slug:""} for +// back-compat. +func convertMCPActions(actions []map[string]any) []service.RuleAction { if len(actions) == 0 { return nil } result := make([]service.RuleAction, len(actions)) for i, a := range actions { act := service.RuleAction{ - Type: a["type"], - CategorySlug: a["category_slug"], - TagSlug: a["tag_slug"], - Content: a["content"], + Type: mcpString(a["type"]), + CategorySlug: mcpString(a["category_slug"]), + TagSlug: mcpString(a["tag_slug"]), + Content: mcpString(a["content"]), + SeriesShortID: mcpString(a["series_short_id"]), + MerchantKey: mcpString(a["merchant_key"]), + CreateIfMissing: mcpBool(a["create_if_missing"]), + MetadataKey: mcpString(a["metadata_key"]), + MetadataValue: a["metadata_value"], } // Legacy shape: {"field":"category","value":""}. if act.Type == "" { - if field, ok := a["field"]; ok && field == "category" { + if field := mcpString(a["field"]); field == "category" { act.Type = "set_category" - act.CategorySlug = a["value"] + act.CategorySlug = mcpString(a["value"]) } } result[i] = act @@ -697,6 +704,20 @@ func convertMCPActions(actions []map[string]string) []service.RuleAction { return result } +// mcpString coerces an untyped JSON action value to a string, returning "" for +// nil or non-string values (the typed fields are always strings). +func mcpString(v any) string { + s, _ := v.(string) + return s +} + +// mcpBool coerces an untyped JSON action value to a bool, returning false for +// nil or non-bool values. +func mcpBool(v any) bool { + b, _ := v.(bool) + return b +} + func parseConditions(m map[string]any) (service.Condition, error) { data, err := json.Marshal(m) if err != nil { @@ -839,7 +860,7 @@ func shortIDPrefix(key string) (string, bool) { // avoiding the unmarshal→walk→remarshal cycle. It scans the byte stream and // collapses each object's own id/short_id pair: // -// {"id":"","short_id":""} → {"id":""} +// {"id":"","short_id":""} → {"id":""} // // Only the bare "short_id" key triggers the rewrite; the "short_id" key is // then dropped. If the sibling "id" field is missing or "short_id" is null, @@ -1134,4 +1155,3 @@ func scanJSONString(data []byte, pos int) (string, int) { } return s, end } - diff --git a/internal/service/rules.go b/internal/service/rules.go index 8f016d76..9606ac83 100644 --- a/internal/service/rules.go +++ b/internal/service/rules.go @@ -68,6 +68,44 @@ var tagsOps = map[string]bool{ "contains": true, "not_contains": true, "in": true, } +// metadataFieldPrefix marks a condition leaf that reads a key from the +// transaction's free-form metadata blob. The substring after the prefix is the +// metadata key, e.g. field "metadata.tax_deductible" reads metadata["tax_deductible"]. +const metadataFieldPrefix = "metadata." + +// metadataOps are operators valid for a metadata. field. Metadata values +// are arbitrary JSON, so the set is the union of string, numeric, and presence +// operators; comparison semantics are resolved at eval time from the expected +// value's type (see evalMetadata). exists / not_exists test key presence and +// ignore the value. +var metadataOps = map[string]bool{ + "eq": true, "neq": true, "contains": true, "not_contains": true, + "matches": true, "in": true, + "gt": true, "gte": true, "lt": true, "lte": true, + "exists": true, "not_exists": true, +} + +// metadataKeyFromField extracts the metadata key from a "metadata." field, +// returning ("", false) when field is not a metadata field. +func metadataKeyFromField(field string) (string, bool) { + if !strings.HasPrefix(field, metadataFieldPrefix) { + return "", false + } + return field[len(metadataFieldPrefix):], true +} + +// conditionFieldType returns the value-type bucket for a condition field, +// dispatching "metadata." dotted fields to the "metadata" bucket and +// otherwise consulting validConditionFields. Returns ("", false) for unknown +// fields. +func conditionFieldType(field string) (string, bool) { + if _, ok := metadataKeyFromField(field); ok { + return "metadata", true + } + t, ok := validConditionFields[field] + return t, ok +} + // conditionIsEmpty reports whether a Condition is a zero-value match-all. // Used to map between DB NULL conditions and the "always match" semantic. func conditionIsEmpty(c Condition) bool { @@ -102,7 +140,7 @@ func validateConditionDepth(c Condition, depth int) error { } if isLeaf { - fieldType, ok := validConditionFields[c.Field] + fieldType, ok := conditionFieldType(c.Field) if !ok { return fmt.Errorf("%w: unknown field %q", ErrInvalidParameter, c.Field) } @@ -165,6 +203,44 @@ func validateConditionDepth(c Condition, depth int) error { return fmt.Errorf("%w: %s on tags requires a string value", ErrInvalidParameter, c.Op) } } + case "metadata": + key, _ := metadataKeyFromField(c.Field) + if err := validateMetadataKey(key); err != nil { + return fmt.Errorf("%w (field %q)", err, c.Field) + } + if !metadataOps[c.Op] { + return fmt.Errorf("%w: operator %q not valid for metadata field %q", ErrInvalidParameter, c.Op, c.Field) + } + switch c.Op { + case "exists", "not_exists": + // Presence test — value is ignored, nothing to validate. + case "in": + vals, ok := toStringSlice(c.Value) + if !ok || len(vals) == 0 { + return fmt.Errorf("%w: 'in' operator requires a non-empty array for field %q", ErrInvalidParameter, c.Field) + } + case "matches": + s, ok := c.Value.(string) + if !ok { + return fmt.Errorf("%w: 'matches' operator requires a string value for field %q", ErrInvalidParameter, c.Field) + } + if _, err := regexp.Compile(s); err != nil { + return fmt.Errorf("%w: invalid regex pattern for field %q: %v", ErrInvalidParameter, c.Field, err) + } + case "gt", "gte", "lt", "lte": + if _, ok := toFloat64(c.Value); !ok { + return fmt.Errorf("%w: numeric value required for operator %q on field %q", ErrInvalidParameter, c.Op, c.Field) + } + default: // eq, neq, contains, not_contains + if c.Value == nil { + return fmt.Errorf("%w: operator %q requires a value for field %q", ErrInvalidParameter, c.Op, c.Field) + } + switch c.Value.(type) { + case string, float64, float32, int, int64, bool, json.Number: + default: + return fmt.Errorf("%w: operator %q on field %q requires a scalar value (string, number, or boolean)", ErrInvalidParameter, c.Op, c.Field) + } + } } } @@ -354,9 +430,154 @@ func evaluateLeaf(c *CompiledCondition, tctx TransactionContext) bool { case "in_series": return evalBool(c, tctx.InSeries) } + if key, ok := metadataKeyFromField(c.Field); ok { + return evalMetadata(c, key, tctx.Metadata) + } return false } +// evalMetadata evaluates a metadata. leaf against the transaction's +// metadata blob. Presence operators (exists / not_exists) test key presence; +// every other operator requires the key to be present (an absent key matches +// only not_exists). Comparison semantics for eq / neq are driven by the +// expected value's type — a numeric expected compares numerically, a bool +// expected compares as bool, otherwise the stored value is stringified and +// compared case-insensitively. contains / not_contains / matches / in always +// operate on the stringified stored value. +func evalMetadata(c *CompiledCondition, key string, meta map[string]any) bool { + raw, present := meta[key] + switch c.Op { + case "exists": + return present + case "not_exists": + return !present + } + if !present { + return false + } + switch c.Op { + case "eq": + return metadataEquals(raw, c.Value) + case "neq": + return !metadataEquals(raw, c.Value) + case "contains": + return strings.Contains(strings.ToLower(metadataString(raw)), strings.ToLower(metadataString(c.Value))) + case "not_contains": + return !strings.Contains(strings.ToLower(metadataString(raw)), strings.ToLower(metadataString(c.Value))) + case "matches": + if c.Regex != nil { + return c.Regex.MatchString(metadataString(raw)) + } + return false + case "in": + actual := metadataString(raw) + if vals, ok := toStringSlice(c.Value); ok { + for _, v := range vals { + if strings.EqualFold(actual, v) { + return true + } + } + } + return false + case "gt", "gte", "lt", "lte": + actual, ok1 := metadataFloat(raw) + expected, ok2 := toFloat64(c.Value) + if !ok1 || !ok2 { + return false + } + switch c.Op { + case "gt": + return actual > expected + case "gte": + return actual >= expected + case "lt": + return actual < expected + case "lte": + return actual <= expected + } + } + return false +} + +// metadataEquals compares a stored metadata value against an expected condition +// value. The expected value's type selects the comparison: bool → bool, +// number → numeric, otherwise case-insensitive string compare on the +// stringified stored value. +func metadataEquals(raw, expected any) bool { + switch exp := expected.(type) { + case bool: + b, ok := metadataBool(raw) + return ok && b == exp + case float64, float32, int, int64, json.Number: + ev, _ := toFloat64(expected) + av, ok := metadataFloat(raw) + return ok && av == ev + default: + return strings.EqualFold(metadataString(raw), metadataString(expected)) + } +} + +// metadataString renders a stored metadata value as a string for string +// operators. Scalars render naturally; objects/arrays fall back to their JSON +// encoding so a regex/contains can still match structured values. +func metadataString(v any) string { + switch val := v.(type) { + case nil: + return "" + case string: + return val + case bool: + if val { + return "true" + } + return "false" + case float64: + return strconv.FormatFloat(val, 'f', -1, 64) + case json.Number: + return val.String() + default: + if b, err := json.Marshal(val); err == nil { + return string(b) + } + return fmt.Sprintf("%v", val) + } +} + +// metadataFloat coerces a stored metadata value to float64 for numeric +// comparison. Returns false when the value is not numeric. +func metadataFloat(v any) (float64, bool) { + switch val := v.(type) { + case float64: + return val, true + case float32: + return float64(val), true + case int: + return float64(val), true + case int64: + return float64(val), true + case json.Number: + f, err := val.Float64() + return f, err == nil + case string: + f, err := strconv.ParseFloat(val, 64) + return f, err == nil + } + return 0, false +} + +// metadataBool coerces a stored metadata value to bool. Returns false (ok=false) +// when the value isn't a recognizable boolean. +func metadataBool(v any) (bool, bool) { + switch val := v.(type) { + case bool: + return val, true + case string: + b, err := strconv.ParseBool(val) + return b, err == nil + } + return false, false +} + // evalTags handles contains / not_contains / in for the tags slice field. func evalTags(c *CompiledCondition, tags []string) bool { switch c.Op { @@ -497,11 +718,13 @@ func toStringSlice(v interface{}) ([]string, bool) { // Unknown types are rejected by ValidateActions. Read-time tolerates unknown // types by skipping them with a logged warning (see sync/rule_resolver). var validActionTypes = map[string]bool{ - "set_category": true, - "add_tag": true, - "remove_tag": true, - "add_comment": true, - "assign_series": true, + "set_category": true, + "add_tag": true, + "remove_tag": true, + "add_comment": true, + "assign_series": true, + "set_metadata": true, + "remove_metadata": true, } // tagSlugPattern enforces the tag slug format: lowercase alphanumerics with @@ -520,7 +743,7 @@ func (s *Service) ValidateActions(ctx context.Context, actions []RuleAction) err seenCategory := false for _, a := range actions { if !validActionTypes[a.Type] { - return fmt.Errorf("%w: unknown action type %q (expected set_category|add_tag|remove_tag|add_comment|assign_series)", ErrInvalidParameter, a.Type) + return fmt.Errorf("%w: unknown action type %q (expected set_category|add_tag|remove_tag|add_comment|assign_series|set_metadata|remove_metadata)", ErrInvalidParameter, a.Type) } switch a.Type { case "set_category": @@ -559,6 +782,21 @@ func (s *Service) ValidateActions(ctx context.Context, actions []RuleAction) err return fmt.Errorf("%w: assign_series series_short_id %q not found", ErrInvalidParameter, a.SeriesShortID) } } + case "set_metadata": + if err := validateMetadataKey(a.MetadataKey); err != nil { + return fmt.Errorf("%w (set_metadata)", err) + } + valBytes, err := json.Marshal(a.MetadataValue) + if err != nil { + return fmt.Errorf("%w: set_metadata value is not JSON-serializable: %v", ErrInvalidParameter, err) + } + if len(valBytes) > maxMetadataValBytes { + return fmt.Errorf("%w: set_metadata value exceeds %d bytes", ErrInvalidParameter, maxMetadataValBytes) + } + case "remove_metadata": + if err := validateMetadataKey(a.MetadataKey); err != nil { + return fmt.Errorf("%w (remove_metadata)", err) + } } } return nil @@ -1226,7 +1464,7 @@ const transactionContextQuery = `SELECT t.id, t.provider_name, COALESCE(t.provid COALESCE(t.provider_category_primary, ''), COALESCE(t.provider_category_detailed, ''), t.pending, bc.provider, t.account_id::text, COALESCE(u.id::text, ''), COALESCE(u.name, ''), COALESCE(array_agg(DISTINCT tag.slug) FILTER (WHERE tag.slug IS NOT NULL), ARRAY[]::text[]), - COALESCE(rs.short_id, ''), (t.series_id IS NOT NULL) + COALESCE(rs.short_id, ''), (t.series_id IS NOT NULL), t.metadata FROM transactions t JOIN accounts a ON t.account_id = a.id JOIN bank_connections bc ON a.connection_id = bc.id @@ -1238,12 +1476,18 @@ const transactionContextQuery = `SELECT t.id, t.provider_name, COALESCE(t.provid AND (a.is_dependent_linked = FALSE OR NOT EXISTS (SELECT 1 FROM transaction_matches tm WHERE tm.dependent_transaction_id = t.id))` // transactionContextGroupBy is the GROUP BY clause matching transactionContextQuery. -const transactionContextGroupBy = ` GROUP BY t.id, t.provider_name, t.provider_merchant_name, t.amount, t.provider_category_primary, t.provider_category_detailed, t.pending, bc.provider, t.account_id, u.id, u.name, rs.short_id, t.series_id` +const transactionContextGroupBy = ` GROUP BY t.id, t.provider_name, t.provider_merchant_name, t.amount, t.provider_category_primary, t.provider_category_detailed, t.pending, bc.provider, t.account_id, u.id, u.name, rs.short_id, t.series_id, t.metadata` + +// transactionContextColumns is the number of columns selected by +// transactionContextQuery (and bound by scanTransactionContextRow). Callers size +// their scan-dest slice with this so adding a column is a single-line change. +const transactionContextColumns = 15 // transactionContextRow holds a scanned transaction row for rule evaluation. type transactionContextRow struct { - id pgtype.UUID - tctx TransactionContext + id pgtype.UUID + tctx TransactionContext + metadataRaw []byte // raw JSONB; unmarshalled into tctx.Metadata by finalize } func scanTransactionContextRow(dest []any) *transactionContextRow { @@ -1262,9 +1506,23 @@ func scanTransactionContextRow(dest []any) *transactionContextRow { dest[11] = &r.tctx.Tags dest[12] = &r.tctx.SeriesShortID dest[13] = &r.tctx.InSeries + dest[14] = &r.metadataRaw return r } +// finalize unmarshals the raw metadata JSONB into tctx.Metadata. Call after +// rows.Scan. A malformed blob leaves Metadata nil (metadata conditions simply +// won't match) rather than failing the whole apply pass. +func (r *transactionContextRow) finalize() { + if len(r.metadataRaw) == 0 { + return + } + var meta map[string]any + if err := json.Unmarshal(r.metadataRaw, &meta); err == nil { + r.tctx.Metadata = meta + } +} + // ApplyRuleRetroactively applies a single rule to all existing non-deleted // transactions matching its condition. Materializes set_category (SQL UPDATE, // skipped on category_override=TRUE rows), add_tag (upsert transaction_tags), @@ -1304,6 +1562,8 @@ func (s *Service) ApplyRuleRetroactively(ctx context.Context, ruleID string) (in var tagAdds []string var tagRemoves []string var seriesAssign *RuleAction + metadataSet := map[string]any{} + var metadataRemove []string for _, a := range rule.Actions { switch a.Type { case "set_category": @@ -1323,9 +1583,29 @@ func (s *Service) ApplyRuleRetroactively(ctx context.Context, ruleID string) (in case "assign_series": aCopy := a seriesAssign = &aCopy + case "set_metadata": + if a.MetadataKey == "" { + continue + } + if i := indexExactStr(metadataRemove, a.MetadataKey); i >= 0 { + metadataRemove = append(metadataRemove[:i], metadataRemove[i+1:]...) + } + metadataSet[a.MetadataKey] = a.MetadataValue + case "remove_metadata": + if a.MetadataKey == "" { + continue + } + // Last-writer-wins: a remove after a set of the same key must delete + // it. Drop any queued set and queue the delete; the SQL + // `metadata - keys` is a harmless no-op on rows lacking the key. + delete(metadataSet, a.MetadataKey) + if indexExactStr(metadataRemove, a.MetadataKey) < 0 { + metadataRemove = append(metadataRemove, a.MetadataKey) + } } } - hasWriteAction := categorySetCatID.Valid || len(tagAdds) > 0 || len(tagRemoves) > 0 || seriesAssign != nil + hasWriteAction := categorySetCatID.Valid || len(tagAdds) > 0 || len(tagRemoves) > 0 || seriesAssign != nil || + len(metadataSet) > 0 || len(metadataRemove) > 0 if !hasWriteAction { return 0, fmt.Errorf("%w: rule has no applicable actions", ErrInvalidParameter) } @@ -1358,12 +1638,13 @@ func (s *Service) ApplyRuleRetroactively(ctx context.Context, ruleID string) (in rowCount := 0 for rows.Next() { rowCount++ - dest := make([]any, 14) + dest := make([]any, transactionContextColumns) r := scanTransactionContextRow(dest) if err := rows.Scan(dest...); err != nil { rows.Close() return totalMatched, fmt.Errorf("scan transaction: %w", err) } + r.finalize() lastID = r.id if EvaluateCondition(compiled, r.tctx) { matchIDs = append(matchIDs, r.id) @@ -1437,6 +1718,16 @@ func (s *Service) ApplyRuleRetroactively(ctx context.Context, ruleID string) (in } } + // set_metadata / remove_metadata: one bulk UPDATE merges the rule's net + // metadata intent into every matched row (sets overwrite, removes delete; + // the two are disjoint by construction). + if len(metadataSet) > 0 || len(metadataRemove) > 0 { + if err := applyMetadataToTxns(ctx, tx, matchIDs, metadataSet, metadataRemove); err != nil { + tx.Rollback(ctx) + return totalMatched, err + } + } + // rule_applied audit trail — one per action intent per matched txn. appliedRule := ruleapply.Rule{ID: ruleUUID, ShortID: ruleShortID, Name: ruleName} for _, action := range rule.Actions { @@ -1493,10 +1784,53 @@ func actionAuditFields(a RuleAction) (string, string) { return "series", a.SeriesShortID } return "series", a.MerchantKey + case "set_metadata": + return "metadata", a.MetadataKey + case "remove_metadata": + return "metadata_remove", a.MetadataKey } return "", "" } +// applyMetadataToTxns merges a rule's net metadata intent into every listed +// transaction in one UPDATE, sharing the caller's tx. set keys overwrite, +// remove keys delete; the two are disjoint by resolver/extraction construction +// so `(metadata - removeKeys) || setObj` applies both unambiguously. +func applyMetadataToTxns(ctx context.Context, tx pgx.Tx, txnIDs []pgtype.UUID, set map[string]any, remove []string) error { + setJSON := "{}" + if len(set) > 0 { + b, err := json.Marshal(set) + if err != nil { + return fmt.Errorf("marshal rule metadata set: %w", err) + } + setJSON = string(b) + } + removeKeys := remove + if removeKeys == nil { + removeKeys = []string{} + } + if _, err := tx.Exec(ctx, + `UPDATE transactions + SET metadata = (metadata - $1::text[]) || $2::jsonb, updated_at = NOW() + WHERE id = ANY($3) AND deleted_at IS NULL`, + removeKeys, setJSON, txnIDs); err != nil { + return fmt.Errorf("apply rule metadata: %w", err) + } + return nil +} + +// indexExactStr returns the index of target in slice using exact +// (case-sensitive) comparison, or -1. Metadata keys are case-sensitive JSONB +// keys, so they must not be folded. +func indexExactStr(slice []string, target string) int { + for i, s := range slice { + if s == target { + return i + } + } + return -1 +} + // ApplyAllRulesRetroactively applies all active rules to existing transactions // in pipeline-stage order (priority ASC, created_at ASC). Materializes // set_category (last-writer-wins), add_tag, and remove_tag. add_comment stays @@ -1568,25 +1902,29 @@ func (s *Service) ApplyAllRulesRetroactively(ctx context.Context) (map[string]in // Per transaction: fold rules to determine net action intents, mirroring // the sync resolver. Category: last-writer-wins. Tags: net-diff. + // Metadata: last-writer-wins per key, net-diff set↔remove. type txnIntent struct { - txnID pgtype.UUID - catID pgtype.UUID - catSlug string - catRule *compiledRule - tagAdds map[string]*compiledRule // slug → first rule that added it - tagRemoves map[string]*compiledRule - matchingRules []*compiledRule // every rule whose condition matched (for rule_applied) + txnID pgtype.UUID + catID pgtype.UUID + catSlug string + catRule *compiledRule + tagAdds map[string]*compiledRule // slug → first rule that added it + tagRemoves map[string]*compiledRule + metadataSet map[string]any // key → net value to write + metadataRemove []string // net keys to delete + matchingRules []*compiledRule // every rule whose condition matched (for rule_applied) } var intents []txnIntent rowCount := 0 for rows.Next() { rowCount++ - dest := make([]any, 14) + dest := make([]any, transactionContextColumns) r := scanTransactionContextRow(dest) if err := rows.Scan(dest...); err != nil { rows.Close() return hitCounts, fmt.Errorf("scan transaction: %w", err) } + r.finalize() lastID = r.id tctx := r.tctx if len(tctx.Tags) > 0 { @@ -1595,9 +1933,10 @@ func (s *Service) ApplyAllRulesRetroactively(ctx context.Context) (map[string]in tctx.Tags = cp } intent := txnIntent{ - txnID: r.id, - tagAdds: make(map[string]*compiledRule), - tagRemoves: make(map[string]*compiledRule), + txnID: r.id, + tagAdds: make(map[string]*compiledRule), + tagRemoves: make(map[string]*compiledRule), + metadataSet: make(map[string]any), } for i := range compiled { cr := &compiled[i] @@ -1641,10 +1980,36 @@ func (s *Service) ApplyAllRulesRetroactively(ctx context.Context) (map[string]in } } tctx.Tags = sliceutil.DropFold(tctx.Tags, a.TagSlug) + case "set_metadata": + if a.MetadataKey == "" { + continue + } + if i := indexExactStr(intent.metadataRemove, a.MetadataKey); i >= 0 { + intent.metadataRemove = append(intent.metadataRemove[:i], intent.metadataRemove[i+1:]...) + } + intent.metadataSet[a.MetadataKey] = a.MetadataValue + if tctx.Metadata == nil { + tctx.Metadata = make(map[string]any) + } + tctx.Metadata[a.MetadataKey] = a.MetadataValue + case "remove_metadata": + if a.MetadataKey == "" { + continue + } + // Last-writer-wins: a remove after a set of the same + // key must delete it. Drop any queued set and queue + // the delete; `metadata - keys` is a no-op on rows + // lacking the key. + delete(intent.metadataSet, a.MetadataKey) + if indexExactStr(intent.metadataRemove, a.MetadataKey) < 0 { + intent.metadataRemove = append(intent.metadataRemove, a.MetadataKey) + } + delete(tctx.Metadata, a.MetadataKey) } } } - if intent.catRule != nil || len(intent.tagAdds) > 0 || len(intent.tagRemoves) > 0 { + if intent.catRule != nil || len(intent.tagAdds) > 0 || len(intent.tagRemoves) > 0 || + len(intent.metadataSet) > 0 || len(intent.metadataRemove) > 0 { intents = append(intents, intent) } } @@ -1713,6 +2078,14 @@ func (s *Service) ApplyAllRulesRetroactively(ctx context.Context) (map[string]in break } + // set_metadata / remove_metadata: merge the net intent into this row. + if len(it.metadataSet) > 0 || len(it.metadataRemove) > 0 { + if err := applyMetadataToTxns(ctx, tx, []pgtype.UUID{it.txnID}, it.metadataSet, it.metadataRemove); err != nil { + aborted = true + break + } + } + // rule_applied audit: one per matching rule per txn. // // Unlike ApplyRuleRetroactively, this path emits one annotation per @@ -1922,11 +2295,12 @@ func (s *Service) loadTransactionContext(ctx context.Context, idOrShort string) } return nil, fmt.Errorf("%w: transaction not found", ErrNotFound) } - dest := make([]any, 14) + dest := make([]any, transactionContextColumns) r := scanTransactionContextRow(dest) if err := rows.Scan(dest...); err != nil { return nil, fmt.Errorf("scan transaction context: %w", err) } + r.finalize() return &r.tctx, nil } @@ -1957,7 +2331,7 @@ func (s *Service) previewRuleInternal(ctx context.Context, excludeRuleID *pgtype baseQuery := `SELECT t.id, t.provider_name, COALESCE(t.provider_merchant_name, ''), t.amount, COALESCE(t.provider_category_primary, ''), COALESCE(t.provider_category_detailed, ''), t.pending, bc.provider, t.account_id::text, COALESCE(u.id::text, ''), COALESCE(u.name, ''), - t.date, COALESCE(c.slug, ''), COALESCE(rs.short_id, ''), (t.series_id IS NOT NULL) + t.date, COALESCE(c.slug, ''), COALESCE(rs.short_id, ''), (t.series_id IS NOT NULL), t.metadata FROM transactions t JOIN accounts a ON t.account_id = a.id JOIN bank_connections bc ON a.connection_id = bc.id @@ -2002,18 +2376,25 @@ func (s *Service) previewRuleInternal(ctx context.Context, excludeRuleID *pgtype for rows.Next() { rowCount++ var ( - id pgtype.UUID - tctx TransactionContext - date pgtype.Date - catSlug string + id pgtype.UUID + tctx TransactionContext + date pgtype.Date + catSlug string + metadataRaw []byte ) if err := rows.Scan(&id, &tctx.Name, &tctx.MerchantName, &tctx.Amount, &tctx.CategoryPrimary, &tctx.CategoryDetailed, &tctx.Pending, &tctx.Provider, &tctx.AccountID, &tctx.UserID, &tctx.UserName, - &date, &catSlug, &tctx.SeriesShortID, &tctx.InSeries); err != nil { + &date, &catSlug, &tctx.SeriesShortID, &tctx.InSeries, &metadataRaw); err != nil { rows.Close() return nil, fmt.Errorf("scan transaction: %w", err) } + if len(metadataRaw) > 0 { + var meta map[string]any + if err := json.Unmarshal(metadataRaw, &meta); err == nil { + tctx.Metadata = meta + } + } lastID = id result.TotalScanned++ diff --git a/internal/service/rules_metadata_integration_test.go b/internal/service/rules_metadata_integration_test.go new file mode 100644 index 00000000..7922ca41 --- /dev/null +++ b/internal/service/rules_metadata_integration_test.go @@ -0,0 +1,179 @@ +//go:build integration && !lite + +package service_test + +import ( + "context" + "encoding/json" + "testing" + + "breadbox/internal/pgconv" + "breadbox/internal/service" + "breadbox/internal/testutil" +) + +// readMeta loads a transaction's metadata blob as a Go map via the service. +func readMeta(t *testing.T, svc *service.Service, id string) map[string]any { + t.Helper() + txn, err := svc.GetTransaction(context.Background(), id) + if err != nil { + t.Fatalf("GetTransaction: %v", err) + } + var m map[string]any + if err := json.Unmarshal(txn.Metadata, &m); err != nil { + t.Fatalf("unmarshal metadata %q: %v", string(txn.Metadata), err) + } + return m +} + +// TestApplyRuleRetroactively_SetAndRemoveMetadata verifies that a rule's +// set_metadata / remove_metadata actions materialize onto existing transactions +// during retroactive apply, touching only the named keys. +func TestApplyRuleRetroactively_SetAndRemoveMetadata(t *testing.T) { + svc, queries, _ := newService(t) + ctx := context.Background() + + user := testutil.MustCreateUser(t, queries, "User") + conn := testutil.MustCreateConnection(t, queries, user.ID, "item_1") + acct := testutil.MustCreateAccount(t, queries, conn.ID, "ext_1", "Checking") + txn := testutil.MustCreateTransaction(t, queries, acct.ID, "txn_1", "IRS Payment", 5000, "2026-04-15") + other := testutil.MustCreateTransaction(t, queries, acct.ID, "txn_2", "Grocery Store", 4500, "2026-04-16") + txnID := pgconv.FormatUUID(txn.ID) + + // Pre-seed a key that remove_metadata should delete, plus a sibling that must survive. + if err := svc.SetTransactionMetadata(ctx, txnID, "needs_receipt", true); err != nil { + t.Fatalf("seed needs_receipt: %v", err) + } + if err := svc.SetTransactionMetadata(ctx, txnID, "keep_me", "yes"); err != nil { + t.Fatalf("seed keep_me: %v", err) + } + + rule, err := svc.CreateTransactionRule(ctx, service.CreateTransactionRuleParams{ + Name: "IRS metadata", + Actions: []service.RuleAction{ + {Type: "set_metadata", MetadataKey: "tax_deductible", MetadataValue: true}, + {Type: "set_metadata", MetadataKey: "category_hint", MetadataValue: "taxes"}, + {Type: "remove_metadata", MetadataKey: "needs_receipt"}, + }, + Conditions: service.Condition{Field: "provider_name", Op: "contains", Value: "irs"}, + Priority: 10, + Actor: service.Actor{Type: "user", Name: "Test"}, + }) + if err != nil { + t.Fatalf("CreateTransactionRule: %v", err) + } + + matched, err := svc.ApplyRuleRetroactively(ctx, rule.ID) + if err != nil { + t.Fatalf("ApplyRuleRetroactively: %v", err) + } + if matched != 1 { + t.Fatalf("expected 1 matched txn, got %d", matched) + } + + meta := readMeta(t, svc, txnID) + if meta["tax_deductible"] != true { + t.Errorf("tax_deductible: got %v, want true", meta["tax_deductible"]) + } + if meta["category_hint"] != "taxes" { + t.Errorf("category_hint: got %v, want taxes", meta["category_hint"]) + } + if _, ok := meta["needs_receipt"]; ok { + t.Errorf("needs_receipt should have been removed; meta=%v", meta) + } + if meta["keep_me"] != "yes" { + t.Errorf("keep_me sibling should survive untouched; got %v", meta["keep_me"]) + } + + // The non-matching transaction must be untouched (empty blob). + otherMeta := readMeta(t, svc, pgconv.FormatUUID(other.ID)) + if len(otherMeta) != 0 { + t.Errorf("non-matching txn metadata should be empty, got %v", otherMeta) + } +} + +// TestApplyRuleRetroactively_SetThenRemoveSameKeyDeletes verifies last-writer-wins: +// a rule that sets a key and then removes it must delete a pre-existing value, +// not silently revert to it. +func TestApplyRuleRetroactively_SetThenRemoveSameKeyDeletes(t *testing.T) { + svc, queries, _ := newService(t) + ctx := context.Background() + + user := testutil.MustCreateUser(t, queries, "User") + conn := testutil.MustCreateConnection(t, queries, user.ID, "item_1") + acct := testutil.MustCreateAccount(t, queries, conn.ID, "ext_1", "Checking") + txn := testutil.MustCreateTransaction(t, queries, acct.ID, "txn_1", "IRS Payment", 5000, "2026-04-15") + txnID := pgconv.FormatUUID(txn.ID) + + if err := svc.SetTransactionMetadata(ctx, txnID, "flag", "old"); err != nil { + t.Fatalf("seed flag: %v", err) + } + + rule, err := svc.CreateTransactionRule(ctx, service.CreateTransactionRuleParams{ + Name: "Set then remove", + Actions: []service.RuleAction{ + {Type: "set_metadata", MetadataKey: "flag", MetadataValue: "new"}, + {Type: "remove_metadata", MetadataKey: "flag"}, + }, + Conditions: service.Condition{Field: "provider_name", Op: "contains", Value: "irs"}, + Priority: 10, + Actor: service.Actor{Type: "user", Name: "Test"}, + }) + if err != nil { + t.Fatalf("CreateTransactionRule: %v", err) + } + + if _, err := svc.ApplyRuleRetroactively(ctx, rule.ID); err != nil { + t.Fatalf("ApplyRuleRetroactively: %v", err) + } + + meta := readMeta(t, svc, txnID) + if _, ok := meta["flag"]; ok { + t.Errorf("set-then-remove of a pre-existing key must delete it; meta=%v", meta) + } +} + +// TestApplyRuleRetroactively_MetadataCondition verifies a rule whose condition +// reads metadata. matches existing transactions by their stored metadata. +func TestApplyRuleRetroactively_MetadataCondition(t *testing.T) { + svc, queries, _ := newService(t) + ctx := context.Background() + + user := testutil.MustCreateUser(t, queries, "User") + conn := testutil.MustCreateConnection(t, queries, user.ID, "item_1") + acct := testutil.MustCreateAccount(t, queries, conn.ID, "ext_1", "Checking") + match := testutil.MustCreateTransaction(t, queries, acct.ID, "m1", "Anything", 1000, "2026-04-15") + noMatch := testutil.MustCreateTransaction(t, queries, acct.ID, "m2", "Anything", 1000, "2026-04-16") + + if err := svc.SetTransactionMetadata(ctx, pgconv.FormatUUID(match.ID), "reimbursable", true); err != nil { + t.Fatalf("seed reimbursable: %v", err) + } + + rule, err := svc.CreateTransactionRule(ctx, service.CreateTransactionRuleParams{ + Name: "Flag reimbursables", + Actions: []service.RuleAction{{Type: "set_metadata", MetadataKey: "expense_report", MetadataValue: "Q2"}}, + Conditions: service.Condition{Field: "metadata.reimbursable", Op: "eq", Value: true}, + Priority: 10, + Actor: service.Actor{Type: "user", Name: "Test"}, + }) + if err != nil { + t.Fatalf("CreateTransactionRule: %v", err) + } + + matched, err := svc.ApplyRuleRetroactively(ctx, rule.ID) + if err != nil { + t.Fatalf("ApplyRuleRetroactively: %v", err) + } + if matched != 1 { + t.Fatalf("expected the metadata condition to match exactly 1 txn, got %d", matched) + } + + matchMeta := readMeta(t, svc, pgconv.FormatUUID(match.ID)) + if matchMeta["expense_report"] != "Q2" { + t.Errorf("matched txn should get expense_report=Q2, got %v", matchMeta["expense_report"]) + } + noMatchMeta := readMeta(t, svc, pgconv.FormatUUID(noMatch.ID)) + if _, ok := noMatchMeta["expense_report"]; ok { + t.Errorf("non-reimbursable txn should not be touched; meta=%v", noMatchMeta) + } +} diff --git a/internal/service/rules_metadata_test.go b/internal/service/rules_metadata_test.go new file mode 100644 index 00000000..2a43d4ef --- /dev/null +++ b/internal/service/rules_metadata_test.go @@ -0,0 +1,92 @@ +//go:build !lite + +package service + +import ( + "context" + "strings" + "testing" +) + +func TestValidateCondition_MetadataField(t *testing.T) { + cases := []struct { + name string + cond Condition + wantErr bool + }{ + {"eq bool", Condition{Field: "metadata.tax_deductible", Op: "eq", Value: true}, false}, + {"eq string", Condition{Field: "metadata.trip", Op: "eq", Value: "japan"}, false}, + {"gt numeric", Condition{Field: "metadata.cents", Op: "gt", Value: 100}, false}, + {"exists no value", Condition{Field: "metadata.foo", Op: "exists"}, false}, + {"not_exists no value", Condition{Field: "metadata.foo", Op: "not_exists"}, false}, + {"matches valid regex", Condition{Field: "metadata.code", Op: "matches", Value: "^ABC"}, false}, + {"in non-empty array", Condition{Field: "metadata.trip", Op: "in", Value: []interface{}{"a", "b"}}, false}, + {"empty key rejected", Condition{Field: "metadata.", Op: "eq", Value: "x"}, true}, + {"unknown op rejected", Condition{Field: "metadata.k", Op: "weird", Value: "x"}, true}, + {"gt non-numeric rejected", Condition{Field: "metadata.k", Op: "gt", Value: "abc"}, true}, + {"matches non-string rejected", Condition{Field: "metadata.k", Op: "matches", Value: 5}, true}, + {"in empty array rejected", Condition{Field: "metadata.k", Op: "in", Value: []interface{}{}}, true}, + {"oversize key rejected", Condition{Field: "metadata." + strings.Repeat("x", 200), Op: "eq", Value: "y"}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if err := ValidateCondition(tc.cond); (err != nil) != tc.wantErr { + t.Errorf("ValidateCondition err=%v wantErr=%v", err, tc.wantErr) + } + }) + } +} + +func TestEvaluateCondition_MetadataField(t *testing.T) { + tctx := TransactionContext{Metadata: map[string]any{ + "tax_deductible": true, + "trip": "japan-2026", + "cents": float64(12050), + "notes": "warranty card inside", + }} + cases := []struct { + cond Condition + want bool + }{ + {Condition{Field: "metadata.tax_deductible", Op: "eq", Value: true}, true}, + {Condition{Field: "metadata.trip", Op: "eq", Value: "JAPAN-2026"}, true}, + {Condition{Field: "metadata.trip", Op: "contains", Value: "japan"}, true}, + {Condition{Field: "metadata.cents", Op: "gte", Value: 12050}, true}, + {Condition{Field: "metadata.cents", Op: "lt", Value: 100}, false}, + {Condition{Field: "metadata.notes", Op: "matches", Value: "warranty"}, true}, + {Condition{Field: "metadata.foo", Op: "exists"}, false}, + {Condition{Field: "metadata.foo", Op: "not_exists"}, true}, + {Condition{Field: "metadata.foo", Op: "eq", Value: "x"}, false}, // absent → no match + {Condition{Field: "metadata.foo", Op: "neq", Value: "x"}, false}, // absent → no match (must be present) + } + for _, tc := range cases { + cc := mustCompileSvc(t, tc.cond) + if got := EvaluateCondition(cc, tctx); got != tc.want { + t.Errorf("EvaluateCondition(%s %s %v) = %v, want %v", tc.cond.Field, tc.cond.Op, tc.cond.Value, got, tc.want) + } + } +} + +func TestValidateActions_Metadata(t *testing.T) { + svc := &Service{} + ctx := context.Background() + cases := []struct { + name string + actions []RuleAction + wantErr bool + }{ + {"set_metadata bool", []RuleAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: true}}, false}, + {"set_metadata object", []RuleAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: map[string]any{"a": 1}}}, false}, + {"remove_metadata", []RuleAction{{Type: "remove_metadata", MetadataKey: "k"}}, false}, + {"set_metadata empty key", []RuleAction{{Type: "set_metadata", MetadataKey: "", MetadataValue: 1}}, true}, + {"remove_metadata empty key", []RuleAction{{Type: "remove_metadata", MetadataKey: ""}}, true}, + {"set_metadata oversize value", []RuleAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: strings.Repeat("x", 5000)}}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if err := svc.ValidateActions(ctx, tc.actions); (err != nil) != tc.wantErr { + t.Errorf("ValidateActions err=%v wantErr=%v", err, tc.wantErr) + } + }) + } +} diff --git a/internal/service/types.go b/internal/service/types.go index 882c3ef4..03a7a02f 100644 --- a/internal/service/types.go +++ b/internal/service/types.go @@ -462,6 +462,11 @@ type RuleAction struct { SeriesShortID string `json:"series_short_id,omitempty"` MerchantKey string `json:"merchant_key,omitempty"` CreateIfMissing bool `json:"create_if_missing,omitempty"` + // set_metadata / remove_metadata fields. MetadataKey is the metadata blob + // key (≤128 chars). MetadataValue is the JSON value to write for + // set_metadata (any JSON-serializable value); unused by remove_metadata. + MetadataKey string `json:"metadata_key,omitempty"` + MetadataValue any `json:"metadata_value,omitempty"` } // ActivityEntry represents a single event in a transaction's activity timeline. @@ -570,6 +575,11 @@ type TransactionContext struct { // InSeries reports whether the transaction is linked to any recurring // series (field: "in_series"). InSeries bool + // Metadata holds the transaction's free-form metadata blob so conditions on + // dotted fields (field: "metadata.") can read arbitrary enrichment + // values. Updated mid-resolver as earlier-stage set_metadata / remove_metadata + // actions apply, so later-stage rules observe the running blob. + Metadata map[string]any } type TransactionRuleResponse struct { diff --git a/internal/sync/engine.go b/internal/sync/engine.go index 830b97fd..6bd5a7a3 100644 --- a/internal/sync/engine.go +++ b/internal/sync/engine.go @@ -717,6 +717,19 @@ func (e *Engine) applyRulesToTransaction(ctx context.Context, tx pgx.Tx, txn *pr tctx.InSeries = true tctx.SeriesShortID = resolver.SeriesShortID(dbTxn.SeriesID) } + // Seed the metadata blob so conditions on field="metadata." can read + // the transaction's current enrichment values, and so chaining rules see + // earlier-stage set_metadata / remove_metadata writes. A new transaction + // starts from {} (DEFAULT); re-synced rows carry their persisted blob. + if len(dbTxn.Metadata) > 0 { + var meta map[string]any + if err := json.Unmarshal(dbTxn.Metadata, &meta); err == nil { + tctx.Metadata = meta + } else { + e.logger.Warn("unmarshal transaction metadata for rule evaluation failed; continuing without metadata context", + "transaction_id", pgconv.FormatUUID(dbTxn.ID), "err", err) + } + } // For changed transactions, load the current tag slugs so tag-based // conditions can match. New transactions start with empty tags (any tags @@ -832,9 +845,48 @@ func (e *Engine) applyRulesToTransaction(ctx context.Context, tx pgx.Tx, txn *pr } } + // set_metadata / remove_metadata: merge net changes into the metadata JSONB + // in a single UPDATE within the sync tx. Removes and sets are disjoint + // (the resolver net-diffs them), so `(metadata - removeKeys) || setObj` + // applies both unambiguously. Like assign_series, metadata writes don't emit + // a dedicated timeline annotation — the rule's hit_count records the firing. + if len(result.MetadataSet) > 0 || len(result.MetadataRemove) > 0 { + if err := e.applyMetadataFromRule(ctx, tx, dbTxn.ID, result.MetadataSet, result.MetadataRemove); err != nil { + return result.Sources, err + } + } + return result.Sources, nil } +// applyMetadataFromRule applies a rule's net set_metadata / remove_metadata +// intents to a transaction's metadata blob in a single UPDATE, sharing the sync +// tx. Set keys overwrite, remove keys delete; the two sets are disjoint by +// resolver construction. +func (e *Engine) applyMetadataFromRule(ctx context.Context, tx pgx.Tx, txnID pgtype.UUID, set map[string]any, remove []string) error { + setJSON := "{}" + if len(set) > 0 { + b, err := json.Marshal(set) + if err != nil { + return fmt.Errorf("marshal rule metadata set: %w", err) + } + setJSON = string(b) + } + removeKeys := remove + if removeKeys == nil { + removeKeys = []string{} + } + _, err := tx.Exec(ctx, + `UPDATE transactions + SET metadata = (metadata - $2::text[]) || $3::jsonb, updated_at = NOW() + WHERE id = $1 AND deleted_at IS NULL`, + txnID, removeKeys, setJSON) + if err != nil { + return fmt.Errorf("apply rule metadata: %w", err) + } + return nil +} + // loadTagSlugsInTx returns the current tag slugs for a transaction using the // sync DB transaction. Used by applyRulesToTransaction for changed txns so // tag-based conditions can match real data. diff --git a/internal/sync/rule_resolver.go b/internal/sync/rule_resolver.go index 64693752..7f0268b1 100644 --- a/internal/sync/rule_resolver.go +++ b/internal/sync/rule_resolver.go @@ -31,10 +31,10 @@ type Condition struct { // TransactionContext holds the fields available for rule evaluation during sync. type TransactionContext struct { Name string - MerchantName string // may be empty + MerchantName string // may be empty Amount float64 - CategoryPrimary string // raw provider category - CategoryDetailed string // raw provider detailed category + CategoryPrimary string // raw provider category + CategoryDetailed string // raw provider detailed category // Category is the transaction's *assigned* category slug (distinct from // CategoryPrimary's raw provider value). It updates mid-resolver as // earlier-stage rules' set_category actions fire, so later-stage rules @@ -59,6 +59,12 @@ type TransactionContext struct { // InSeries reports whether the transaction is linked to any recurring // series (field: "in_series"). Same timing caveat as SeriesShortID. InSeries bool + // Metadata holds the transaction's free-form metadata blob (the JSONB + // `metadata` column) so conditions on dotted fields (field: + // "metadata.") can read arbitrary enrichment values. Updated + // mid-resolver as earlier-stage set_metadata / remove_metadata actions + // apply, so later-stage rules observe the running blob. + Metadata map[string]any } // RuleActionSource tracks which rule contributed which action for audit. @@ -68,8 +74,8 @@ type RuleActionSource struct { RuleID pgtype.UUID RuleShortID string RuleName string - ActionField string // "category", "tag", "comment" - ActionValue string // slug for category/tag, content for comment + ActionField string // "category", "tag", "tag_remove", "comment", "series", "metadata", "metadata_remove" + ActionValue string // slug for category/tag, content for comment, key for metadata } // RuleActions holds the merged actions to apply to a transaction after resolving @@ -105,6 +111,15 @@ type RuleActions struct { // Last-writer-wins: the highest-priority matching rule owns the assignment // (a transaction belongs to at most one series). SeriesAssign *SeriesAssignIntent + // MetadataSet maps metadata keys to the value the net-surviving set_metadata + // action wrote. Last-writer-wins per key across the pipeline. A later-stage + // remove_metadata for the same key cancels the set (the key won't appear + // here). + MetadataSet map[string]any + // MetadataRemove is the net list of metadata keys to delete. A key set by an + // earlier-stage rule then removed by a later one cancels (appears in + // neither map); a key removed then re-set ends up only in MetadataSet. + MetadataRemove []string // Sources records per-action provenance for the audit trail. For // set_category, only the winning (last) rule's source is retained. // For tag actions, only net-surviving adds/removes have sources. @@ -122,6 +137,8 @@ type typedAction struct { SeriesShortID string MerchantKey string CreateIfMissing bool + MetadataKey string + MetadataValue any } // SeriesAssignIntent is the resolved assign_series action: link the @@ -138,9 +155,9 @@ type SeriesAssignIntent struct { // All matching rules contribute actions (merge non-conflicting). type RuleResolver struct { rules []compiledRule - slugCache map[[16]byte]string // category UUID bytes -> slug - slugToID map[string]pgtype.UUID // category slug -> UUID (reverse cache) - seriesShortID map[[16]byte]string // recurring_series UUID bytes -> short_id + slugCache map[[16]byte]string // category UUID bytes -> slug + slugToID map[string]pgtype.UUID // category slug -> UUID (reverse cache) + seriesShortID map[[16]byte]string // recurring_series UUID bytes -> short_id uncategorizedID pgtype.UUID hitCounts map[[16]byte]int // rule UUID bytes -> hit count accumulator } @@ -349,6 +366,12 @@ func parseTypedActions(raw []byte, ruleID pgtype.UUID, logger *slog.Logger) []ty merchantKey, _ := m["merchant_key"].(string) createIfMissing, _ := m["create_if_missing"].(bool) out = append(out, typedAction{Type: t, SeriesShortID: seriesShortID, MerchantKey: merchantKey, CreateIfMissing: createIfMissing}) + case "set_metadata": + key, _ := m["metadata_key"].(string) + out = append(out, typedAction{Type: t, MetadataKey: key, MetadataValue: m["metadata_value"]}) + case "remove_metadata": + key, _ := m["metadata_key"].(string) + out = append(out, typedAction{Type: t, MetadataKey: key}) default: logger.Warn("skipping unknown rule action type", "rule_id", pgconv.FormatUUID(ruleID), "type", t) @@ -489,6 +512,20 @@ func (r *RuleResolver) ResolveWithContext(providerName string, txn TransactionCo if len(txn.Tags) > 0 { tctx.Tags = append([]string(nil), txn.Tags...) } + // Snapshot the keys present in the original DB blob and work on a copy of the + // metadata map so chaining mutations don't leak into the caller's map. + // origMetaKeys lets the remove_metadata net-diff below distinguish a + // pre-existing key (a set-then-remove must still delete it) from a key + // created within this pass (a true no-op). + var origMetaKeys map[string]struct{} + if len(txn.Metadata) > 0 { + origMetaKeys = make(map[string]struct{}, len(txn.Metadata)) + tctx.Metadata = make(map[string]any, len(txn.Metadata)) + for k, v := range txn.Metadata { + origMetaKeys[k] = struct{}{} + tctx.Metadata[k] = v + } + } var result *RuleActions for i := range r.rules { @@ -612,6 +649,65 @@ func (r *RuleResolver) ResolveWithContext(providerName string, txn TransactionCo ActionField: "series", ActionValue: seriesActionValue(a), }) + case "set_metadata": + if a.MetadataKey == "" { + continue + } + // Last-writer-wins per key. Metadata keys are case-sensitive + // (JSONB), so matching is exact. If a prior-stage remove queued + // this key, the later set cancels the remove. + if i := indexExact(result.MetadataRemove, a.MetadataKey); i >= 0 { + result.MetadataRemove = append(result.MetadataRemove[:i], result.MetadataRemove[i+1:]...) + result.Sources = dropMetadataSource(result.Sources, "metadata_remove", a.MetadataKey) + } + if result.MetadataSet == nil { + result.MetadataSet = make(map[string]any) + } + result.MetadataSet[a.MetadataKey] = a.MetadataValue + // Refresh the source so the audit trail credits the winning rule. + result.Sources = dropMetadataSource(result.Sources, "metadata", a.MetadataKey) + result.Sources = append(result.Sources, RuleActionSource{ + RuleID: rule.id, + RuleShortID: rule.shortID, + RuleName: rule.name, + ActionField: "metadata", + ActionValue: a.MetadataKey, + }) + // Mirror into the live context so later-stage rules' metadata + // conditions observe the write. + if tctx.Metadata == nil { + tctx.Metadata = make(map[string]any) + } + tctx.Metadata[a.MetadataKey] = a.MetadataValue + case "remove_metadata": + if a.MetadataKey == "" { + continue + } + // If a same-pass earlier-stage rule set this key, drop that + // queued set and its source. + if _, was := result.MetadataSet[a.MetadataKey]; was { + delete(result.MetadataSet, a.MetadataKey) + result.Sources = dropMetadataSource(result.Sources, "metadata", a.MetadataKey) + } + // Mirror into the live context so later-stage rules see it gone. + delete(tctx.Metadata, a.MetadataKey) + // Queue a DB delete only when the key exists in the original blob. + // Last-writer-wins: a remove of a pre-existing key wins over an + // earlier set; a key created and removed within this pass (or one + // that never existed) is a net no-op needing no write or source. + if _, existed := origMetaKeys[a.MetadataKey]; !existed { + continue + } + if indexExact(result.MetadataRemove, a.MetadataKey) < 0 { + result.MetadataRemove = append(result.MetadataRemove, a.MetadataKey) + result.Sources = append(result.Sources, RuleActionSource{ + RuleID: rule.id, + RuleShortID: rule.shortID, + RuleName: rule.name, + ActionField: "metadata_remove", + ActionValue: a.MetadataKey, + }) + } } } } @@ -661,6 +757,36 @@ func dropSeriesSource(src []RuleActionSource) []RuleActionSource { return kept } +// dropMetadataSource removes a (field, key) metadata source entry — used when a +// later-stage rule supersedes (set→set, last-writer-wins) or cancels +// (set↔remove net-diff) an earlier-stage rule's metadata intent. field is +// "metadata" (set) or "metadata_remove" (remove); value is the metadata key. +func dropMetadataSource(src []RuleActionSource, field, key string) []RuleActionSource { + if len(src) == 0 { + return src + } + kept := src[:0] + for _, s := range src { + if s.ActionField == field && s.ActionValue == key { + continue + } + kept = append(kept, s) + } + return kept +} + +// indexExact returns the index of target in slice using exact (case-sensitive) +// comparison, or -1. Metadata keys are JSONB keys and therefore case-sensitive, +// unlike tag slugs which match case-insensitively. +func indexExact(slice []string, target string) int { + for i, s := range slice { + if s == target { + return i + } + } + return -1 +} + // dropTagSource removes a specific (field, value) source entry — used when a // later-stage rule cancels an earlier-stage rule's add/remove_tag intent // (the cancelled intent produces no DB write, so its source doesn't belong @@ -805,10 +931,159 @@ func evaluateLeaf(c *compiledCondition, tctx TransactionContext) bool { case "in_series": return evaluateBool(c, tctx.InSeries) default: + if key, ok := metadataKeyFromField(c.field); ok { + return evaluateMetadata(c, key, tctx.Metadata) + } return false // unknown field } } +// metadataFieldPrefix marks a condition leaf that reads a key from the +// transaction's free-form metadata blob. Mirrors service.metadataFieldPrefix. +const metadataFieldPrefix = "metadata." + +// metadataKeyFromField extracts the metadata key from a "metadata." field. +func metadataKeyFromField(field string) (string, bool) { + if !strings.HasPrefix(field, metadataFieldPrefix) { + return "", false + } + return field[len(metadataFieldPrefix):], true +} + +// evaluateMetadata mirrors service.evalMetadata: presence operators test key +// presence; every other operator requires the key to be present. eq/neq are +// driven by the expected value's type, while contains/not_contains/matches/in +// operate on the stringified stored value. Keeping the two implementations in +// lockstep means a rule that passes write-time validation evaluates the same at +// sync time as it does in preview / retroactive apply. +func evaluateMetadata(c *compiledCondition, key string, meta map[string]any) bool { + raw, present := meta[key] + switch c.op { + case "exists": + return present + case "not_exists": + return !present + } + if !present { + return false + } + switch c.op { + case "eq": + return metadataEquals(raw, c.value) + case "neq": + return !metadataEquals(raw, c.value) + case "contains": + return strings.Contains(strings.ToLower(metadataString(raw)), strings.ToLower(metadataString(c.value))) + case "not_contains": + return !strings.Contains(strings.ToLower(metadataString(raw)), strings.ToLower(metadataString(c.value))) + case "matches": + if c.regex != nil { + return c.regex.MatchString(metadataString(raw)) + } + return false + case "in": + actual := metadataString(raw) + if list, ok := c.value.([]interface{}); ok { + for _, item := range list { + if strings.EqualFold(actual, metadataString(item)) { + return true + } + } + } + return false + case "gt", "gte", "lt", "lte": + actual, ok1 := metadataFloat(raw) + expected, ok2 := metadataFloat(c.value) + if !ok1 || !ok2 { + return false + } + switch c.op { + case "gt": + return actual > expected + case "gte": + return actual >= expected + case "lt": + return actual < expected + case "lte": + return actual <= expected + } + } + return false +} + +// metadataEquals compares a stored metadata value against the expected +// condition value, selecting the comparison from the expected value's type. +func metadataEquals(raw, expected any) bool { + switch exp := expected.(type) { + case bool: + b, ok := metadataBool(raw) + return ok && b == exp + case float64, float32, int, int64, json.Number: + ev, ok1 := metadataFloat(expected) + av, ok2 := metadataFloat(raw) + return ok1 && ok2 && av == ev + default: + return strings.EqualFold(metadataString(raw), metadataString(expected)) + } +} + +// metadataString renders a metadata value as a string for string operators. +func metadataString(v any) string { + switch val := v.(type) { + case nil: + return "" + case string: + return val + case bool: + if val { + return "true" + } + return "false" + case float64: + return strconv.FormatFloat(val, 'f', -1, 64) + case json.Number: + return val.String() + default: + if b, err := json.Marshal(val); err == nil { + return string(b) + } + return fmt.Sprintf("%v", val) + } +} + +// metadataFloat coerces a metadata value to float64 for numeric comparison. +func metadataFloat(v any) (float64, bool) { + switch val := v.(type) { + case float64: + return val, true + case float32: + return float64(val), true + case int: + return float64(val), true + case int64: + return float64(val), true + case json.Number: + f, err := val.Float64() + return f, err == nil + case string: + f, err := strconv.ParseFloat(val, 64) + return f, err == nil + } + return 0, false +} + +// metadataBool coerces a metadata value to bool for eq/neq against a bool expected. +func metadataBool(v any) (bool, bool) { + switch val := v.(type) { + case bool: + return val, true + case string: + b, err := strconv.ParseBool(val) + return b, err == nil + } + return false, false +} + // evaluateString applies string operators. func evaluateString(c *compiledCondition, fieldVal string) bool { switch c.op { @@ -980,4 +1255,3 @@ func stringInList(fieldVal string, v interface{}) bool { return strings.EqualFold(fieldVal, toString(v)) } } - diff --git a/internal/sync/rule_resolver_metadata_test.go b/internal/sync/rule_resolver_metadata_test.go new file mode 100644 index 00000000..264ac673 --- /dev/null +++ b/internal/sync/rule_resolver_metadata_test.go @@ -0,0 +1,247 @@ +//go:build !lite + +package sync + +import "testing" + +// --- Metadata conditions ----------------------------------------------------- + +func TestEvaluateMetadata_Conditions(t *testing.T) { + meta := map[string]any{ + "tax_deductible": true, + "trip": "japan-2026", + "amount_cents": float64(12050), // numbers arrive as float64 from JSON + "notes": "extended warranty included", + "flag_str": "true", + } + cases := []struct { + name string + field string + op string + value any + want bool + }{ + {"bool eq true", "metadata.tax_deductible", "eq", true, true}, + {"bool eq false (mismatch)", "metadata.tax_deductible", "eq", false, false}, + {"bool neq", "metadata.tax_deductible", "neq", false, true}, + {"string-bool eq true", "metadata.flag_str", "eq", true, true}, + {"string eq", "metadata.trip", "eq", "japan-2026", true}, + {"string eq case-insensitive", "metadata.trip", "eq", "JAPAN-2026", true}, + {"string neq", "metadata.trip", "neq", "paris", true}, + {"string contains", "metadata.notes", "contains", "warranty", true}, + {"string not_contains", "metadata.notes", "not_contains", "refund", true}, + {"string matches", "metadata.trip", "matches", "^japan-\\d+$", true}, + {"string in", "metadata.trip", "in", []any{"paris", "japan-2026"}, true}, + {"string in (miss)", "metadata.trip", "in", []any{"paris", "rome"}, false}, + {"numeric gt", "metadata.amount_cents", "gt", float64(10000), true}, + {"numeric gte exact", "metadata.amount_cents", "gte", float64(12050), true}, + {"numeric lt (miss)", "metadata.amount_cents", "lt", float64(100), false}, + {"numeric eq", "metadata.amount_cents", "eq", float64(12050), true}, + {"exists present", "metadata.trip", "exists", nil, true}, + {"not_exists present (miss)", "metadata.trip", "not_exists", nil, false}, + // Absent key: only not_exists matches; every comparison is false. + {"absent exists (miss)", "metadata.missing", "exists", nil, false}, + {"absent not_exists", "metadata.missing", "not_exists", nil, true}, + {"absent eq (miss)", "metadata.missing", "eq", "x", false}, + {"absent neq (miss — key must be present)", "metadata.missing", "neq", "x", false}, + {"absent not_contains (miss — key must be present)", "metadata.missing", "not_contains", "x", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + cc := mustCompile(t, &Condition{Field: tc.field, Op: tc.op, Value: tc.value}) + got := evaluateCondition(cc, TransactionContext{Metadata: meta}) + if got != tc.want { + t.Errorf("metadata cond %s %s %v: got %v, want %v", tc.field, tc.op, tc.value, got, tc.want) + } + }) + } +} + +func TestEvaluateMetadata_NilBlob(t *testing.T) { + cc := mustCompile(t, &Condition{Field: "metadata.foo", Op: "eq", Value: "bar"}) + if evaluateCondition(cc, TransactionContext{}) { + t.Error("eq against a nil metadata blob must not match") + } + ccNotExists := mustCompile(t, &Condition{Field: "metadata.foo", Op: "not_exists"}) + if !evaluateCondition(ccNotExists, TransactionContext{}) { + t.Error("not_exists against a nil metadata blob must match") + } +} + +// --- Metadata actions (resolve merge) --------------------------------------- + +func TestResolveWithContext_SetMetadata(t *testing.T) { + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{{ + id: testUUID(1), + shortID: "rmeta001", + name: "tag deductible", + actions: []typedAction{ + {Type: "set_metadata", MetadataKey: "tax_deductible", MetadataValue: true}, + {Type: "set_metadata", MetadataKey: "reviewed_by", MetadataValue: "rules"}, + }, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "irs"}), + }}, + } + result := r.ResolveWithContext("plaid", TransactionContext{Name: "IRS payment"}, true) + if result == nil { + t.Fatal("expected non-nil result") + } + if v, ok := result.MetadataSet["tax_deductible"]; !ok || v != true { + t.Errorf("expected tax_deductible=true, got %v (ok=%v)", v, ok) + } + if v, ok := result.MetadataSet["reviewed_by"]; !ok || v != "rules" { + t.Errorf("expected reviewed_by=rules, got %v (ok=%v)", v, ok) + } +} + +func TestResolveWithContext_SetMetadataLastWriterWins(t *testing.T) { + // Two rules set the same key; the later (higher-priority-stage) rule wins. + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{ + { + id: testUUID(1), shortID: "a", name: "A", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: "first"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + { + id: testUUID(2), shortID: "b", name: "B", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: "second"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + }, + } + result := r.ResolveWithContext("plaid", TransactionContext{Name: "x thing"}, true) + if result.MetadataSet["k"] != "second" { + t.Errorf("expected last-writer-wins value 'second', got %v", result.MetadataSet["k"]) + } + // Only the winning rule's source survives for this key. + metaSources := 0 + for _, s := range result.Sources { + if s.ActionField == "metadata" && s.ActionValue == "k" { + metaSources++ + } + } + if metaSources != 1 { + t.Errorf("expected 1 surviving metadata source for key k, got %d", metaSources) + } +} + +func TestResolveWithContext_MetadataNetDiff_SetThenRemoveCancels(t *testing.T) { + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{ + { + id: testUUID(1), shortID: "a", name: "A", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: "v"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + { + id: testUUID(2), shortID: "b", name: "B", + actions: []typedAction{{Type: "remove_metadata", MetadataKey: "k"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + }, + } + result := r.ResolveWithContext("plaid", TransactionContext{Name: "x thing"}, true) + if _, ok := result.MetadataSet["k"]; ok { + t.Errorf("set then remove of same key should cancel; MetadataSet still has k: %v", result.MetadataSet) + } + if indexExact(result.MetadataRemove, "k") >= 0 { + t.Errorf("cancelled set/remove should not queue a delete; MetadataRemove=%v", result.MetadataRemove) + } +} + +// A set followed by a remove of a key that ALREADY exists in the DB blob must +// still delete it — last-writer-wins. Cancelling only the queued set would +// leave the pre-existing value in place, silently dropping the remove. +func TestResolveWithContext_MetadataNetDiff_SetThenRemoveDeletesPreexisting(t *testing.T) { + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{ + { + id: testUUID(1), shortID: "a", name: "A", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: "v"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + { + id: testUUID(2), shortID: "b", name: "B", + actions: []typedAction{{Type: "remove_metadata", MetadataKey: "k"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + }, + } + // Seed an existing value: set-then-remove must delete it, not revert to it. + result := r.ResolveWithContext("plaid", TransactionContext{Name: "x thing", Metadata: map[string]any{"k": "old"}}, true) + if _, ok := result.MetadataSet["k"]; ok { + t.Errorf("set then remove should not persist a set; MetadataSet=%v", result.MetadataSet) + } + if indexExact(result.MetadataRemove, "k") < 0 { + t.Errorf("set then remove of a pre-existing key must queue a delete; MetadataRemove=%v", result.MetadataRemove) + } +} + +func TestResolveWithContext_MetadataNetDiff_RemoveThenSetWins(t *testing.T) { + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{ + { + id: testUUID(1), shortID: "a", name: "A", + actions: []typedAction{{Type: "remove_metadata", MetadataKey: "k"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + { + id: testUUID(2), shortID: "b", name: "B", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "k", MetadataValue: "v"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "x"}), + }, + }, + } + // Seed an existing value so the remove has something to target. + result := r.ResolveWithContext("plaid", TransactionContext{Name: "x thing", Metadata: map[string]any{"k": "old"}}, true) + if result.MetadataSet["k"] != "v" { + t.Errorf("remove-then-set should end as set; got MetadataSet=%v", result.MetadataSet) + } + if indexExact(result.MetadataRemove, "k") >= 0 { + t.Errorf("remove-then-set should not also queue a delete; MetadataRemove=%v", result.MetadataRemove) + } +} + +func TestResolveWithContext_MetadataChaining(t *testing.T) { + // Rule A writes metadata.bucket; rule B (later) conditions on it. + r := &RuleResolver{ + hitCounts: make(map[[16]byte]int), + rules: []compiledRule{ + { + id: testUUID(1), shortID: "a", name: "A", + actions: []typedAction{{Type: "set_metadata", MetadataKey: "bucket", MetadataValue: "subscriptions"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "provider_name", Op: "contains", Value: "spotify"}), + }, + { + id: testUUID(2), shortID: "b", name: "B", + actions: []typedAction{{Type: "add_tag", TagSlug: "recurring"}}, + trigger: "always", + condition: mustCompile(t, &Condition{Field: "metadata.bucket", Op: "eq", Value: "subscriptions"}), + }, + }, + } + result := r.ResolveWithContext("plaid", TransactionContext{Name: "Spotify"}, true) + if result == nil { + t.Fatal("expected non-nil result") + } + if len(result.TagsToAdd) != 1 || result.TagsToAdd[0] != "recurring" { + t.Errorf("later rule should observe rule A's metadata write and add 'recurring'; got %v", result.TagsToAdd) + } +}