Feat/issue 322 portfolio optimization

[Lansa-18]

Closes #322

feat: Portfolio Optimization & Optimal Allocation Suggestion Engine (#322)

Summary

Adds an engine that computes an optimal portfolio allocation from historical protocol APY data, per-protocol risk scores, and the user's risk tolerance — returned with an efficient frontier and a backtest of its simulated consequences, as a one-tap suggestion. The suggestion is advisory: it is persisted and audited, but it is structurally incapable of mutating User.strategyConfig.

Context reviewers need first: the platform already consumed allocations but never produced one. User.strategyConfig.targetAllocations existed and the rebalancing agent worked toward it, but every { "Blend": 50, "Stellar DEX": 30, "Luma": 20 } in the repo was a hand-written guess. There was no answer to "given my risk tolerance, what should my allocation be?"

Three findings shaped the design, and reviewers should know them before reading the diff:

• The agent has no multi-protocol position model. Position.protocolName is a single string, StrategyDecision.targetProtocol is a single string, and TargetAllocationStrategy (strategies.ts:162) uses weights only to rank a single hop — it never holds several protocols at once. A weight vector is therefore genuinely advisory input to the existing engine, not a new execution mode. The backtest is framed accordingly and honestly (see Response Contract).
• The plan's specified objective was mathematically degenerate. Under Σ = cov(apy/100/365.25) × 365.25 with λ ∈ [1,25], the risk term sits 5–6 orders of magnitude below the return term. (λ/2)wᵀΣw can never offset μᵀw at any λ in that range, so the optimum is always the max-return corner and the efficient frontier collapses to a single point. Implementing it as written would have shipped a feature that always answers "100% into the highest-APY protocol."
• ProtocolRiskScore was never being refreshed. src/jobs/protocolRiskScoring.ts existed but was never imported or started. Because risk-ceiling filtering is fail-closed, a stale table silently disables every ceiling-constrained rebalance, not just this feature.

This PR is greenfield in src/analytics/, but it necessarily corrects the objective's units and wires up a job that was dead on main.

---

Changes Made

New Module — src/analytics/optimizer.ts (new, 812 lines)

• Mean-variance solver: maximize μᵀw − (λ/2)·wᵀΣw subject to Σw = 1, lo_i ≤ w_i ≤ hi_i, and an optional stable-group floor Σ_{i∈S} w_i ≥ f
• Exact Euclidean projection onto the capped simplex — w_i(θ) = clamp(v_i − θ, lo_i, hi_i) is monotone in θ, so bisection converges to machine precision with no tolerance to tune
• Exact projection onto the floor constraint via a disjoint-group split, not the planned Dykstra alternating projections (see Design Decisions)
• Accelerated projected gradient (FISTA + adaptive restart) with a Gershgorin step bound
• Log-spaced λ mapping over [5, 500]; efficient-frontier sweep, default 12 points, hard max 25
• Up-front feasibility check that names the binding constraint before any solving
• toPercentageAllocations — the single fraction→percent boundary for the whole package
• Zero I/O, zero randomness, no new dependency

New Module — src/analytics/estimation.ts (new, 316 lines)

• Builds μ and Σ from gappy ProtocolRate history
• Daily aggregation to one value per (protocol, UTC day) — ProtocolRate is keyed by (protocolName, assetSymbol, network, fetchedAt), so without this the series would depend on scan ordering
• Index alignment via the existing buildDailyRateSeries, then keeping only days on which every admitted protocol has a value
• Sample covariance, n−1 convention, symmetric by construction (the (j,i) entry is assigned from the (i,j) computation, so float associativity cannot produce a matrix that fails a PSD assertion at the 1e-18 level)
• Fail-closed universe eligibility with machine-readable exclusion reasons
• Deliberately does not use periodReturns (see Design Decisions)

New Module — src/analytics/service.ts (new, 497 lines)

• suggestAllocation(userId): effective-ceiling resolution → estimation → optimization → backtest → persistence
• Mirrors the agent's two deliberately different ceiling merge rules verbatim
• Canonical sha256:-prefixed input-snapshot hash, numbers fixed to 9 dp
• Backtest harness running suggested vs. current config over identical history
• The only module in the package that touches the database

New Module — src/analytics/types.ts (new, 248 lines)

Shared types and the units contract for the package. Zero imports — a pure type module.

New Module — src/utils/concurrency.ts (new, 102 lines)

ConcurrencyLimiter — per-key + global in-flight semaphore, non-blocking, idempotent release, no per-key map growth. The repo's first. Bounds a different thing from the rate limiter (see Design Decisions).

New Job — src/jobs/allocationSuggestions.ts (new, 137 lines)

6-hour precompute over users with an ACTIVE Position. Standard observability quartet (logBackgroundJob + recordBackgroundJob + recordJobSuccess/recordJobFailure), correlation-ID scope, batched with serial await, per-user failure isolation, errors logged and never rethrown.

New Validators — src/validators/allocation-validators.ts (new, 100 lines)

Whole-request Zod schemas matching the file they mount in. .strict() on the body so a typo'd knob is a 400 rather than a confidently-wrong answer computed over the wrong window. Every numeric bound is also a CPU bound, expressed with the optimizer's own exported constants rather than duplicated magic numbers.

Migration — prisma/migrations/20260817000000_add_allocation_suggestions/ (new, 25 + 25 lines)

allocation_suggestions table, two indexes (userId, computedAt and userId, inputHash), FK to users with ON DELETE CASCADE. Hand-written rollback.sql alongside, per the CI gate, documenting that this is advisory data only — no funds, keys, or agent behaviour depend on it.

Handlers — src/routes/portfolio.ts (+133)

• POST /:userId/suggest-allocation and GET /:userId/suggestions, both requireAuth → enforceUserAccess → validate(...)
• Registered before GET /:userId, following the router.use('/goals', …) precedent
• Module-level ConcurrencyLimiter; 429 with Retry-After; slot released in finally so a throw cannot permanently wedge a user out of the endpoint

Shared — src/agent/strategyMetrics.ts (+11 / −4)

Exports mean and sampleStdev, previously module-private, so estimation reuses them. That file declares itself the one definition of risk-adjusted return math in the codebase; a private copy in src/analytics/ would have been the third.

Schema — prisma/schema.prisma (+47 real)

AllocationSuggestion model + allocationSuggestions relation on User. The raw diff shows ±215 lines; --ignore-all-space shows 47 added / 0 removed — the remainder is prisma format column reflow from running codegen, with no semantic change.

Config / Middleware / Startup / Formatters

• src/config/env.ts (+37) — security.optimizerRateLimit and allocationSuggestions blocks, all optional with defaults
• src/middleware/rateLimiter.ts (+21) — optimizerRateLimiter via the existing buildRateLimiter factory
• src/index.ts (+26) — registers the new job and scheduleProtocolRiskScoring, which existed on main but was never started; two module handles; two gracefulShutdown clear blocks
• src/utils/api-formatters.ts (+27) — mapAllocationSuggestionToResponse, hand-written allowlist, userId deliberately omitted, isSuggestion: true on every row

Tests — 7 new suites, +161 tests (2,097 lines)

optimizer 60 · service 28 · estimation 23 · integration 16 · structural 15 · concurrency 11 · job 8. One existing file touched: tests/integration/rateLimiter.integration.test.ts (+3) needed optimizerRateLimit in its config mock — every limiter is constructed at module load, so a missing block fails that suite at import, not at use.

Documentation

docs/PORTFOLIO_OPTIMIZATION.md (new, 393 lines), docs/openapi.yaml (+370: 2 paths, 8 schemas), ASSUMPTIONS.md (+52: entries 12–18), docs/DOCUMENTATION_INDEX.md (+1).

---

Invariants

	Invariant
I1	Nothing in src/analytics/ or src/jobs/allocationSuggestions.ts writes User.strategyConfig or User.rebalanceStrategy. allocationSuggestion.create is the only write in the entire package
I2	Nothing in the package imports from src/stellar/ or anything matching wallet. No suggestion path can touch custody
I3	Emitted weights sum to 100 within ±0.01 percentage points — precisely the tolerance publishableConfigSchema.superRefine already enforces, so a suggestion is directly acceptable by the existing update path with no re-conversion
I4	Every returned weight respects its [lo, hi] bound and any stable-group floor. Every solver iterate is a projection onto the feasible set, so even an abandoned run yields a constraint-satisfying vector
I5	A configured riskCeiling is fail-closed: a protocol with no known score is excluded, never given the benefit of the doubt. Mirrors applyRiskCeiling rather than reimplementing it
I6	A follow may only ever tighten a follower's risk ceiling (Math.max); an ACTIVE SavingsGoal overrides outright (??). Both mirror the agent exactly
I7	Identical inputs produce byte-identical output — protocols sorted by name with μ/Σ permuted to match, fixed feasible start, fixed budget, no randomness
I8	Equal inputHash implies equal weights, which is what makes "did my recommendation change, or only my inputs?" answerable

I1 and I2 are enforced by tests/unit/analytics/structural.test.ts, which scans source text and fails on any user.update/upsert/delete/create, any strategyConfig: write outside a select, any forbidden import, and any Prisma model access outside a fixed allowlist. It uses the stricter specifier-parsing form from strategy-follow.integration.test.ts — asserting imports.length > 0 — so a regex that silently stops matching cannot pass vacuously. Comments are stripped first, so the files can document exactly what is banned without tripping their own assertions.

---

Failure Modes and Protections

Failure Mode	Protection
Optimizer silently rewrites a user's live strategy	Structural source-scan test; allocationSuggestion.create is the only permitted write (I1)
A precompute job rewrites strategy configs on a 6-hour timer	Same structural test applied to the job file — otherwise this is an autonomous trading system wearing an analytics job's clothes
Solver returns a plausible vector that violates a constraint	4-variant discriminated outcome union; every iterate is a feasible-set projection, so non_converged still carries a constraint-satisfying vector
Contradictory constraints	Up-front checkFeasibility naming the binding constraint, run before any solving
Risk ceiling too tight to form a portfolio	insufficient_universe with per-protocol exclusion reasons and bindingConstraint: riskCeiling — never a ceiling-violating vector
Misaligned covariance matrix (silent, looks well-formed)	Only days on which every admitted protocol has a value are kept; periodReturns deliberately avoided
Smoothed YieldSnapshot.apy understating volatility	Inputs derive from ProtocolRate.supplyApy only — the trap documented in STRATEGY_MARKETPLACE.md §2
Cross-user access to suggestions	Routes keyed on req.params.userId, which is what keeps enforceUserAccess effective; a body-only or /suggestions/:id route would make it a silent no-op
CPU exhaustion by one user	ConcurrencyLimiter per-key bound of 1, non-blocking 429
CPU exhaustion in aggregate	Global in-flight budget + optimizerRateLimiter (5/min) — the two bound different things
Concurrency slot leaked by a thrown handler	Released in finally; asserted by a test that a 404 is followed by a successful 200
Unbounded query param translating into event-loop time	.strict() Zod body with bounds drawn from the optimizer's own constants; frontier hard-capped at 25
Weights drift outside the ±0.01 sum tolerance	Rounding residual settled onto the single largest weight, ties broken by name; asserted over 300 seeded inputs
Stale ProtocolRiskScore silently disabling all ceiling-constrained rebalancing	scheduleProtocolRiskScoring wired up and started before the suggestion job

---

Response Contract (clients need these)

weights are PERCENTAGES, not fractions, and sum to 100 ± 0.01 — directly acceptable by the existing strategy update endpoint with no conversion. Everything else in the payload (expectedReturn, expectedVolatility, frontier risk/return) is a decimal fraction: 0.082 means 8.2 % APY, 0.014 means 1.4 percentage points of APY volatility.

All four outcome statuses return HTTP 200. "The optimizer ran and could not produce a portfolio" is a result, not a request error:

status	Meaning
ok	Weights, expected return/volatility, λ, frontier, iterations, portfolio risk score
infeasible	Constraints contradict; bindingConstraint names which
insufficient_universe	Fewer than 2 eligible protocols, with per-protocol exclusion reasons
non_converged	Budget exhausted; carries the best feasible vector plus the residual

Two caveats ship inside the payload, not just in the docs, because the chart would otherwise be misread:

• backtest.caveat — the agent holds one protocol at a time, so the comparison is what the agent would have done under each configuration. It is not a simulation of holding the weighted basket. backtest.current is null when the user has no allocation configured; an invented baseline would be worse than none.
• disclaimer — Σ measures APY co-movement, not capital risk. ProtocolRate is a yield-quote series, so the optimizer minimizes yield volatility; it does not model principal loss, depeg, or smart-contract failure. Those enter only through ProtocolRiskScore as a universe filter.

ceilingSource (goal | follow | own | none) reports which layer supplied the effective ceiling, so the resolution is visible rather than inferred.

---

Breaking Changes

None. Both endpoints are new, no existing response shape changed, no existing column changed, and no dependency was added or removed.

The AllocationSuggestion table is additive with a nullable-safe ON DELETE CASCADE FK. The only non-additive change anywhere is src/agent/strategyMetrics.ts widening two functions from module-private to exported, which cannot break a caller.

---

How Has This Been Tested?

npx prisma generate                       # required after the schema edit

npm run lint                              # CI-blocking
npm run format:check                      # CI-blocking
npm run build                             # CI-blocking
npm test                                  # CI-blocking
npm run typecheck                         # NOT run in CI — must be run locally

npx jest tests/unit/analytics             # solver + estimation + service + structural
npx jest tests/integration/allocation-suggestions.integration.test.ts

npm run validate:spec                     # redocly gate on docs/openapi.yaml
npx redocly bundle docs/openapi.yaml --output /tmp/o.yaml   # verify all $refs resolve
bash scripts/check-migration-rollback.sh  # rollback.sql presence gate
npm audit --audit-level=high              # CI security-audit job
npx license-checker --onlyAllow '...' --excludePrivatePackages

Check	Result
npm test	817 passed / 66 suites (was 770 / 62)
New suites	161 passed, 0 failed
npm run lint	clean
npm run format:check	clean
npm run build	clean
npm run typecheck	clean
scripts/check-migration-rollback.sh	pass (this gate was failing on main before this branch)
redocly lint	0 errors, 16 warnings — all pre-existing categories
redocly bundle	all $refs resolve
npm audit --audit-level=high	exit 0
license-checker	exit 0
Jest "worker failed to exit gracefully" warning	pre-existing — reproduced on a 55-suite run with every new test excluded

Acceptance criteria, mapped to tests:

• weights sum to 1 and respect every bound / sums to 100 within the ±0.01 tolerance — property-style over a seeded mulberry32 generator, 10 seeds × 20 problems, plus 300 randomized percentage conversions
• explicit min/max bounds are always respected and the stablecoin floor is never violated — 10 seeds × 15 problems each, floor asserted to 1e-9
• higher riskTolerance gives weakly higher expected return and volatility — monotonicity across the full 1–10 range
• is non-decreasing in risk and in return / every frontier point is itself a valid portfolio — a point that took on risk for less return would mean the sweep is not tracing an efficient frontier
• produces byte-identical output for the same input twice and is invariant to the caller ordering the protocols differently — determinism (I7)
• a follow can never LOOSEN a tighter own ceiling / an active goal OVERRIDES outright, even to loosen — the two merge rules (I6)
• NEVER writes to the user record + the whole structural.test.ts suite — the advisory invariant (I1, I2)
• 429s the second concurrent optimization for the same user — holds the first request inside the service by stalling a DB read, fires the second, asserts 429 + Retry-After, then releases and asserts the first still completes 200
• releases the concurrency slot after a failure — a leaked slot would wedge a user out permanently with nothing logged
• only counts days on which EVERY protocol has a value / forward-fills gaps so a gappy protocol stays index-aligned — the silent-corruption path
• one user failure does not abort the batch — asserts the third user is still attempted after the second throws

Verified by hand beyond the assertions: the optimizer was re-run end-to-end on realistic 90-day data to confirm behaviour, not just passing assertions. As riskTolerance rises 1 → 10, expected return climbs 7.42 % → 10.31 %, volatility climbs 0.697 pp → 1.573 pp, and the weighted risk score falls 70.6 → 65.2 — the conservative profile diversifies into the safest protocol (score 88), the aggressive one concentrates in the high-yield pair. Weights summed to exactly 100 at every level and output was byte-identical on repeat.

Measured cost (full solve + 12-point frontier sweep): 0.11 ms at 3 protocols, 1.33 ms at 10, 7.10 ms at 25. Across 450 seeded problems: 0 non-converged, median 104 iterations, max 802 of a 2000 budget.

• Unit tests added / updated
• Integration tests added / updated
• Manual testing completed (pure-computation paths only — see below)

⚠️ Not verified, and a reviewer with a database should close this gap. migration-smoke and api-contract's contract smoke tests both need a live Postgres and could not run locally, so the new migration has never been executed against a real database. Please: (1) npx prisma migrate dev, confirm allocation_suggestions is created, then apply rollback.sql on a scratch DB and confirm it drops cleanly; (2) seed ProtocolRate history + ProtocolRiskScore rows for ≥ 3 protocols and exercise both endpoints against a real server — npx ts-node src/index.ts, not npm run dev; (3) confirm SELECT strategy_config FROM users is unchanged throughout, verifying I1 against a real database rather than a mock.

---

Pre-existing Bugs Found — NOT Fixed Here

1. npm run dev and npm start both point at a stale demo file. src/index.ts is the real application, but "dev": "nodemon --exec ts-node src/app.ts" runs src/app.ts — a standalone CORS demo with its own express app, its own listen, and hard-coded /api/data handlers that nothing imports. Worse, "start": "node dist/src/app.js" targets a path the build never produces: tsconfig.build.json sets rootDir: ./src, so the output is dist/index.js. Verified both: dist/src/app.js does not exist after npm run build. The Dockerfile is correct and authoritative. Fixing the scripts is a one-line change but touches how everyone runs the project locally, so it deserves its own PR.

2. src/routes/backtest.ts is dead code masquerading as a route file. It is a near-identical stale copy of src/agent/backtest.ts, differing only in import paths and a missing BigInt fix for the 1e21 exponential-notation bug. Verified nothing imports it and nothing mounts it. Deleting it is safe but out of scope here.

3. enforceUserAccess returns 401, not 403, for a cross-user target. authenticate.ts:145 answers AUTH_ERRORS.UNAUTHORIZED when req.auth.userId !== targetUserId — but the caller is authenticated; they are simply not authorised, which is a 403. This PR's tests assert 401, the behaviour that actually exists, rather than the behaviour one might assume. Changing it is a cross-cutting API contract change affecting every owner-scoped route.

4. enforceUserAccess silently no-ops when neither req.params.userId nor req.body.userId is present. Already documented in CLAUDE.md as a known trap. Not a new bug, but it is the direct reason both new routes are keyed on req.params.userId — noted so the keying is not "simplified" later.

5. docs/openapi.yaml gains one new no-ambiguous-paths warning, between /portfolio/{userId}/suggest-allocation and /portfolio/goals/{id} — the same spec-level ambiguity the pre-existing /transactions/* pair already has. It resolves correctly at runtime because router.use('/goals', …) is mounted before /:userId. Fixing it properly means restructuring the /goals mount, which is a separate change.

---

Design Decisions

Σ is the covariance of annual rate levels, and λ is log-spaced over [5, 500]. This deviates from the issue plan and is one decision, not two. Under the plan's Σ = cov(apy/100/365.25) × 365.25 with λ ∈ [1,25], the risk term is 5–6 orders of magnitude below the return term:

λ	(λ/2)·wᵀΣw at w=1	μ	ratio
1	5.48e-7	0.08	1.5e+5
25	1.37e-5	0.08	5.8e+3

The consequence is not "slightly aggressive" — the optimum is always the max-return corner and the frontier collapses to a point. The chosen Σ is exactly the plan's matrix × 365.25, which also makes expectedVolatility read in the same units as the APY, so "8.2 % expected return, 1.4 % expected volatility" is directly legible. Log spacing is the natural parameterization for a risk-aversion coefficient — each step is a constant ratio of risk appetite.

A default 60 % per-protocol concentration cap, applied as an explicit constraint. Even after rescaling, unconstrained mean-variance is corner-seeking (Michaud's "error maximization" — it treats a historical mean as if known exactly). Measured on this repo's scale, every riskTolerance from 3 to 10 returned a single protocol at 100 %. That is a true optimum of the stated objective and simultaneously terrible advice for an endpoint whose purpose is to suggest a diversified allocation. Applied as an overridable bounds.max default, raised to 1/n on small universes so the guardrail can never empty the feasible set — rather than quietly distorting μ or λ until the answer looked reasonable.

Exact split projection instead of Dykstra. The plan called for Dykstra's alternating projections for the stable-group floor. For a convex set ∩ halfspace, the projection is either the plain capped-simplex projection or it lies on Σ_S w = f, which is two capped simplices over disjoint index groups — and Euclidean distance separates across disjoint coordinates. So the split is exact and non-iterative. This is not merely tidier: the Dykstra draft's final re-projection pushed the iterate back out of the floor halfspace, returning 0.30 for a 0.60 floor.

Nesterov acceleration is load-bearing, not an optimization. Σ is a sample covariance over ~90 observations, so with many protocols — or protocols whose APYs move together, which is the norm in DeFi — it is ill-conditioned or rank-deficient. The objective is then concave but not strongly concave, and plain projected gradient converges at O(1/k). A 7-protocol problem still had a 5.2e-8 residual after the full 2000-iteration budget and reported non_converged while sitting on a perfectly good allocation; reaching 1e-10 that way needs ~10⁶ iterations.

buildDailyRateSeries, deliberately not periodReturns. A covariance matrix requires index-aligned vectors — entry (i,j) must pair protocols i and j on the same day. periodReturns skips any interval whose starting value is non-positive, which is correct for its own job (a portfolio funded from empty is a deposit, not a return) and fatal here: two protocols skipping different intervals produce vectors of different lengths whose k-th entries are different days. The resulting matrix looks perfectly well-formed and is silently, badly wrong.

weights stored as Json percentages, not Decimal. A deliberate deviation from the issue's "use Decimal for storage". The vector's entire purpose is to round-trip into User.strategyConfig.targetAllocations, itself a Json map of numbers validated by publishableConfigSchema; Decimal would force a lossy re-conversion at the one place the value is ever consumed. PublishedStrategyMetric already sets the Float-for-computed-analytics precedent.

The rate limiter is applied per-endpoint, not through the apiRoutes table. The table convention is right for a resource whose routes are uniformly costly, but /portfolio is mostly cheap reads that must not inherit a 5/min budget because one POST on the same router is expensive. This is not the double-application the warning in src/routes/admin.ts guards against — no table-level limiter applies to this route.

A concurrency semaphore in addition to the rate limiter, because they bound different things. A rate limiter bounds requests per window; it says nothing about how many run at once. The optimizer is the first genuinely CPU-bound thing in this API and runs on the single event-loop thread, so ten concurrent solves do not merely run slower — they block every other request in the process, including /health/ready. Non-blocking by design: queueing would convert a CPU problem into a latency-and-memory problem and hold sockets open behind work the client has probably abandoned.

The scheduled job stores suggestions without backtest legs. They are two full historical replays per user and are only interesting when a human is looking — which is exactly when the POST endpoint computes them live. Job rows carry a null backtest, documented so an absent comparison never reads as a failure.

The two ceiling merge rules were mirrored, not unified. A follow merges via Math.max (may only tighten); an ACTIVE goal overrides via ?? (may loosen). Unifying them into one rule would be tidier and wrong — a suggestion computed under a third rule would be advice about a portfolio the agent will never build.

---

Checklist

• Code follows the project's style guidelines (npm run lint, npm run format:check)
• Self-review performed
• Comments added for complex logic (the units contract, the projection geometry, and why acceleration is required are each documented at the site that depends on them)
• Documentation updated (docs/PORTFOLIO_OPTIMIZATION.md new; docs/openapi.yaml + ASSUMPTIONS.md + docs/DOCUMENTATION_INDEX.md updated in the same change, per the repo's route-change rule)
• No new warnings (npm run lint clean; redocly gains 1 pre-existing-category warning, noted above)
• Tests prove the fix/feature works (+161 tests; the acceptance criterion is a structural test, not a comment)
• All existing tests pass locally (817 / 66 suites; the jest worker-teardown warning is pre-existing and was reproduced without any new test loaded)
• Branch merged with latest main — clean descendant of 96a47e8; one add/add conflict on add_sub_accounts/rollback.sql resolved by keeping main's SQL byte-identical (verified with comments stripped) while preserving an operator-safety header
• No breaking changes
• Migration executed against a real database (cannot be done locally — migration-smoke needs a live Postgres; see the warning under How Has This Been Tested?)