fix(multi_format_api): explode nested JSON record arrays into rows

[gabrielbressan-tfy]

Problem

A generic API response shaped as an object with a nested record array collapsed into a single row, with the entire array dumped into one string cell.

Real failure (massive.com snapshot): { "tickers": [ …13k objects… ], "status": "OK", "count": 12962 } → 1 row, tickers = a ~9.8MB string. That blob propagated to a downstream code-execution step and blew the context budget: prompt is too long: 4,656,292 tokens > 1,000,000.

The crash is caused by shape, not raw size. The same data as a proper table:

Input	Shape	Size	Result
Original	1 row, giant tickers cell	9.88 MB	❌ 4,656,292 tokens
Flattened	12,962 rows × 43 cols	4.68 MB	✅ success, 8,548 input tokens

Root cause

format_parsers.py::parse_json only picked the record array from a hardcoded whitelist (data/results/items/records/rows/entries). Domain keys like tickers missed it and fell through to a single-row json_normalize.

Changes

• Generic record-array detection (_resolve_records): explicit record_path → top-level list → whitelist → bounded auto-detect of any list-of-objects (≤3 levels deep). One candidate wins; multiple → longest wins with a warning advising record_path.
• Scalar-sibling reattachment: top-level scalars (status, count, …) become constant columns; meta_ prefix on name collisions.
• New config args: record_path (STR) and auto_explode (STR 'true'/'false', default 'true') — settable at connection level and overridable per query in the WHERE clause.
• Back-compat preserved: empty whitelist arrays ({"data": []}) → empty DataFrame; auto_explode='false' keeps the legacy single-row shape.
• Docs: README + USAGE_EXAMPLES document the new behavior and options.

Tests

Net-new tests/test_format_parsers.py — 13 cases covering the bug, whitelist/list/single-object regressions, list-of-scalars, multi-array warning, record_path (nested + miss), auto_explode=false, dotted columns, meta_ collisions, empty array, and primitives. All 13 pass in the dev-mindsdb container.

Notes / risk

• Output shape only changes for the previously-broken single-row-blob case. Consumers wanting the old shape opt out via auto_explode='false' or record_path.
• One deliberate deviation from the original plan: empty lists under whitelist keys are still accepted (→ empty DataFrame) to preserve prior behavior; covered by a test.
• Not run here: the full cross-repo integration (flattened CSV → mktplace code-execution container → success=True), which lives on the mktplace side.

🤖 Generated with Claude Code