feat: add %%sql --limit and --no-display support

[djanand]

Which issue does this PR close?

Closes #.

Rationale for this change

The %%sql Jupyter cell magic advertises two options in its help text that were never actually implemented:
--no-display - Don't display results
--limit N - Limit displayed rows (default: 50)
The argument-parsing loop only extracted a variable name, it scanned all tokens and kept the last non-flag, never breaking, silently dropping any --flag. So the documented options were ignored. This PR makes the behavior match
the documented contract, including the (default: 50) already promised in the help text.

What changes are included in this PR?

In python/python/ballista/jupyter.py:

• Added _parse_cell_magic_args(), which parses --limit N and --no-display plus the optional variable name, using the same hand-rolled line.split() convention as the other magics in this module (%ballista, %register). Invalid --limit values (missing / non-integer / non-positive) return a clear message instead of raising.
• --limit N caps the display to N rows via datafusion's configure_formatter(max_rows=N, min_rows=N). This is display-only — it never truncates the underlying data, so an in-query LIMIT always takes effect. When --limit is omitted, the display falls back to a default of 50.
• --no-display runs the query, stores the result in the variable if one is given, and renders nothing.
• Updated the sql() docstring and the %ballista help text.

Docs: added a short --limit / --no-display example to the "SQL Magic Commands" section of python/README.md.

Tests in python/python/tests/test_jupyter.py:

• A cluster-free unit test for the argument parser (valid cases + invalid --limit rejection).
• An integration test covering --limit (display cap set, data not truncated), the default cap, --no-display, and the invalid---limit error path.

Are there any user-facing changes?

Yes, the previously documented --limit and --no-display options on the %%sql cell magic now work. The README is updated accordingly.
The only behavioral change to existing usage: a %%sql cell now renders up to 50 rows by default (the value the help text already documented here) instead of datafusion's built-in default of 10. This affects rendered rows only. Query results are never truncated, and in-query LIMIT is unaffected.

Note: --limit is implemented via datafusion's global HTML formatter, so each %%sql cell re-sets the display cap. The cap also applies to subsequent non-%%sql
DataFrame displays in the same kernel until changed

[djanand]

@coderfender @martin-g - ptal

[djanand]

Design note: why not use @magic_arguments?

IPython's @magic_arguments / parse_argstring is the conventional way to parse magic options (and would give --limit=5 and type validation for free). I deliberately avoided it here:

• This module is written to import and be unit-tested without IPython — it provides stub decorators for the IPYTHON_AVAILABLE == False path, ipython is only an optional [jupyter] extra (not a base or dev dependency), and the test suite imports BallistaMagics with no IPython installed. magic_arguments/parse_argstring are not stubbed and would be imported unconditionally, breaking that IPython-less path.
• The hand-rolled line.split() parsing matches the existing convention used by the other magics in this file (%ballista, %register); there is no argparse/magic_arguments usage anywhere in the module today.

If the project would prefer the @magic_arguments style, that'd be better done as a separate refactor converting all the magics at once IMO.

[djanand]

Why python release build fails ?

python/ is a separate Cargo workspace (excluded from the main one) with its own Cargo.lock, and it path-depends on the main ballista crates. But:

• Dependabot only manages directory: "/" (the main workspace) there's no /python entry. So every dep bump updates /Cargo.lock and never python/Cargo.lock.
• CI does not catches the drift early, it only surfaces later as the cryptic --locked maturin error.

So the lock goes stale on main itself every time deps move, and the next contributor inherits a red build. Kindly let me know your thoughts

edit: Raised #1887

[djanand]

requires #1899 to be reviewed & merged since python transitive deps diverge. #1899 helps bridge the deps gap

[djanand]

• --limit 3 shows only 3 rows , dataframe has 5.

---

• --no-display doesn't print results. .count() on the next cell prints the df.

---

• %%sql with no flags works as-is

[coderfender]

Could we probably push this as a limit option to datafusion ? That might help with probing high volume data

[djanand]

The --limit is a display cap atm. It only changes how many rows render. It does not reduce compute or collection i.e the full result is still materialized. On datafusion side it seems we already have df.limit(n, offset), df.head(n) which pushes the limit into the plan. I think you mean that setting %%sql --limit x should push it to the execution layer rather than just display side. right ? I can look into that side lmk

[coderfender]

Yeah. I personally think that without that wiring in place , this feature might still OOM and users might not have an idea re the root cause

[coderfender]

Could be a good practice to store constant ipython flags separately?

[djanand]

sure makes sense. addressed