feat(notebook): add a notebook plugin

[Fan Shangxiang (MartinForReal)]

Summary

Adds the TauGrid notebook integration as a JupyterLab 4 prebuilt extension plus a
Jupyter Server extension shipped inside the tau Python wheel. The researcher stays in
JupyterLab: submit the notebook they are already working in, then watch status, logs and a loss
curve until the run finishes.

The browser never talks to Kubernetes. The Jupyter Server extension drives the Python SDK on the
Jupyter Server identity, so the notebook needs no cluster credentials and no tau binary.

The workflow

1. Run TauGrid: Open runs (palette or launcher) - the run browser docks in the left sidebar.
2. Click Submit to TauGrid on the notebook toolbar.
3. The review tab resolves the destination from the cluster and shows exactly what will be created:
   run name, namespace, queue, profile, payload sizes, excluded cells, submission mode, retention.
4. Submit notebook -> native confirmation dialog -> the run is created.
5. Open the run to watch lifecycle evidence, logs, and the loss curve until it completes.

Surface placement

Each capability sits on the JupyterLab surface it belongs to, rather than one mega-panel:

Surface	Capability
Left sidebar (taugrid-runs)	Run browser: a table that loads itself and follows the namespace + queue, a filtered namespace picker, exact lookup, portal link
Main-area tab per run (taugrid-detail)	Lifecycle phases, admission/pods/execution tiles, diagnostics, pods, identifiers, loss curve
Logs tab (taugrid-logs)	One pod/container, bounded snapshot
Notebook toolbar + palette	Submit the current notebook: review, then confirm
About dialog	Whether submission is enabled here, and why

The panel uses the TauGrid Portal design system exactly - the same tokens, type scale, cards,
stat tiles, badges, tables, buttons, focus ring and chart treatment.

What is in it

Server (Jupyter Server extension)

• _payload.py - CLI-parity embedded payload: envelope v2, immutable encoding, the 64 KiB
  environment-entry cap and 1 MiB decoded ceiling, digest verified by the tau-payload
  initContainer.
• _notebook_pkg.py - validates the saved notebook, strips outputs and execution counts before
  measuring, drops approved launcher cells, and embeds the notebook with the runner and a digest
  context manifest.
• _render.py - renderer parity with cli/internal/rayjobrender: Kueue queue label,
  tau.azure.com/managed-by, K8sJobMode for plugin submissions so the driver output is a
  readable pod log, and a longer retention so a finished run can still be read.
• submit.py / _notebook_submit.py - plan-first submission behind
  TAUGRID_SUBMISSION_ENABLED with an explicit confirm, bound to the reviewed manifest digest.
• runs.py, metrics.py - run discovery for Jobs and RayJobs with lifecycle phases and
  owner-UID-scoped evidence, and bounded loss metrics with explicit truncation reasons.

Frontend

• A native React console: no iframe, no in-notebook widget. Everything is theme-styled to the
  Portal and keyboard operable.
• The loss curve plots the run's own step=N loss=V observations; a bounded window, not the
  complete history, and it says so. One observation stays a point; nothing is smoothed or invented.

Demos and packaging

• examples/notebook-ray-cpu-demo.ipynb (Ray CPU tasks) and
  examples/notebook-loss-curve-demo.ipynb (a deterministic 40-step loss curve).
• images/notebook-runtime/Dockerfile - the runtime image the notebook executor needs, built
  offline from vendored wheels; selected with TAUGRID_RUNTIME_IMAGE.
• The wheel ships the prebuilt labextension to share/jupyter/labextensions and enables the server
  extension through etc/jupyter/jupyter_server_config.d, so a clean install registers it.

Whole-PR audit and re-implementation

The PR was audited end to end and re-implemented rather than patched. Fourteen findings with
file:line evidence, the decisions, and a 50-path inventory are in
docs/design/notebook-plugin.md. The corrections that matter:

• The confirmation is bound to the reviewed manifest digest, so a plan that changes between
  review and confirm is rejected instead of silently submitted.
• The native and legacy submission paths were unified behind one implementation, keeping their
  separate policies, so they cannot drift apart again.
• Kubernetes reads are bounded with client cleanup, and pod/log ownership is verified by UID so a
  replacement pod cannot be read under a stale identity.
• The metrics cache and fallback are isolated per source; profile-declared resources are
  preserved; the launcher-cell detection that could discard user code was replaced.

Removed as dead weight, with rationale recorded: the branch's uncalled Go TensorBoard renderer and
Portal proxy route (unbounded reads, non-empty-string-only validation, superseded by the native
extension), the _conformance scaffold, a stale executed example notebook, committed notebook
outputs, and the superseded ipywidgets-era screenshots.

Live verification

Driven against a local JupyterLab and a docker-desktop cluster with Kueue + KubeRay:

• tools/run-cpu-ray-demo.py submitted examples/notebook-loss-curve-demo.ipynb as
  cpu-loss-demo-2 on the CPU profile azure.research.cpu.small (queue jobqueue,
  K8sJobMode, gpusPerWorker=[0]). The submitter ran the notebook, streamed real observations
  (step=0 loss=9 ... step=6 loss=0.086), GET /status?includeMetrics=true returned a bounded
  window, and the driver reported RESULT SUCCEEDED WITH LOSS.
• tools/run-labextension-e2e.mjs drives Edge through five journeys - open the runs sidebar, open a
  run detail, render the loss curve, open logs, submit a notebook through review and confirmation.
  PASS, with loss curve points: 40, polyline: 1.
• This verification is what caught the empty submitterPodTemplate: the K8sJobMode submitter had no
  /script payload and no /data mount, the entrypoint died on cd /data, and the RayJob still
  reported success. Now wired and locked by tests.

Checks

Check	Result
python scripts/check-license-headers.py	863 files pass
Notebook pytest suites (incl. contract and packaging)	215 passed
npm run build:lib / build:labextension	exit 0
npm test	33 passed
ruff check tau tests setup.py	clean
Go builds, core / cli / portal	pass
Wheel build + ZIP inspection	current sources, both asset locations, server config
Browser e2e (5 journeys)	PASS

The broader SDK (54) and Portal (16) suites are not green, but both failure sets are identical on
untouched main - pre-existing Windows/environment failures, deliberately not fixed here.

Deferred

• Choosing which files ship with a submitted job. The plugin currently embeds exactly three files
  (the prepared notebook, the runner, and the context manifest); there is no picker yet. The CLI
  offers --extra-script SRC:DEST, a project archive handed to Ray as working_dir, and
  image/PVC sources for anything large. Bringing that choice into the review is the obvious next
  step, bounded by the payload caps.
• Artifact download, cancel/resume, durable run history, streaming or multiplexed logs,
  authenticated portal series, and multi-rank aggregation.
• Publishing the notebook runtime image to MCR. The demo builds it locally; submission stays gated
  behind TAUGRID_SUBMISSION_ENABLED until a certified image is recorded.
• Notebook 7 runtime compatibility and light/dark visual review beyond the light Portal palette.

Try it

pip install "tau[widgets]"
docker build -t taugrid-notebook-runtime:local images/notebook-runtime

TAUGRID_SUBMISSION_ENABLED=1 \
TAUGRID_RUNTIME_IMAGE=taugrid-notebook-runtime:local \
TAUGRID_PORTAL_URL=https://your-portal \
  jupyter lab

Design source, API surface, metrics contract and the audit record:
docs/design/notebook-plugin.md. Task-oriented docs:
site/content/en/docs/examples/notebook-plugin.md.