Forward-merge release/1.3 into develop (#919)

## Description

This is a manual forward merge of #895 due to merge conflicts

Closes #895 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing
Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This
certifies that the contribution is your original work, or you have
rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will
not be accepted.
- When the PR is ready for review, new or existing tests cover these
changes.
- When the PR is ready for review, the documentation is up to date with
these changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- New Features
- MCP: session-aware client, configurable token storage (incl. object
stores), new client config; deprecated legacy mcp_tool.
- Observability: workflow run/trace IDs, traceparent/header ingestion,
enriched span attribution.
- Agents/CLI: usage metrics in responses; improved workflow scaffolding
with prerelease-aware installs and updated templates.
  - Docker: build args for Python/UV/NAT across images.

- Documentation
- Overhauled notebooks (new Getting Started/BYO Agent/Tools &
Agents/Observability; removed older notebooks).
  - Expanded CLI reference (MCP, Object Store, Optimize, GPU sizing).
  - Secure token storage guide; install extras; contributor ulimit note.

- Refactor
- Chat models: typed roles, split choices (streaming vs non-streaming),
updated defaults.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: dagardner-nv

ci/scripts/documentation_checks.sh

@@ -17,8 +17,37 @@
 set +e
 
 # Intentionally excluding CHANGELOG.md as it immutable
-DOC_FILES=$(git ls-files "*.md" "*.rst" | grep -v -E '^(CHANGELOG|LICENSE)\.md$' | grep -v -E '^nv_internal/')
+DOC_FILES=$(git ls-files "*.md" "*.rst" | grep -v -E '^(CHANGELOG|LICENSE)\.md$')
+NOTEBOOK_FILES=$(git ls-files "*.ipynb")
 
-vale ${DOC_FILES}
+if [[ -v ${WORKSPACE_TMP} ]]; then
+    MKTEMP_ARGS=""
+else
+    MKTEMP_ARGS="--tmpdir=${WORKSPACE_TMP}"
+fi
+
+EXPORT_DIR=$(mktemp -d ${MKTEMP_ARGS} nat_converted_notebooks.XXXXXX)
+if [[ ! -d "${EXPORT_DIR}" ]]; then
+    echo "ERROR: Failed to create temporary directory" >&2
+    exit 1
+fi
+
+jupyter nbconvert -y --log-level=WARN --to markdown --output-dir ${EXPORT_DIR} ${NOTEBOOK_FILES}
+if [[ $? -ne 0 ]]; then
+    echo "ERROR: Failed to convert notebooks" >&2
+    rm -rf "${EXPORT_DIR}"
+    exit 1
+fi
+
+CONVERTED_NOTEBOOK_FILES=$(find ${EXPORT_DIR} -type f  -name "*.md")
+
+vale ${DOC_FILES} ${CONVERTED_NOTEBOOK_FILES}
 RETVAL=$?
+
+if [[ "${PRESERVE_TMP}" == "1" ]]; then
+    echo "Preserving temporary directory: ${EXPORT_DIR}"
+else
+    rm -rf "${EXPORT_DIR}"
+fi
+
 exit $RETVAL

ci/scripts/path_checks.py

@@ -162,8 +162,8 @@
     ),
     # ignore notebook-relative paths
     (
-        r"^examples/notebooks/retail_sales_agent/.*configs/",
-        r"^\./retail_sales_agent/data/",
+        r"^examples/notebooks/",
+        r".*(configs|data|src).*",
     ),
     (
         r"^examples/frameworks/haystack_deep_research_agent/README.md",

ci/vale/styles/config/vocabularies/nat/accept.txt

@@ -25,13 +25,14 @@ Authlib
 [Cc]hatbot(s?)
 # clangd is never capitalized even at the start of a sentence https://clangd.llvm.org/
 clangd
+Colab
 CMake
 [Cc]omposability
 [Cc]omposable
 Conda
 concurrencies
 config
-Configurability
+[Cc]onfigurability
 [Cc]oroutine(s?)
 CPython
 [Cc]ryptocurrenc[y|ies]
@@ -56,6 +57,7 @@ Dynatrace
 [Ee]val
 [Ee]xplainability
 Faiss
+Gantt
 [Gg]eneratable
 GitHub
 glog
@@ -65,7 +67,9 @@ groundedness
 [Gg]ranularities
 [Hh]ashable
 [Hh]yperparameter(s?)
+impactful
 [Ii]nferencing
+[Ii]nterquartile
 isort
 Jira
 jsonlines
@@ -114,6 +118,7 @@ Pydantic
 PyPI
 pytest
 [Rr]edis
+[Rr]eimplement(ing)?
 [Rr]einstall(s?)
 [Rr]eplatform(ing)?
 [Rr]epo
@@ -135,6 +140,7 @@ Tavily
 [Tt]okenization
 [Tt]okenizer(s?)
 triages
+[Uu]ncomment
 [Uu]nencrypted
 [Uu]nittest(s?)
 [Uu]nprocessable
@@ -150,4 +156,4 @@ zsh
 Zep
 Optuna
 [Oo]ptimizable
-[Cc]heckpointed
+[Cc]heckpointed

docker/Dockerfile

@@ -22,6 +22,10 @@ ARG NAT_VERSION
 
 FROM --platform=$TARGETPLATFORM ${BASE_IMAGE_URL}:${BASE_IMAGE_TAG}
 
+ARG PYTHON_VERSION
+ARG UV_VERSION
+ARG NAT_VERSION
+
 COPY --from=ghcr.io/astral-sh/uv:${UV_VERSION} /uv /uvx /bin/
 
 ENV PYTHONDONTWRITEBYTECODE=1

docs/source/conf.py

@@ -27,32 +27,60 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 
+import glob
 import os
 import shutil
 import subprocess
 import typing
+from pathlib import Path
 
 if typing.TYPE_CHECKING:
     from autoapi._objects import PythonObject
 
-CUR_DIR = os.path.dirname(os.path.abspath(__file__))
-DOC_DIR = os.path.dirname(CUR_DIR)
-ROOT_DIR = os.path.dirname(os.path.dirname(CUR_DIR))
-NAT_DIR = os.path.join(ROOT_DIR, "src", "nat")
 
-# Work-around for https://github.com/readthedocs/sphinx-autoapi/issues/298
-# AutoAPI support for implicit namespaces is broken, so we need to manually
-# construct an nat package with an __init__.py file
-BUILD_DIR = os.path.join(DOC_DIR, "build")
-API_TREE = os.path.join(BUILD_DIR, "_api_tree")
+def _build_api_tree() -> Path:
+    # Work-around for https://github.com/readthedocs/sphinx-autoapi/issues/298
+    # AutoAPI support for implicit namespaces is broken, so we need to manually
 
-if os.path.exists(API_TREE):
-    shutil.rmtree(API_TREE)
+    cur_dir = Path(os.path.abspath(__file__)).parent
+    docs_dir = cur_dir.parent
+    root_dir = docs_dir.parent
+    nat_dir = root_dir / "src" / "nat"
+    plugins_dir = root_dir / "packages"
 
-os.makedirs(API_TREE)
-shutil.copytree(NAT_DIR, os.path.join(API_TREE, "nat"))
-with open(os.path.join(API_TREE, "nat", "__init__.py"), "w") as f:
-    f.write("")
+    build_dir = docs_dir / "build"
+    api_tree = build_dir / "_api_tree"
+    dest_dir = api_tree / "nat"
+
+    if api_tree.exists():
+        shutil.rmtree(api_tree.absolute())
+
+    os.makedirs(api_tree.absolute())
+    shutil.copytree(nat_dir, dest_dir)
+    dest_plugins_dir = dest_dir / "plugins"
+
+    for sub_dir in (dest_dir, dest_plugins_dir):
+        with open(sub_dir / "__init__.py", "w", encoding="utf-8") as f:
+            f.write("")
+
+    plugin_dirs = [Path(p) for p in glob.glob(f'{plugins_dir}/nvidia_nat_*')]
+    for plugin_dir in plugin_dirs:
+        src_dir = plugin_dir / 'src/nat/plugins'
+        if src_dir.exists():
+            for plugin_subdir in src_dir.iterdir():
+                if plugin_subdir.is_dir():
+                    dest_subdir = dest_plugins_dir / plugin_subdir.name
+                    shutil.copytree(plugin_subdir, dest_subdir)
+                    package_file = dest_subdir / "__init__.py"
+                    if not package_file.exists():
+                        with open(package_file, "w", encoding="utf-8") as f:
+                            f.write("")
+
+    return api_tree
+
+
+API_TREE = _build_api_tree()
+print(f"API tree built at {API_TREE}")
 
 # -- Project information -----------------------------------------------------
 
@@ -87,7 +115,7 @@
     "sphinxmermaid"
 ]
 
-autoapi_dirs = [API_TREE]
+autoapi_dirs = [str(API_TREE.absolute())]
 
 autoapi_root = "api"
 autoapi_python_class_content = "both"

docs/source/quick-start/installing.md

@@ -36,14 +36,18 @@ To install these first-party plugin libraries, you can use the full distribution
 
 - `nvidia-nat[adk]` or `nvidia-nat-adk` - [Google ADK](https://github.com/google/adk-python)
 - `nvidia-nat[agno]` or `nvidia-nat-agno` - [Agno](https://agno.com/)
+- `nvidia-nat[all]` or `nvidia-nat-all` - Pseudo-package for installing all optional dependencies
 - `nvidia-nat[crewai]` or `nvidia-nat-crewai` - [CrewAI](https://www.crewai.com/)
 - `nvidia-nat[data-flywheel]` or `nvidia-nat-data-flywheel` - [NeMo DataFlywheel](https://github.com/NVIDIA-AI-Blueprints/data-flywheel)
+- `nvidia-nat[ingestion]` or `nvidia-nat-ingestion` - Additional dependencies needed for data ingestion
 - `nvidia-nat[langchain]` or `nvidia-nat-langchain` - [LangChain](https://www.langchain.com/), [LangGraph](https://www.langchain.com/langgraph)
 - `nvidia-nat[llama-index]` or `nvidia-nat-llama-index` - [LlamaIndex](https://www.llamaindex.ai/)
+- `nvidia-nat[mcp]` or `nvidia-nat-mcp` - [Model Context Protocol (MCP)](https://modelcontextprotocol.io/)
 - `nvidia-nat[mem0ai]` or `nvidia-nat-mem0ai` - [Mem0](https://mem0.ai/)
 - `nvidia-nat[mysql]` or `nvidia-nat-mysql` - [MySQL](https://www.mysql.com/)
 - `nvidia-nat[opentelemetry]` or `nvidia-nat-opentelemetry` - [OpenTelemetry](https://opentelemetry.io/)
 - `nvidia-nat[phoenix]` or `nvidia-nat-phoenix` - [Arize Phoenix](https://arize.com/docs/phoenix)
+- `nvidia-nat[profiling]` or `nvidia-nat-profiling` - Additional dependencies needed for profiling
 - `nvidia-nat[ragaai]` or `nvidia-nat-ragaai` - [RagaAI Catalyst](https://raga.ai/)
 - `nvidia-nat[redis]` or `nvidia-nat-redis` - [Redis](https://redis.io/)
 - `nvidia-nat[s3]` or `nvidia-nat-s3` - [Amazon S3](https://aws.amazon.com/s3/)