NVIDIA-AI-Blueprints · tanleach · Jun 28, 2026 · Jun 29, 2026 · Jun 29, 2026 · Jun 30, 2026
diff --git a/deploy/helm/README.md b/deploy/helm/README.md
@@ -189,9 +189,14 @@ aiq-postgres-xxx                1/1     Running   0          30s
 
 ```bash
 kubectl port-forward -n ns-aiq svc/aiq-backend 8000:8000 &
+curl http://localhost:8000/live
 curl http://localhost:8000/health
 ```
 
+The backend liveness probe uses `/live`, which checks only that the API process
+responds. The readiness probe uses `/health`, which checks required dependencies
+and can return HTTP 503 without causing Kubernetes to restart the process.
+
 The backend API docs are available at `http://localhost:8000/docs` while the port-forward is active.
 
 ### Access the application

diff --git a/deploy/helm/deployment-k8s/README.md b/deploy/helm/deployment-k8s/README.md
@@ -68,9 +68,14 @@ aiq-postgres-xxx                1/1     Running   0          30s
 
 ```bash
 kubectl port-forward -n ns-aiq svc/aiq-backend 8000:8000 &
+curl http://localhost:8000/live
 curl http://localhost:8000/health
 ```
 
+The backend liveness probe uses `/live`, which checks only that the API process
+responds. The readiness probe uses `/health`, so database or content-encryption
+outages remove the pod from service without triggering a restart loop.
+
 ## Configuration, FRAG, Secrets, and Access
 
 These topics apply to all deployment methods and are documented in the [Helm README](../README.md):

diff --git a/deploy/helm/deployment-k8s/values.yaml b/deploy/helm/deployment-k8s/values.yaml
@@ -49,7 +49,7 @@ aiq:
         enabled: true
         livenessProbe:
           httpGet:
-            path: /health
+            path: /live
             port: 8000
           initialDelaySeconds: 30
           periodSeconds: 15

diff --git a/docs/source/deployment/content-encryption.md b/docs/source/deployment/content-encryption.md
@@ -0,0 +1,216 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Async Job Content Encryption
+
+AI-Q can encrypt sensitive async job content before it is persisted by the
+AI-Q async jobs API. This is application-level envelope encryption for final
+report output and selected artifact event payload fields.
+
+Content encryption is disabled by default. Operators must explicitly set
+`AIQ_CONTENT_ENCRYPTION=key` or `vault` on every API and worker process to
+enable it.
+
+This feature is intentionally narrow in its first milestone. It protects the
+serialized final output payload returned by
+`GET /v1/jobs/async/job/{job_id}/report`, such as `{"report": "..."}`, and
+the `content` field of `artifact.update` events for `output` and `file`
+artifacts.
+
+## Scope and Limitations
+
+Encrypted when enabled:
+
+- `job_info.output` for jobs submitted through `/v1/jobs/async/submit` or
+  `aiq_api.jobs.submit.submit_agent_job`.
+- `job_events.event_data` field `artifact.update.data.content` when the
+  artifact `data.type` is `output` or `file`.
+
+Still plaintext:
+
+- Other `job_events.event_data` fields, including artifact metadata, tool
+  events, heartbeat events, cancellation events, error events, citation source
+  artifacts, citation use artifacts, and todo artifacts.
+- Job status, ownership metadata, timestamps, event type, and other control
+  plane fields.
+- `job_info.error`.
+- PostgreSQL notification payloads.
+- `summaries.summary`.
+- LangGraph checkpoints in `aiq_checkpoints`.
+- Historical `job_info.output` rows written before encryption was enabled.
+- Inline CLI and local NeMo Agent Toolkit runs that do not use the AI-Q async
+  API job runner.
+
+Because checkpoints, errors, citations, todos, and other event fields can
+contain equivalent research content, this phase does not provide full
+database-level job-content confidentiality.
+
+This feature protects the configured fields from storage readers and detects
+ciphertext tampering within the authenticated job and field context. It is not
+a database integrity or event-freshness control. A principal with database
+write access can still delete or reorder rows, replay an entire encrypted row,
+or swap encrypted artifact content between events in the same job when the
+authenticated field path is identical.
+
+## Modes
+
+Set `AIQ_CONTENT_ENCRYPTION` on every API and worker process.
+
+| Mode | Behavior |
+|------|----------|
+| `off` | Default. Preserves existing plaintext behavior and never attempts to decrypt `aiqenc:` values. |
+| `key` | Uses one operator-managed static 32-byte key to wrap per-job data encryption keys. Intended only for development, testing, or deployments that cannot use Vault. |
+| `vault` | Uses HashiCorp Vault Transit to generate and wrap per-job data encryption keys. Recommended for production. |
+
+Encrypted values are stored as `aiqenc:` envelopes. Encrypted event fields are
+stored as marker objects containing the envelope so the surrounding event JSON
+remains inspectable. The envelope contains non-secret metadata, the wrapped data
+encryption key, nonce, ciphertext, tag, algorithm, key id, and an AAD hint that
+binds the value to either `job_info.output:{job_id}` or the encrypted
+`job_events.event_data` field path for that job.
+
+## Static Key Configuration
+
+Static key mode requires a base64 or base64url value that decodes to exactly
+32 raw bytes.
+
+```bash
+AIQ_CONTENT_ENCRYPTION=key
+AIQ_CONTENT_ENCRYPTION_KEY=<base64url-encoded-32-byte-key>
+AIQ_CONTENT_ENCRYPTION_KEY_ID=<operator-managed-key-id>
+```
+
+`AIQ_CONTENT_ENCRYPTION_KEY_ID` is optional, but it is cryptographic identity,
+not cosmetic metadata: it is authenticated while wrapping each data encryption
+key. If omitted, envelopes use `static-key` as the key id. Keep the configured
+key id unchanged while encrypted jobs are retained. The first implementation
+supports one active static key only; changing the key or key id makes existing
+envelopes unreadable unless those jobs have expired or a future rewrap process
+has migrated them.
+
+Invalid static-key configuration fails startup.
+
+## Vault Transit Configuration
+
+Vault mode uses AppRole authentication and Transit data keys. Token fallback is
+not supported in the first implementation.
+
+```bash
+AIQ_CONTENT_ENCRYPTION=vault
+VAULT_ADDR=<vault-address>
+VAULT_ROLE_ID=<approle-role-id>
+VAULT_SECRET_ID=<approle-secret-id>
+AIQ_ENCRYPTION_TRANSIT_KEY=<transit-key-name>
+VAULT_TRANSIT_MOUNT=<transit-mount>
+AIQ_CONTENT_ENCRYPTION_KEY_ID=<logical-key-id>
+```
+
+`VAULT_TRANSIT_MOUNT` defaults to `transit` if omitted.
+`AIQ_CONTENT_ENCRYPTION_KEY_ID` is optional; if omitted, envelopes use
+`<transit-mount>/<transit-key-name>`.
+
+Set `VAULT_NAMESPACE=<vault-namespace>` only when your Vault deployment
+requires a namespace.
+
+Missing Vault configuration fails startup. If Vault configuration is present
+but Vault is temporarily unreachable, unauthorized, or otherwise operationally
+unready, the API starts unhealthy instead of exiting.
+
+The application relies on Vault Transit versioned ciphertext for decrypting
+after Transit key rotation. Do not disable or destroy old Transit key versions
+until corresponding encrypted jobs have expired or have been rewrapped by a
+future migration process.
+
+## Vault Client and Retry Behavior
+
+Vault mode uses the synchronous `hvac.Client`. API routes that may call Vault
+offload that work to worker threads so Vault latency does not block the FastAPI
+event loop. Each API or worker process keeps one process-local Vault client and
+guards client creation, AppRole login, and Transit calls with a lock. This
+serializes Vault Transit operations per process and avoids concurrent mutation
+of the shared `hvac.Client`.
+
+Vault Transit operations use bounded retries for transient failures only:
+
+- request connection failures and timeouts,
+- Vault rate limiting,
+- Vault 5xx-style operational failures, and
+- unauthorized responses after forcing one AppRole re-login.
+
+The retry policy does not retry missing configuration, invalid requests,
+permission-denied responses, missing Transit paths, malformed Vault responses,
+or decrypt denials. Tune `VAULT_TIMEOUT_SECONDS` to bound individual Vault
+requests and `AIQ_CONTENT_ENCRYPTION_READINESS_TTL_SECONDS` to control how
+often health and submit readiness recheck Vault.
+
+## Rollout Behavior
+
+The feature defaults to `off`, so upgrading AI-Q does not change persisted job
+content unless an operator explicitly enables encryption. The first
+implementation is forward-only after enablement:
+
+- New `job_info.output` writes are encrypted after enablement.
+- New `artifact.update.data.content` event fields are encrypted after
+  enablement for `output` and `file` artifacts.
+- Existing plaintext `job_info.output` rows are intentionally unreadable while
+  `AIQ_CONTENT_ENCRYPTION=key` or `vault`.
+- Existing plaintext event rows remain readable in encrypted modes.
+- No historical plaintext backfill is included.
+- No rewrap tooling is included.
+
+Migration, backfill, application-managed key rotation, and decrypt-on-rollback
+tooling are outside the scope of this release. Enable encryption only for a new
+or empty job history, after existing jobs have expired, or after operators
+accept that old plaintext final-report rows in `job_info.output` cannot be read
+in encrypted modes.
+
+Keep the encryption mode, static key and key id, or Vault Transit mount and key
+name stable while encrypted jobs are retained. Switching the mode to `off`
+does not decrypt existing values: report reads can expose the stored `aiqenc:`
+value and encrypted event fields remain marker objects. Restore the original
+encryption configuration to read those jobs.
+
+## Health and Failure Behavior
+
+`/live` reports process liveness without calling Vault or the database. `/health`
+reports dependency readiness and includes encryption status. When encryption is
+configured but unready, `/health` returns HTTP 503 and new async submissions are
+rejected with HTTP 503, while `/live` remains successful so an orchestrator does
+not restart an otherwise live API process during a dependency outage.
+
+Workers independently validate encryption before marking a job `RUNNING`. If
+encryption is unavailable at worker startup, the job is marked `FAILURE` and
+the agent does not run.
+
+Each submission also carries a non-secret encryption policy identity from the
+API process to the worker. The identity covers the mode and key id, a static-key
+fingerprint in `key` mode, or the Vault address, namespace, Transit mount, and
+Transit key in `vault` mode. The worker fails the job before `RUNNING` if its
+local policy does not match, including `key` or `vault` submissions received by
+a worker configured with `off`.
+
+If final-report encryption, artifact event-content encryption, or encrypted
+persistence fails after an agent has completed, the job is marked `FAILURE`.
+Timer-triggered event flush failures are surfaced before the worker can mark
+the job `SUCCESS`. The worker does not fall back to writing plaintext output.
+When encryption is `off`, event persistence retains its legacy best-effort
+behavior: database write failures are logged without changing the job status.
+
+Report reads fail closed:
+
+- Vault or crypto unavailability returns HTTP 503.
+- Plaintext, malformed, or undecryptable `job_info.output` in encrypted mode
+  returns HTTP 500.
+- Job access is authorized before decryption is attempted.
+
+## Cache
+
+Decrypt paths use an in-memory plaintext data encryption key cache per process.
+The default TTL is 15 minutes with a maximum of 1024 entries. Set
+`AIQ_CONTENT_ENCRYPTION_DEK_CACHE_TTL_SECONDS=0` to disable this cache.
+
+The readiness cache defaults to 60 seconds. Health checks and submit requests
+reuse the cached state until it becomes stale. Set
+`AIQ_CONTENT_ENCRYPTION_READINESS_TTL_SECONDS` to override the default.
diff --git a/docs/source/deployment/index.md b/docs/source/deployment/index.md
@@ -33,6 +33,8 @@ All containerized deployments run the same three services:
 
 - **[Authentication](./authentication.md)** -- Enable OAuth/OIDC sign-in, configure backend JWT validation, and use AIQ user tokens in tools and MCP pass-through integrations.
 
+- **[Async Job Content Encryption](./content-encryption.md)** -- Configure encryption at rest for async final reports and selected artifact event content, including Vault Transit and static-key modes.
+
 - **[Observability](./observability.md)** -- Tracing and monitoring with Phoenix, LangSmith, Weave, and OpenTelemetry.
 
 - **[Production Considerations](./production.md)** -- Guidance on managed databases, horizontal scaling, security hardening, monitoring, and resource requirements.

diff --git a/docs/source/deployment/kubernetes.md b/docs/source/deployment/kubernetes.md
@@ -154,9 +154,14 @@ Once all pods are running, verify the backend is responding:
 
 ```bash
 kubectl port-forward -n ns-aiq svc/aiq-backend 8000:8000 &
+curl http://localhost:8000/live
 curl http://localhost:8000/health
 ```
 
+The chart uses `/live` for the liveness probe and `/health` for readiness. The
+liveness endpoint checks only that the API process responds; database or content-
+encryption outages make the pod unready without causing a restart loop.
+
 The backend API docs are available at `http://localhost:8000/docs` while the port-forward is active.
 
 ### Access the application

diff --git a/docs/source/deployment/production.md b/docs/source/deployment/production.md
@@ -115,11 +115,18 @@ Store API keys in `deploy/.env` and ensure the file is not committed to version
 
 ## Monitoring
 
-### Health Endpoint
+### Liveness and Readiness Endpoints
 
-The backend exposes a health endpoint at `/health` for liveness and readiness probes.
+The backend exposes separate probe endpoints:
+
+- `/live` checks only that the API process can respond. Use it for liveness probes.
+- `/health` checks database and content-encryption dependencies. Use it for readiness probes.
+
+This separation keeps a temporary dependency outage from restarting a live API process while still removing an
+unready instance from service traffic.
 
 ```bash
+curl http://localhost:8000/live
 curl http://localhost:8000/health
 ```
 

diff --git a/docs/source/index.md b/docs/source/index.md
@@ -98,6 +98,7 @@ Overview <./deployment/index.md>
 Docker Compose <./deployment/docker-compose.md>
 Docker Build System <./deployment/docker-build.md>
 Authentication <./deployment/authentication.md>
+Async Job Content Encryption <./deployment/content-encryption.md>
 Observability <./deployment/observability.md>
 Production <./deployment/production.md>
 Kubernetes <./deployment/kubernetes.md>

diff --git a/docs/source/integration/rest-api.md b/docs/source/integration/rest-api.md
@@ -35,7 +35,8 @@ Base path: `/v1/jobs/async`
 | `GET` | `/v1/jobs/async/job/{job_id}/state` | Get accumulated job artifacts |
 | `GET` | `/v1/jobs/async/job/{job_id}/report` | Get final research report |
 | `GET` | `/v1/data_sources` | List available data sources |
-| `GET` | `/health` | Health check (includes Dask status) |
+| `GET` | `/live` | Process liveness check (no dependency checks) |
+| `GET` | `/health` | Dependency readiness check (database, Dask, and content encryption) |
 
 ### List Available Agents
 
@@ -451,7 +452,23 @@ curl http://localhost:8000/v1/data_sources
 
 The `knowledge_layer` entry only appears when a knowledge retrieval function is configured.
 
-### Health Check
+### Liveness and Readiness Checks
+
+Use `/live` for process liveness probes. It returns success without checking the
+database, Dask, or content-encryption dependencies.
+
+```bash
+curl http://localhost:8000/live
+```
+
+```json
+{
+  "status": "alive"
+}
+```
+
+Use `/health` for readiness checks. It returns HTTP 503 when a required
+dependency is unavailable.
 
 ```bash
 curl http://localhost:8000/health
@@ -461,8 +478,13 @@ curl http://localhost:8000/health
 
 ```json
 {
-  "status": "ok",
-  "dask_available": true
+  "status": "healthy",
+  "dask_available": true,
+  "db": "ok",
+  "encryption": {
+    "mode": "off",
+    "ready": true
+  }
 }
 ```
 

diff --git a/frontends/aiq_api/pyproject.toml b/frontends/aiq_api/pyproject.toml
@@ -47,6 +47,8 @@ dependencies = [
     "python-multipart>=0.0.27,<0.1",
     # JWT validation for AuthMiddleware (RS256 via JWKS)
     "PyJWT[cryptography]>=2.8.0",
+    # Vault Transit support for async final-report encryption at rest.
+    "hvac>=2.3.0,<3",
 ]
 
 [project.optional-dependencies]

diff --git a/frontends/aiq_api/src/aiq_api/auth/middleware.py b/frontends/aiq_api/src/aiq_api/auth/middleware.py
@@ -182,6 +182,7 @@ def build_request_trace_tags(
 # Paths reachable from outside the cluster.  Any external request whose path
 # does NOT match one of these receives 404.  Prefix entries must end with "/".
 EXTERNAL_ALLOWED_PATHS: list[str] = [
+    "/live",
     "/health",
     "/docs",
     "/redoc",
@@ -197,7 +198,7 @@ def build_request_trace_tags(
 ]
 
 # External paths that require no token (monitoring, etc.)
-AUTH_EXEMPT_PATHS: set[str] = {"/health", "/docs", "/redoc", "/openapi.json"}
+AUTH_EXEMPT_PATHS: set[str] = {"/live", "/health", "/docs", "/redoc", "/openapi.json"}
 
 
 def _is_oauth_callback_path(path: str) -> bool: