feat(artifacts): add S3-compatible blob storage

deploy/.env.example

@@ -47,6 +47,21 @@ DASK_DISTRIBUTED__LOGGING__DISTRIBUTED=warning
 # -----------------------------------------------------------------------------
 # AIQ_SUMMARY_DB=
 
+# -----------------------------------------------------------------------------
+# Artifact byte storage
+# Leave AIQ_ARTIFACT_BLOB_PROVIDER unset (or set it to sql) to keep bytes in SQL.
+# Set it to s3 for AWS S3, MinIO, Ceph, R2, or another S3-compatible service.
+# S3 storage requires the `s3` package extra and a bucket. It does not fall back
+# to SQL when an object upload fails.
+# -----------------------------------------------------------------------------
+# AIQ_ARTIFACT_BLOB_PROVIDER=sql
+# AIQ_ARTIFACT_S3_BUCKET=aiq-artifacts       # Required when provider=s3
+# AIQ_ARTIFACT_S3_ENDPOINT_URL=              # Unset for AWS; set for MinIO/compatible storage
+# AIQ_ARTIFACT_S3_REGION=us-west-2           # Optional
+# AIQ_ARTIFACT_S3_PREFIX=artifacts/v1        # Optional; shown value is the default
+# Credentials use workload identity or the standard AWS credential chain.
+# Local MinIO may set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in deploy/.env.
+
 # -----------------------------------------------------------------------------
 # Environment Variables for Modal Sandbox for Skills Execution (optional)
 # -----------------------------------------------------------------------------

deploy/Dockerfile

@@ -69,8 +69,9 @@ COPY configs/ ./configs/
 # to avoid leaking .env, Helm charts, compose files, or other dev artifacts.
 COPY deploy/entrypoint.py deploy/start_web.py ./deploy/
 
-# Install dependencies using uv sync
-RUN uv sync --frozen --no-dev --no-install-workspace
+# Install dependencies using uv sync. The image includes the optional S3 SDK so
+# deployments can opt into object storage through configuration at runtime.
+RUN uv sync --frozen --no-dev --no-install-workspace --extra s3
 
 # Install workspace packages (without CLI for base)
 RUN uv pip install --no-deps -e . \

deploy/compose/README.md

@@ -75,6 +75,17 @@ Choose one database configuration in `deploy/.env`:
 - You can keep the `postgres` service running or remove the `depends_on` block
   for `aiq-agent` if you want a SQLite-only setup.
 
+### Artifact Storage
+
+Files generated by sandbox execution, such as charts and CSVs, are captured as
+artifacts. Their bytes remain in SQL when `AIQ_ARTIFACT_BLOB_PROVIDER` is unset or
+set to `sql`. When the provider is `s3`, artifact bytes are stored in the configured
+bucket and SQL stores artifact metadata only. For MinIO, Ceph, R2, or another
+compatible service, set `AIQ_ARTIFACT_S3_ENDPOINT_URL`; leave it unset for AWS S3.
+Region and prefix are optional, with `artifacts/v1` as the default prefix.
+Credentials use the standard AWS credential chain. See `deploy/.env.example` for
+the complete variable list.
+
 ## Start Services
 
 ### Option 1: Build locally (default)

docs/source/architecture/agents/sandbox.md

@@ -34,6 +34,16 @@ runtime (`.../job/{job_id}/artifacts`), which is also auth-scoped to the job.
   be written through DeepAgents virtual paths such as `/shared/`; durable binaries
   (charts, CSVs) are captured by the artifact runtime.
 
+## Artifact Storage
+
+Metadata for generated files such as charts and CSVs is stored in SQL. File content can
+also be stored in SQL, but object storage such as AWS S3 or MinIO is recommended for
+production deployments. The selected artifact storage provider determines where the file
+content is stored.
+
+For configuration variables and examples, see [Docker Compose](../../deployment/docker-compose.md#artifact-storage)
+and [Production Considerations](../../deployment/production.md#artifact-storage).
+
 ## Operational Notes
 
 - High-concurrency Modal runs create one sandbox per job. OpenShell runs share the named

docs/source/deployment/docker-compose.md

@@ -100,6 +100,30 @@ AIQ_CHECKPOINT_DB=/app/data/checkpoints.db
 
 When using SQLite, you can optionally remove the `depends_on` block for the `aiq-agent` service since the `postgres` container is no longer needed.
 
+### Artifact Storage
+
+Artifact metadata always uses the job database. Artifact bytes use SQL BLOB storage
+by default. To store bytes in AWS S3, set:
+
+```bash
+AIQ_ARTIFACT_BLOB_PROVIDER=s3
+AIQ_ARTIFACT_S3_BUCKET=aiq-artifacts
+AIQ_ARTIFACT_S3_REGION=us-west-2
+AIQ_ARTIFACT_S3_PREFIX=artifacts/v1
+```
+
+For MinIO or another S3-compatible service, use the same provider and add its endpoint:
+
+```bash
+AIQ_ARTIFACT_S3_ENDPOINT_URL=http://minio:9000
+```
+
+The bucket is required when the provider is `s3`. Endpoint, region, and prefix are
+optional; leave the endpoint unset for AWS S3, and the prefix defaults to
+`artifacts/v1`. Configure credentials through workload identity, deployment secrets, or
+the standard AWS credential chain. When the provider is `s3`, artifact bytes are stored
+in the configured bucket and SQL stores artifact metadata only.
+
 ### Frontend Runtime Settings
 
 | Variable | Default | Description |

docs/source/deployment/production.md

@@ -49,6 +49,23 @@ pg_dump -U aiq -d aiq_jobs > aiq_jobs_$(date +%Y%m%d).sql
 pg_dump -U aiq -d aiq_checkpoints > aiq_checkpoints_$(date +%Y%m%d).sql
 ```
 
+## Artifact Storage
+
+Keep artifact metadata in PostgreSQL and use S3-compatible object storage for artifact
+bytes in production. SQL BLOB storage remains the default when the provider is unset.
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `AIQ_ARTIFACT_BLOB_PROVIDER` | No | `sql` by default; set to `s3` for object storage. |
+| `AIQ_ARTIFACT_S3_BUCKET` | With S3 | Destination bucket. |
+| `AIQ_ARTIFACT_S3_ENDPOINT_URL` | No | Leave unset for AWS S3; set for MinIO, Ceph, R2, or another compatible endpoint. |
+| `AIQ_ARTIFACT_S3_REGION` | No | S3 region when required by the provider. |
+| `AIQ_ARTIFACT_S3_PREFIX` | No | Object-key prefix; defaults to `artifacts/v1`. |
+
+Configure credentials through workload identity, deployment secrets, or the standard
+AWS credential chain. When the provider is `s3`, artifact bytes are stored in the
+configured bucket and SQL stores artifact metadata only.
+
 ## Scaling
 
 ### Horizontal Backend Scaling

frontends/aiq_api/src/aiq_api/routes/jobs.py

@@ -818,12 +818,12 @@ async def get_job_state(job_id: str) -> JobStateResponse:
     )
     async def list_job_artifacts(job_id: str) -> dict:
         """List durable artifact metadata for a job (no bytes)."""
-        from aiq_agent.agents.deep_researcher.sandbox.artifacts import SqlArtifactStore
+        from aiq_agent.agents.deep_researcher.sandbox.artifacts import build_artifact_store
 
         principal = require_verified_principal()
         await authorize_job_access(job_store, db_url, job_id, principal)
 
-        store = SqlArtifactStore(db_url)
+        store = build_artifact_store(db_url)
         artifacts = await asyncio.to_thread(store.list, job_id)
         # Exclude storage internals (storage_uri embeds the db_url, which may carry
         # credentials/hostnames; sandbox_path is an internal layout detail) from the
@@ -842,12 +842,12 @@ async def list_job_artifacts(job_id: str) -> dict:
     )
     async def get_job_artifact_content(job_id: str, artifact_id: str) -> StreamingResponse:
         """Stream an artifact's bytes (auth-scoped to the owning job)."""
-        from aiq_agent.agents.deep_researcher.sandbox.artifacts import SqlArtifactStore
+        from aiq_agent.agents.deep_researcher.sandbox.artifacts import build_artifact_store
 
         principal = require_verified_principal()
         await authorize_job_access(job_store, db_url, job_id, principal)
 
-        store = SqlArtifactStore(db_url)
+        store = build_artifact_store(db_url)
         artifact = await asyncio.to_thread(store.get, job_id, artifact_id)
         if artifact is None:
             raise HTTPException(404, f"Artifact not found: {artifact_id}")
@@ -1188,10 +1188,10 @@ def _do_cleanup() -> tuple[int, int, int]:
     # Artifact retention shares the job expiry boundary (best-effort; the artifacts
     # table only exists when artifact capture has been used).
     try:
-        from aiq_agent.agents.deep_researcher.sandbox.artifacts import SqlArtifactStore
+        from aiq_agent.agents.deep_researcher.sandbox.artifacts import build_artifact_store
 
         artifacts_deleted = await loop.run_in_executor(
-            None, lambda: SqlArtifactStore(db_url).cleanup_old_artifacts(retention_seconds)
+            None, lambda: build_artifact_store(db_url).cleanup_old_artifacts(retention_seconds)
         )
         if artifacts_deleted:
             logger.info("Artifact cleanup: %d old artifacts removed", artifacts_deleted)

pyproject.toml

@@ -52,6 +52,9 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
+s3 = [
+    "boto3>=1.35.0,<2",
+]
 dev = [
     "pytest>=7.4.0",
     "pytest-asyncio>=0.21.0",

src/aiq_agent/agents/deep_researcher/deepagents_runtime.py

@@ -273,12 +273,12 @@ def _maybe_build_artifact_manager(
     if capture is None or not getattr(capture, "enabled", False):
         return None
     from .sandbox.artifacts import ArtifactManager
-    from .sandbox.artifacts import SqlArtifactStore
+    from .sandbox.artifacts import build_artifact_store
 
     return ArtifactManager(
         job_id=job_id,
         backend=provider,
-        store=SqlArtifactStore(artifact_db_url),
+        store=build_artifact_store(artifact_db_url),
         config=capture,
         artifact_dir=artifact_dir,
         emit=artifact_emit,

src/aiq_agent/agents/deep_researcher/sandbox/README.md

@@ -63,7 +63,9 @@ plus `/shared/` for durable text. Binary artifacts are harvested host-side via
 | `providers/openshell.py` | OpenShell provider (enterprise/on-prem). Lazy, ad-hoc deps; policy requires a named sandbox. |
 | `artifacts/models.py` | `Artifact` record (id, mime, sha256, size, provenance, status). Metadata only. |
 | `artifacts/manifest.py` | `manifest.json` schema + parser. |
-| `artifacts/store.py` | `SqlArtifactStore` on the shared job `db_url` (metadata table + capped BLOB; pluggable to S3). |
+| `artifacts/store.py` | `SqlArtifactStore` coordinates SQL metadata with the configured byte provider. |
+| `artifacts/blob_store.py` | Byte adapters for SQL BLOBs and S3-compatible object storage. |
+| `artifacts/factory.py` | Builds the application store from `AIQ_ARTIFACT_*` environment variables. |
 | `artifacts/manager.py` | Harvest pipeline: manifest-first + scan, path-traversal confinement, MIME-from-bytes, SVG sanitize, render-gate, quotas, dedup, store-then-emit, `artifact://` resolution. |
 
 ## Adding a provider (the whole surface)
@@ -154,7 +156,7 @@ is lifted into `providers.modal`.
 - Once at the end of the agent run (`agent.run()` -> `ArtifactManager.final_harvest`),
   the `ArtifactManager` pulls bytes via `download_files`, runs the validation pipeline
   (path-traversal confinement -> extension allowlist -> size cap -> MIME-from-bytes/spoof
-  reject -> quota -> SVG sanitize -> sha256), stores via `SqlArtifactStore`, then emits an
+  reject -> quota -> SVG sanitize -> sha256), stores via `ArtifactStore`, then emits an
   `artifact` SSE event (`to_sse_payload`, metadata + `content_url`, never bytes).
 - Failed or cancelled runs are not harvested in the current implementation.
 - Reports reference artifacts as `![caption](artifact://<filename-or-id>)`; the report
@@ -248,13 +250,52 @@ to delegate uploads to the official adapter and validate the upstream argv fix
 always use AI-Q's bounded shim so realpath confinement and pre-transfer size checks remain
 in force. Once the upstream adapter provides equivalent guards, drop the shim and toggle.
 
+## Artifact byte storage
+
+SQL BLOB storage remains the default when `AIQ_ARTIFACT_BLOB_PROVIDER` is unset or
+set to `sql`. Production deployments can set it to `s3` to store bytes in AWS S3 or
+an S3-compatible service while retaining artifact metadata in the job database.
+
+AWS S3:
+
+```bash
+AIQ_ARTIFACT_BLOB_PROVIDER=s3
+AIQ_ARTIFACT_S3_BUCKET=aiq-artifacts
+AIQ_ARTIFACT_S3_REGION=us-west-2
+AIQ_ARTIFACT_S3_PREFIX=artifacts/v1
+```
+
+MinIO or another S3-compatible service uses the same provider and additionally sets
+the custom endpoint:
+
+```bash
+AIQ_ARTIFACT_BLOB_PROVIDER=s3
+AIQ_ARTIFACT_S3_BUCKET=aiq-artifacts
+AIQ_ARTIFACT_S3_ENDPOINT_URL=http://minio:9000
+AIQ_ARTIFACT_S3_REGION=us-east-1
+AIQ_ARTIFACT_S3_PREFIX=artifacts/v1
+```
+
+`AIQ_ARTIFACT_S3_BUCKET` is required for S3 storage. The endpoint is optional: leave
+it unset for AWS S3 and set it for MinIO, Ceph, R2, or another compatible endpoint.
+The region and prefix are optional; the prefix defaults to `artifacts/v1`. Credentials
+come from workload identity, deployment secrets, or the standard AWS credential chain.
+For local MinIO, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` can hold the local
+MinIO credentials. Install the optional dependency with `uv sync --extra s3`.
+
+Selecting `s3` does not automatically fall back to SQL if object storage fails. The selected
+provider applies to the whole application. Artifact cleanup follows the retention period and
+removes object bytes before SQL metadata.
+
+Custom endpoints use path-style bucket addressing for MinIO compatibility.
+
 ## Operational knobs
 
 - `AIQ_MAX_SANDBOXES_PER_PRINCIPAL` / `AIQ_MAX_SANDBOXES_GLOBAL` (default-off): submit-path
   concurrency/cost caps for sandbox-enabled jobs.
 - `AIQ_OPENSHELL_ADAPTER_FILE_TRANSFER` (default-off): route OpenShell uploads through the
   official adapter instead of the env-free shim (see OpenShell gotcha above).
-- Artifact retention reuses the job-expiry periodic cleanup (`expiry_seconds`).
+- Artifact retention reuses the existing periodic cleanup (`expiry_seconds`).
 - In-container OpenShell log verbosity (opt-in): `agent.execute()` calls and their output are
   already logged on the AI-Q side (the `execute` tool-call events). To also see what runs
   inside the OpenShell container, rebuild the sandbox image with a higher `RUST_LOG`:

src/aiq_agent/agents/deep_researcher/sandbox/artifacts/__init__.py

@@ -17,6 +17,10 @@
 
 from __future__ import annotations
 
+from .blob_store import ArtifactBlobStore
+from .blob_store import S3ArtifactBlobStore
+from .blob_store import SqlArtifactBlobStore
+from .factory import build_artifact_store
 from .manager import ArtifactManager
 from .manifest import Manifest
 from .manifest import ManifestEntry
@@ -38,6 +42,10 @@
     "ManifestEntry",
     "parse_manifest",
     "ArtifactStore",
+    "ArtifactBlobStore",
+    "SqlArtifactBlobStore",
+    "S3ArtifactBlobStore",
+    "build_artifact_store",
     "LocalArtifactStore",
     "SqlArtifactStore",
     "ArtifactManager",