Add M4Client capabilities and download endpoints

README.md

@@ -213,6 +213,8 @@ Switch datasets or backends anytime:
 ```bash
 m4 use mimic-iv     # Switch to full MIMIC-IV
 m4 backend bigquery # Switch to BigQuery (or duckdb)
+m4 capabilities     # Show available interfaces, datasets, tools, and policies
+m4 doctor           # Diagnose local, BigQuery, and MCP setup
 m4 status           # Show active dataset and backend
 m4 status --all     # List all available datasets
 m4 status --derived # Show per-table derived materialization status
@@ -223,6 +225,9 @@ commands that do not mutate active configuration unless explicitly documented:
 
 ```bash
 m4 agent-env --dataset mimic-iv --backend duckdb --json
+m4 capabilities --json
+m4 doctor --json
+m4 download mimic-iv --json
 m4 list-datasets --json --no-interactive
 m4 schema --dataset mimic-iv --backend duckdb --json --no-interactive
 m4 describe-table mimiciv_hosp.patients --dataset mimic-iv --json --no-interactive
@@ -234,6 +239,17 @@ Machine-facing status and backend metadata hide local filesystem paths by
 default. Use `--paths` or `M4_PATH_DISCLOSURE=1` only when the caller is allowed
 to see raw local paths.
 
+Long-running dataset setup can emit newline-delimited JSON progress events:
+
+```bash
+m4 init mimic-iv --json --events ndjson --no-interactive --download \
+  --physionet-credentials-file /path/to/physionet-credentials.json
+```
+
+When `--events ndjson` is used, stdout is an NDJSON stream instead of a single
+JSON object. The final result is emitted as `operation_completed.result`; setup
+failures are emitted as `operation_failed.error`.
+
 **Derived concept tables** (MIMIC-IV only):
 ```bash
 m4 init-derived mimic-iv         # Materialize ~63 derived tables (SOFA, sepsis3, KDIGO, etc.)
@@ -247,21 +263,47 @@ After running `m4 init mimic-iv`, you are prompted whether to materialize derive
 
 1. **Get PhysioNet credentials:** Complete the [credentialing process](https://physionet.org/settings/credentialing/) and sign the data use agreement for the dataset.
 
-2. **Download the data:**
+2. **Download the data with M4:**
+   ```bash
+   cat > physionet-credentials.json <<'JSON'
+   {
+     "username": "YOUR_USERNAME",
+     "password": "YOUR_PASSWORD"
+   }
+   JSON
+
+   m4 init mimic-iv --download --physionet-credentials-file physionet-credentials.json
+   ```
+
+   Do not pass PhysioNet passwords as command-line flags. Use a scoped
+   credentials file with restrictive permissions and delete it after setup.
+
+   M4 implements the same recursive, resumable pattern PhysioNet documents for
+   `wget -r -N -c -np` against `/files/...` dataset URLs, while preserving the
+   expected raw layout under `m4_data/raw_files/<dataset>/`.
+
+   You can still download manually if needed:
+   ```bash
+   m4 download mimic-iv
+   ```
+
+   For credentialed datasets, `m4 download` validates the expected local layout
+   and prints a dataset-specific resumable `wget` command.
+
    ```bash
    # For MIMIC-IV
-   wget -r -N -c -np --cut-dirs=2 -nH --user YOUR_USERNAME --ask-password \
+   wget -r -N -c -np --cut-dirs=3 -nH --user YOUR_USERNAME --ask-password \
      https://physionet.org/files/mimiciv/3.1/ \
      -P m4_data/raw_files/mimic-iv
 
    # For eICU
-   wget -r -N -c -np --cut-dirs=2 -nH --user YOUR_USERNAME --ask-password \
+   wget -r -N -c -np --cut-dirs=3 -nH --user YOUR_USERNAME --ask-password \
      https://physionet.org/files/eicu-crd/2.0/ \
      -P m4_data/raw_files/eicu
    ```
-   The `--cut-dirs=2 -nH` flags ensure CSV files land directly in `m4_data/raw_files/mimic-iv/` rather than a nested `physionet.org/files/...` structure.
+   The `--cut-dirs=3 -nH` flags remove the PhysioNet `files/<dataset>/<version>/` prefix so CSV files land under `m4_data/raw_files/<dataset>/` with only dataset-internal folders preserved.
 
-3. **Initialize:**
+3. **Initialize after a manual download:**
    ```bash
    m4 init mimic-iv   # or: m4 init eicu
    ```

docs/BIGQUERY.md

@@ -2,6 +2,10 @@
 
 Use Google Cloud BigQuery to access full clinical datasets without downloading files locally.
 
+For local DuckDB workflows, use `m4 download DATASET` followed by
+`m4 init DATASET`. For BigQuery workflows, skip local download entirely and
+configure credentials/project billing as described below.
+
 ## Prerequisites
 
 1. **Google Cloud account** with BigQuery access
@@ -38,6 +42,13 @@ m4 config --backend bigquery --project-id YOUR_PROJECT_ID
 
 Replace `YOUR_PROJECT_ID` with your own billing project for BigQuery usage, not the PhysioNet dataset project. The variable is mandatory to ensure billing is correctly attributed.
 
+You can also emit agent-ready environment guidance without changing config:
+
+```bash
+m4 setup-agent --backend bigquery --project-id YOUR_PROJECT_ID --format dotenv
+m4 doctor --json
+```
+
 ### 4. Set the dataset
 
 ```bash
@@ -100,7 +111,15 @@ BigQuery charges based on data scanned. Tips to minimize costs:
 **"Project not found" error:**
 - Check the project ID is correct
 - Ensure BigQuery API is enabled in your project
+- Confirm `M4_PROJECT_ID` or `m4 config --project-id` refers to your billing project
 
 **Slow queries:**
 - BigQuery has network latency; consider local DuckDB for development
 - Use smaller `LIMIT` values while exploring
+
+**Local download layout problems:**
+- Run `m4 download mimic-iv` or `m4 download eicu` to print dataset-specific
+  recovery guidance and a resumable `wget` command.
+- If files landed under `physionet.org/files/...`, move the dataset contents up
+  to `m4_data/raw_files/DATASET` or rerun `wget` with the generated
+  `--cut-dirs` and `-nH` flags.

docs/CUSTOM_DATASETS.md

@@ -147,10 +147,13 @@ For datasets requiring PhysioNet credentials (most full datasets):
 1. Get credentialed access on PhysioNet
 2. Download manually using wget:
    ```bash
-   wget -r -N -c -np --user YOUR_USERNAME --ask-password \
+   wget -r -N -c -np --cut-dirs=3 -nH --user YOUR_USERNAME --ask-password \
      https://physionet.org/files/dataset-name/version/ \
      -P m4_data/raw_files/dataset-name
    ```
+   Set `--cut-dirs` to the number of path components in the listing URL
+   (`files/dataset-name/version` is 3) so files land in the top-level raw
+   layout expected by `m4 init`.
 3. Initialize:
    ```bash
    m4 init dataset-name

docs/DEVELOPMENT.md

@@ -67,8 +67,12 @@ non-zero.
     {
       "name": "mimic-iv",
       "active": true,
+      "raw_present": true,
       "parquet_present": true,
       "db_present": true,
+      "requires_authentication": true,
+      "download_available": true,
+      "setup_state": "ready",
       "bigquery_available": true,
       "row_count": 431231,
       "parquet_size_gb": 8.5,
@@ -86,7 +90,7 @@ non-zero.
 
 Raw local paths are hidden by default in machine-facing output. Pass
 `--paths` or set `M4_PATH_DISCLOSURE=1` to include path fields such as
-`parquet_root` and `db_path`.
+`raw_root`, `parquet_root`, and `db_path`.
 
 Dataset `warnings` is a list of stable warning codes. Currently documented
 status warnings:
@@ -125,7 +129,11 @@ Command errors use the same envelope with `ok: false`:
 
 Stable command error codes are `dataset_not_found`, `backend_incompatible`,
 `invalid_backend`, `invalid_option`, `project_id_required`, and
-`dataset_incompatible`.
+`dataset_incompatible`. Dataset setup can also return `missing_credentials`,
+`physionet_auth_failed`, `physionet_access_forbidden`,
+`download_network_failed`, `download_filesystem_failed`,
+`download_interrupted`, `raw_files_missing`, `conversion_failed`,
+`duckdb_init_failed`, and `verification_failed`.
 
 `m4 init DATASET --json` uses the same result/error envelope and runs
 non-interactively. Human prompts, progress panels, and download output are
@@ -142,17 +150,42 @@ states:
   "parquet_root": "/absolute/path/to/parquet/mimic-iv",
   "raw_root": "/absolute/path/to/raw_files/mimic-iv",
   "steps": [
-    {"name": "raw_files", "status": "blocked", "message": "Download manually and rerun init."},
-    {"name": "parquet", "status": "skipped"},
-    {"name": "database", "status": "skipped"}
+    {"name": "raw_files", "status": "completed", "message": "Raw files are present."},
+    {"name": "parquet", "status": "completed", "message": "Converted CSV to Parquet."},
+    {"name": "database", "status": "completed", "message": "Created DuckDB views."}
   ],
   "warnings": []
 }
 ```
 
 Init step status is one of `skipped`, `completed`, `blocked`, or `failed`.
-Credentialed/manual-download cases that the human CLI treats as informational
-return `ok: true` with blocked or skipped steps.
+For credentialed datasets, missing raw/parquet/database artifacts are an
+`ok: false` `raw_files_missing` result unless `--download` is requested.
+
+For progress-aware wrappers, add `--events ndjson`:
+
+```bash
+m4 init mimic-iv --json --events ndjson --no-interactive --download \
+  --physionet-credentials-file /path/to/physionet-credentials.json
+```
+
+With `--events ndjson`, stdout changes from one JSON object to a newline-delimited
+JSON stream. The final command payload appears in `operation_completed.result`;
+setup failures appear in `operation_failed.error`.
+
+PhysioNet credentials files are JSON:
+
+```json
+{
+  "username": "YOUR_USERNAME",
+  "password": "YOUR_PASSWORD"
+}
+```
+
+Do not expose passwords as command-line flags. M4 follows the same source basis
+as PhysioNet's documented recursive resumable downloads (`wget -r -N -c -np`
+from `/files/...`) but performs the download internally so wrappers can receive
+structured progress and error events.
 
 Agent-oriented commands use a stable envelope with `version`, `ok`, `command`,
 `context`, `data`, `warnings`, and optional provenance fields:

src/m4/__init__.py

@@ -13,21 +13,23 @@
 For MCP server usage, run: m4 serve
 """
 
-__version__ = "0.4.5"
+__version__ = "0.5.0"
 
 # Expose API functions at package level for easy imports
 from vitrine import show
 
 from m4.api import (
     # Exceptions
     DatasetError,
+    M4Client,
     M4Error,
     ModalityError,
     QueryError,
     # Tabular data
     execute_query,
     # Dataset management
     get_active_dataset,
+    get_capabilities,
     # Clinical notes
     get_note,
     get_schema,
@@ -43,12 +45,14 @@
 
 __all__ = [
     "DatasetError",
+    "M4Client",
     "M4Error",
     "ModalityError",
     "QueryError",
     "__version__",
     "execute_query",
     "get_active_dataset",
+    "get_capabilities",
     "get_note",
     "get_schema",
     "get_table_info",