Simplify dataset onboarding for clinical researchers

[hannesill]

Problem

Adding custom datasets to M4 currently requires understanding technical concepts that create friction for clinical researchers:

1. Manual JSON authoring - Users must create m4_data/datasets/*.json with correct field names
2. Non-intuitive schema mapping - The relationship between directories (hosp/) and schemas (mimiciv_hosp) isn't obvious
3. No auto-discovery - Even with data files ready, users must manually configure the dataset definition
4. Silent failures - Missing or incorrect schema_mapping fails at query time with unclear errors

Current Flow (Custom Dataset)

# 1. Create JSON definition manually
cat > m4_data/datasets/my-dataset.json << 'JSON'
{
  "name": "my-dataset",
  "description": "My hospital data",
  "modalities": ["TABULAR"],
  "subdirectories_to_scan": ["core", "labs"],
  "default_duckdb_filename": "my_dataset.duckdb",
  "primary_verification_table": "core.patients",
  "schema_mapping": {
    "core": "my_core",
    "labs": "my_labs"
  }
}
JSON

# 2. Place files in correct structure
# 3. Run init
m4 init my-dataset

Proposed Simplifications

Option 1: Convention-Based Auto-Discovery

m4 init --from ./my_hospital_data/

• Auto-detect directories → schemas (identity mapping by default)
• Auto-detect CSV/Parquet files → tables
• Infer primary_verification_table from first table found
• No JSON required for simple cases

Option 2: Interactive Wizard

m4 init --wizard

# Where is your data? [./data/]
# Found directories: core/, labs/, notes/
# Map "core" to schema name? [core] 
# Primary table for verification? [core.patients]
# Generate dataset definition? [Y/n]

Option 3: Direct Database/Directory Init

m4 init my-data.duckdb        # Point to existing DuckDB file
m4 init ./parquet/            # Auto-discover from Parquet directory

Option 4: Simplify DatasetDefinition Defaults

• schema_mapping defaults to identity (directory name = schema name)
• subdirectories_to_scan auto-detected if not specified
• primary_verification_table auto-selected from first schema/table

Success Criteria

A clinical researcher with Parquet files in a directory should be able to start querying with minimal commands and zero JSON authoring.

Related Files

• src/m4/core/datasets.py - DatasetDefinition, DatasetRegistry
• src/m4/cli.py - CLI commands including init
• src/m4/core/backends/duckdb.py - DuckDB initialization logic