Update broker endpoints

BROKER_PREREQUISITES_ENHANCEMENT.md

@@ -0,0 +1,115 @@
+# Broker Prerequisites Enhancement
+
+## Problem
+The broker API was returning null values for `prerequisites`, `validation_hints`, and `file_metadata` fields because:
+1. Missing accession data in submission payloads
+2. No way to distinguish between "not required" vs "required but not yet submitted"
+
+## Solution
+Enhanced the broker contract to include **existing** accessions (resolved from `accession_registry`), allowing clients to decide whether dependency state is complete enough to submit.
+
+### Key Architecture Understanding
+
+**Crucial clarification**: Accessions are stored in different places:
+- **Existing accessions**: Stored in database row fields (`project_accession`, `sample_accession`, etc.) and `accession_registry` table
+- **prepared_payload**: Contains data for ENA submission, NOT existing accessions
+- **Required accessions**: Specified in payload using `expected_*_accession` fields for dependencies not yet submitted
+
+### Changes Made
+
+#### 1. BrokerPrerequisites Schema
+```python
+class BrokerPrerequisites(BaseModel):
+    # Existing accessions (from database row fields)
+    project_accession: Optional[str] = None
+    sample_accession: Optional[str] = None
+    experiment_accession: Optional[str] = None
+    run_accession: Optional[str] = None
+    study_accession: Optional[str] = None
+    analysis_accession: Optional[str] = None
+```
+
+#### 2. Enhanced Prerequisite Extraction Logic
+- **Existing accessions**: Looked up from `accession_registry` via FK relationships on submission tables
+
+#### 3. Updated Test Cases
+- Tests now verify both existing and required accessions
+- Demonstrates mixed states where some dependencies exist and others don't
+
+### Usage Examples
+
+#### Sample Submission with Existing Project Accession
+**Database state**: `accession_registry` contains a project accession for the sample's `project_id`
+```json
+{
+  "alias": "sample-1",
+  "requires_project_accession": true
+}
+```
+Result:
+```json
+{
+  "prerequisites": {
+    "project_accession": "PRJ123456"     // ✅ Resolved from accession_registry
+  }
+}
+```
+
+#### Experiment with Missing Study Accession
+**Database state**: `accession_registry` does not contain a project accession for the experiment's `project_id`
+```json
+{
+  "alias": "experiment-1",
+  "requires_study_accession": true,
+  "expected_study_accession": "PRJ123456"  // Required but not submitted
+}
+```
+Result:
+```json
+{
+  "prerequisites": {
+    "sample_accession": "SAMEA123456",     // ✅ Resolved from accession_registry
+    "study_accession": null               // ❌ Missing from registry
+  }
+}
+```
+
+#### Run with Missing Experiment Accession
+**Database state**: `accession_registry` does not contain an experiment accession for the run's `experiment_id`
+```json
+{
+  "alias": "run-1",
+  "file_name": "reads.fastq.gz",
+  "file_format": "fastq",
+  "expected_experiment_accession": "ERX123456"  // Required but not submitted
+}
+```
+Result:
+```json
+{
+  "prerequisites": {
+    "experiment_accession": null               // ❌ Missing from registry
+  },
+  "files": [
+    {"filename": "reads.fastq.gz", "filetype": "fastq"}
+  ]
+}
+```
+
+### Data Flow Summary
+
+1. **prepared_payload**: Contains ENA submission data (metadata, files, etc.)
+2. **Database row fields**: Store actual accessions once submitted (`*_accession` fields)
+3. **Broker response**: Shows resolved accessions (or `null`), and clients decide completeness
+
+### Client Benefits
+1. **Clear dependency visibility**: See what exists vs what is missing
+2. **Single source of truth**: Accessions resolved from `accession_registry`
+
+### Payload Fields for Required Accessions
+- `expected_*_accession`: Placeholder for the accession that will be assigned
+- `requires_*_accession`: Boolean flag indicating the dependency is required
+- `requires_project_accession`: For samples
+- `requires_study_accession`: For experiments (maps to `required_project_accession` in broker prerequisites)
+
+This enhancement maintains backward compatibility while providing much richer dependency information to broker clients, with the correct understanding of where different types of data are stored.

README.md

@@ -18,6 +18,7 @@ A dedicated broker workflow enables integration with external submission pipelin
 - Submission workflow endpoints (sample-submissions, experiment-submissions, read-submissions)
 - Broker endpoints to support external submission pipelines:
   - Claim drafts and obtain a lease: `/api/v1/broker/organisms/{organism_key}/claim`
+  - ENA broker contract endpoints: `/api/v1/broker/claims/ready`, `/api/v1/broker/claims/entity`, `/api/v1/broker/validation`, `/api/v1/broker/reports/{attempt_id}`
   - Renew lease, finalise, and report results: `/api/v1/broker/attempts/{attempt_id}/...`
   - Attempt listing and summaries for dashboard views
 - Bulk import endpoints for organisms, samples, and experiments
@@ -236,6 +237,7 @@ pyproject.toml, uv.lock, docker-compose.yml, Dockerfile, schema.sql, scripts/, d
       -d '{"samples": [], "experiments": [], "reads": [], "projects": []}'
     ```
 
+  For the flat ENA broker contract used by Canopy, see [docs/ena_broker_contract.md](docs/ena_broker_contract.md).
   For a deeper overview of attempt leasing and statuses, see the `broker` endpoints in `app/api/v1/endpoints/broker.py` and the interactive docs.
 
   ## Bulk Import API

alembic/versions/0006_consolidated_submission_schema_refactor.py

@@ -0,0 +1,136 @@
+"""Consolidated submission schema refactor.
+
+This migration consolidates the following changes:
+- Add project_id to sample_submission (NOT NULL, FK to project)
+- Add project_id to experiment (NOT NULL, FK to project)
+- Drop project_id from read_submission (derivable via experiment)
+- Drop sample_id and project_id from experiment_submission (derivable via experiment)
+
+Note: No backfill logic included - database will be repopulated after merge.
+
+Revision ID: 0006_consolidated_refactor
+Revises: 0005_org_sci_name_nullable
+Create Date: 2026-04-22 00:00:00.000000
+"""
+
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "0006_consolidated_refactor"
+down_revision = "0005_org_sci_name_nullable"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # 1) Add sample_submission.project_id (NOT NULL, FK to project)
+    op.add_column(
+        "sample_submission",
+        sa.Column("project_id", postgresql.UUID(as_uuid=True), nullable=False),
+    )
+    op.create_foreign_key(
+        "fk_sample_submission_project_id",
+        "sample_submission",
+        "project",
+        ["project_id"],
+        ["id"],
+    )
+    op.create_index(
+        "idx_sample_submission_project_id",
+        "sample_submission",
+        ["project_id"],
+        unique=False,
+    )
+
+    # 2) Add experiment.project_id (NOT NULL, FK to project with CASCADE)
+    op.add_column(
+        "experiment",
+        sa.Column("project_id", postgresql.UUID(as_uuid=True), nullable=False),
+    )
+    op.create_foreign_key(
+        "fk_experiment_project_id",
+        "experiment",
+        "project",
+        ["project_id"],
+        ["id"],
+        ondelete="CASCADE",
+    )
+    op.create_index(
+        "idx_experiment_project_id",
+        "experiment",
+        ["project_id"],
+        unique=False,
+    )
+
+    # 3) Drop read_submission.project_id (derivable via experiment)
+    op.execute(
+        "ALTER TABLE read_submission DROP CONSTRAINT IF EXISTS read_submission_project_id_fkey"
+    )
+    op.execute(
+        "ALTER TABLE read_submission DROP CONSTRAINT IF EXISTS read_submission_project_id_fkey1"
+    )
+    op.execute("ALTER TABLE read_submission DROP COLUMN IF EXISTS project_id")
+
+    # 4) Drop experiment_submission.sample_id and project_id (derivable via experiment)
+    op.execute(
+        "ALTER TABLE experiment_submission DROP CONSTRAINT IF EXISTS experiment_submission_sample_id_fkey"
+    )
+    op.execute(
+        "ALTER TABLE experiment_submission DROP CONSTRAINT IF EXISTS experiment_submission_project_id_fkey"
+    )
+    op.execute("ALTER TABLE experiment_submission DROP COLUMN IF EXISTS sample_id")
+    op.execute("ALTER TABLE experiment_submission DROP COLUMN IF EXISTS project_id")
+
+
+def downgrade():
+    # Reverse order of upgrade
+
+    # 4) Re-add experiment_submission.sample_id and project_id
+    op.add_column(
+        "experiment_submission",
+        sa.Column("sample_id", postgresql.UUID(as_uuid=True), nullable=True),
+    )
+    op.add_column(
+        "experiment_submission",
+        sa.Column("project_id", postgresql.UUID(as_uuid=True), nullable=True),
+    )
+    op.create_foreign_key(
+        "experiment_submission_sample_id_fkey",
+        "experiment_submission",
+        "sample",
+        ["sample_id"],
+        ["id"],
+    )
+    op.create_foreign_key(
+        "experiment_submission_project_id_fkey",
+        "experiment_submission",
+        "project",
+        ["project_id"],
+        ["id"],
+    )
+
+    # 3) Re-add read_submission.project_id
+    op.add_column(
+        "read_submission",
+        sa.Column("project_id", postgresql.UUID(as_uuid=True), nullable=True),
+    )
+    op.create_foreign_key(
+        "read_submission_project_id_fkey",
+        "read_submission",
+        "project",
+        ["project_id"],
+        ["id"],
+    )
+
+    # 2) Drop experiment.project_id
+    op.drop_index("idx_experiment_project_id", table_name="experiment")
+    op.drop_constraint("fk_experiment_project_id", "experiment", type_="foreignkey")
+    op.drop_column("experiment", "project_id")
+
+    # 1) Drop sample_submission.project_id
+    op.drop_index("idx_sample_submission_project_id", table_name="sample_submission")
+    op.drop_constraint("fk_sample_submission_project_id", "sample_submission", type_="foreignkey")
+    op.drop_column("sample_submission", "project_id")