Add support for QC read and update assembly pipeline manifest

[emilylm]

📌 Summary

Replaces the read_submission ENA submission flow with a new qc_read / qc_read_file / qc_read_submission flow. Previously, raw read file metadata was submitted directly to ENA. Now, the genome launcher runs QC first and posts the resulting file metadata (CRAM or FASTQ pair) back to Canopy via a new callback endpoint. Canopy creates the submission record from that QC output, and the broker claims and submits it as before.

read records and the broker API contract are unchanged.

---

🔄 Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🧹 Refactor (code change that neither fixes a bug nor adds a feature)
• 📚 Documentation update
• ⚙️ CI / build / tooling change

---

🧩 Key Changes

Schema / migration (0007_add_qc_read_tables)

• Adds qc_read, qc_read_file, qc_read_submission tables
• Extends the entity_type PostgreSQL enum with 'qc_read'
• Drops read_submission table (no data migration required)

New QC reads endpoint (POST /qc-reads)

• Genome launcher posts QC metrics + output file metadata (1 CRAM or 1 R1 + 1 R2 FASTQ pair) identified by bpa_package_id
• Creates qc_read, qc_read_file, and a draft qc_read_submission atomically
• prepared_payload stored on the submission is in the files-list format expected by the broker's to_run_xml()

Broker endpoints updated

• All ReadSubmission references replaced with QcReadSubmission throughout broker.py
• entity_type strings updated from "read" to "qc_read" in events and accession registry inserts
• On rejection (both old-style and new-style report endpoints), a new draft qc_read_submission row is created from the rejected row so the entity is immediately re-claimable; experiment_id is preserved on the new row

Update assembly pipeline manifest

• Updated the format of the manifest file generated for an assembly run, to match the file here. When a schema is available we may need to make further changes. We will also need to add in some taxonomic information at a later date.

Add base_url to reads schema and assembly manifest

• Added a base_url field to read table, which is also included in the assembly manifest.

Cleanup

• reads.py endpoint: ReadSubmission creation/update logic removed; Read CRUD is unaffected
• experiment_service.py bulk import: ReadSubmission creation removed; Read rows are still created as before
• aggregate.py: ReadSubmissionJson replaced with QcReadSubmissionJson
• read_submissions.py endpoint: no longer registered in the router (file retained but inactive)

broker-test fix

• to_run_xml() had dead code: it read data.get("files", []) but then ignored it, building a single <FILE> from flat file_name/file_format/file_checksum fields instead. Fixed to iterate the files list; raises TransformError if the list is missing or empty

---

✅ Checklist

• Tests are passing locally / in CI
• New tests have been added for new or changed functionality
• Self-review completed
• Documentation updated (if needed)
• No sensitive data or secrets included

---

🔍 Review Notes

• The read_submission table is dropped with no data migration.
• qc_read_submission.experiment_id is a denormalised column (the canonical FK is qc_read_id). It is populated on creation via the QC callback and preserved when a new draft is created after rejection.
• File type fastq_r1 / fastq_r2 is normalised to "fastq" in the broker prepared_payload (ENA does not distinguish R1/R2 in filetype).
• The read_submissions.py endpoint file still exists on disk but is no longer wired into the router. It can be deleted in a follow-up cleanup PR once the team confirms nothing external depends on it.
• Two pre-existing test failures in test_endpoints_broker_contract.py (sample prerequisites assertion, validation override merge) are unrelated to this PR and were failing on main before this branch.

---

📎 Related Issues / Tickets

• Closes # AP-350, AP-395
• Related to #