Implement Storage Migration Protocol

[nycomp]

Overview

Implement the storage migration protocol proposed in #382. This issue tracks the implementation work required to add migration capabilities to Campus.

Proposed Protocol

See docs/migration-protocol.md for full specification.

Key decisions:

• Separate package: Migrations live in campus-admin, not campus
• Separate storage module: campus-admin/storage/ (independent module, not under campus.storage)
• Separate database: Admin uses campus_admin_db, not campus databases
• Manual deployment: Migrations run manually via SSH, NOT auto-run from Campus apps
• CLI: admin command, not campus migrate
• No Campus deploy utils: Does NOT use campus.common.devops.deploy (Campus namespace only)

---

Implementation Plan

Phase 1: campus-admin Package Structure

Create new package: campus-admin/

campus-admin/
├── __init__.py
├── main.py                    # `admin` CLI entry point
├── storage/
│   ├── __init__.py            # Separate storage module, independent from campus.storage
│   ├── migrations/
│   │   ├── __init__.py
│   │   ├── state.py
│   │   ├── runner.py
│   │   └── migrations/
│   │       ├── 001_init_assignments.py
│   │       ├── 002_init_submissions.py
│   │       └── ...
│   └── audit/
│       ├── __init__.py
│       ├── log.py             # High-volume audit writer
│       └── queries.py         # Audit query helpers
├── yapper/
│   ├── __init__.py
│   └── base.py                # Event logging for migrations
└── pyproject.toml

Key point: campus-admin/storage/ is its own storage module, completely separate from campus/storage/.

Phase 2: Core Migration Infrastructure

File: campus-admin/storage/migrations/__init__.py

• Export public API: MigrationRunner, MigrationState
• Define migration module protocol (REVISION_ID, DESCRIPTION, STORAGE_TYPE, upgrade, downgrade)

File: campus-admin/storage/migrations/state.py

• Implement MigrationState class (connects to campus_admin_db)
• init_state_table() - create _migrations table
• record_migration() - log applied migration with status="applied"
• record_failure() - log failed migration with error_message
• get_applied_migrations() - return set of applied IDs (filter by status="applied")
• mark_rollback() - set status="rolled_back"
• is_applied() - check if status="applied"

File: campus-admin/storage/migrations/runner.py

• Implement MigrationRunner class
• discover_migrations() - find and parse migration files
• upgrade(target) - apply pending migrations
• downgrade(target) - rollback to revision
• pre_flight_checks() - validate before running
• Error handling with Yapper event emission

Phase 3: Migration Files

Directory: campus-admin/storage/migrations/migrations/

• 001_init_assignments.py - Operates on campus.api.assignments
• 002_init_submissions.py - Operates on campus.api.submissions
• Each imports from campus.api.resources for type-safe access
• Each has upgrade() and downgrade()

Phase 4: Admin CLI

File: campus-admin/main.py

• migrate(target) - run forward migrations
• rollback(target) - reverse migrations
• status() - show current migration state
• Entry point: admin command

File: campus-admin/pyproject.toml

• Define CLI entry points: [project.scripts] admin = "campus_admin.main:cli"
• Dependencies: campus, campus-admin internal storage

Phase 5: Manual Deployment Process

Deployment workflow (manual, NOT automatic):

# 1. SSH into server
ssh staging.example.com

# 2. Pull latest code
cd campus-admin
git pull main

# 3. Run migrations manually
admin migrate

# 4. Verify success
admin status

No integration with Campus apps:

• Campus apps (campus.auth, campus.api) do NOT auto-run migrations
• No changes to wsgi.py or app factories
• Migrations are operational infrastructure, separate from application code

Phase 6: Audit Logging

File: campus-admin/storage/audit/log.py

• AuditWriter class with direct PostgreSQL connection
• bulk_insert(rows) - high-volume insert using executemany
• Separate from campus.storage (optimized for writes, not queries)

File: campus-admin/yapper/base.py

• Yapper event logging for migrations
• Events: migration.applied, migration.rolled_back, migration.failed

Phase 7: Testing

File: tests/unit/admin/storage/test_migrations.py (new)

• Test MigrationState CRUD operations
• Test MigrationRunner.discover_migrations()
• Test upgrade() applies migrations in order
• Test downgrade() rolls back correctly
• Test status field tracking

File: tests/integration/admin/storage/test_migration_runner.py (new)

• Test full migration cycle with real databases
• Test rollback with data verification
• Test Yapper event emission
• Test migrations operate on Campus data via resources

Manual Testing

• Run migrations on staging with production-like data
• Verify rollback works end-to-end
• Check admin CLI commands work correctly
• Verify manual deployment workflow

---

Files to Create

campus-admin/
├── __init__.py
├── main.py
├── storage/
│   ├── __init__.py
│   ├── migrations/
│   │   ├── __init__.py
│   │   ├── state.py
│   │   ├── runner.py
│   │   └── migrations/
│   │       ├── 001_init_assignments.py
│   │       ├── 002_init_submissions.py
│   │       └── ...
│   └── audit/
│       ├── __init__.py
│       ├── log.py
│       └── queries.py
├── yapper/
│   ├── __init__.py
│   └── base.py
└── pyproject.toml

Files NOT Modified (campus)

No changes to Campus app files:

• wsgi.py - NOT modified (no auto-run integration)
• campus/*/factory.py - NOT modified

---

Dependencies

• Blocked by docs: Storage Migration Protocol (issue #27) #382 merge (protocol documentation)
• Creates new: campus-admin package with independent storage
• Uses existing: campus.storage, campus.api.resources, campus.auth.resources (via imports, not inheritance)

---

Definition of Done

• campus-admin package created with storage/migrations/
• campus-admin/storage/ is independent module (not under campus.storage)
• All migration files have upgrade() and downgrade()
• admin migrate applies pending migrations
• admin rollback <revision> reverts migrations
• admin status shows migration state
• _migrations table tracks state correctly (in campus_admin_db)
• Yapper events emitted for all migration actions
• Manual deployment workflow documented
• NO auto-run integration with Campus apps
• Unit tests for core infrastructure
• Integration test for full cycle
• Documented in README or GETTING-STARTED

---

References

• Protocol Documentation
• PR #382
• Issue #27

[passionworkeer]

A good way to keep this low-risk is to split it into 3 seams: (1) MigrationState for the _migrations table only, (2) MigrationRunner that just discovers/sorts/applies revisions, (3) thin CLI wrappers. For discover_migrations(), sort by revision prefix and validate duplicate IDs early. For upgrade()/downgrade(), wrap each migration in a transaction and record state only after success, otherwise rollback and emit one failure event to Yapper with the revision id. That gives you idempotency and makes the tests much simpler.