DRAFT: DB Optimizations

[mjiori]

This PR is a draft. There are known issues and tests may not pass.

This PR provides a combination of DB migrations and code changes to help address the rising size and cost of the Registry RDS database.

• column removed: created_at
• column removed: updated_at
• premis_events.event_type has been fully converted from a varchar type to an int type. A new lookup table has been added to match ints to strings, so we don’t lose any data.
• premis_events.object
• premis_events.agent
• storage URLs

The migrations are separated simply to make them easier to run on the large production database, and also in the service of separation of concerns.

Numerous changes were necessary in Registry and Preservation Services to accommodate these database changes.

Manual testing:

• Ingest bag
• Restore bag
• Delete bag
• View events (type, object, agent)

[diamondap]

This looks good so far. One question: in the migration 013_shrink_db_size.sql, you add three lookup tables for event type, agent and object, and you create foreign key constraints in premis_events pointing to the lookup tables. Where/when do you actually populate the lookup tables? If the lookup tables aren't populated, it seems the migration will fail as the foreign keys aren't there.

[mjiori]

This looks good so far. One question: in the migration 013_shrink_db_size.sql, you add three lookup tables for event type, agent and object, and you create foreign key constraints in premis_events pointing to the lookup tables. Where/when do you actually populate the lookup tables? If the lookup tables aren't populated, it seems the migration will fail as the foreign keys aren't there.

Correct, I have not added those INSERT statements to populate the lookup tables just yet. Will do.