Add runtime migration creation and application #37415

thromel · 2025-12-23T07:41:45Z

Summary

Implements #37342: Allow creating and applying migrations at runtime without recompiling.

This adds support for creating and applying migrations at runtime using Roslyn compilation, enabling scenarios like .NET Aspire and containerized applications where recompilation isn't possible.

CLI Usage

# Standard update (existing behavior)
dotnet ef database update [migration]

# Create and apply a new migration in one step
dotnet ef database update MigrationName --add [--output-dir <DIR>] [--namespace <NS>] [--json]

The -o/--output-dir, -n/--namespace, and --json options require --add to be specified.

PowerShell Usage

# Standard update (existing behavior)
Update-Database [-Migration <migration>]

# Create and apply a new migration in one step
Update-Database -Migration MigrationName -Add [-OutputDir <DIR>] [-Namespace <NS>]

Architecture

Component	Purpose
`IMigrationCompiler` / `CSharpMigrationCompiler`	Internal: Roslyn-based compilation of scaffolded migrations
`IMigrationsAssembly.AddMigrations(Assembly)`	Registers dynamically compiled migrations
`MigrationsOperations.AddAndApplyMigration()`	Orchestrates scaffold → compile → register → apply workflow

Design Decisions

Extends existing services: Uses IMigrationsScaffolder for scaffolding and IMigrator for applying, adding only the new IMigrationCompiler service
AddMigrations(Assembly): Extended IMigrationsAssembly interface to accept additional assemblies containing runtime-compiled migrations
Always persists to disk: Like AddMigration, files are always saved to enable source control and future recompilation
No pending changes behavior: If no model changes are detected, applies any existing pending migrations without creating a new one
Internal compiler API: IMigrationCompiler and CSharpMigrationCompiler are in the .Internal namespace as they require design work for public API
Error handling with cleanup: If compilation or migration application fails, saved migration files are cleaned up to prevent orphans
Thread safety: MigrationsAssembly uses locking to protect against race conditions when adding migrations concurrently

Workflow

User runs: dotnet ef database update InitialCreate --add
    │
    ▼
MigrationsOperations.AddAndApplyMigration()
    │
    ├─► Check for pending model changes
    │       └─► If none: apply existing migrations, return
    │
    ├─► IMigrationsScaffolder.ScaffoldMigration() - Generate code
    │
    ├─► try {
    │       ├─► IMigrationsScaffolder.Save() - Write files to disk
    │       ├─► IMigrationCompiler.CompileMigration() - Roslyn compile
    │       ├─► IMigrationsAssembly.AddMigrations() - Register migration
    │       └─► IMigrator.Migrate() - Apply to database
    │   } catch {
    │       └─► Clean up saved files on failure
    │   }
    │
    └─► Return migration files

Robustness Features

Exception handling with cleanup: AddAndApplyMigration wraps the save-compile-register-apply chain in try-catch, deleting saved files on failure to prevent orphans
Context disposal on validation failure: PrepareForMigration ensures the DbContext is disposed if validation or service building fails
Thread-safe migration registration: MigrationsAssembly uses locking to protect shared state (migrations dictionary, model snapshot, additional assemblies list)

Limitations

Requires dynamic code generation (incompatible with NativeAOT) - marked with [RequiresDynamicCode]
C# only (no VB.NET/F# support)

Test plan

Unit tests for CSharpMigrationCompiler
Unit tests for MigrationsOperations.AddAndApplyMigration
Integration tests in RuntimeMigrationTestBase (SQLite and SQL Server implementations)
Tests for validation (empty name, invalid characters)
Tests for RemoveMigration with dynamically created migrations
All existing EFCore.Design.Tests pass
All existing EFCore.Relational.Tests pass

Fixes #37342

thromel · 2025-12-25T06:51:08Z

Note on SQL Server Integration Tests

The RuntimeMigrationSqlServerTest tests are marked with [SqlServerCondition(SqlServerCondition.IsNotAzureSql | SqlServerCondition.IsNotCI)] and are skipped in CI. This follows the same pattern used by MigrationsInfrastructureSqlServerTest.

Why these tests are skipped in CI:

They require creating fresh databases dynamically for each test to properly test the migration flow from scratch
The Helix CI environment has SQL Server available but with limited permissions configured for shared/pre-configured databases
Tests that use SqlServerTestStore.CreateInitializedAsync with dynamic database names don't work in the CI SQL Server setup

Test coverage is still maintained:

Core runtime migration logic is covered by unit tests in EFCore.Design.Tests (which run in CI)
The SQL Server integration tests run locally for developers with SQL Server configured
SQLite integration tests in EFCore.Sqlite.FunctionalTests also validate the end-to-end flow

This is consistent with how other complex migration infrastructure tests handle CI limitations.

test/EFCore.SqlServer.FunctionalTests/Migrations/RuntimeMigrationSqlServerTest.cs

src/ef/Commands/DatabaseUpdateCommand.Configure.cs

src/ef/IOperationExecutor.cs

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

src/EFCore.Design/Design/DesignTimeServiceCollectionExtensions.cs

src/EFCore.Design/Migrations/Design/IDynamicMigrationsAssembly.cs

thromel · 2025-12-31T05:05:26Z

Thank you for the thorough review @AndriySvyryd! I've addressed all your feedback:

SQL Server tests: Now using static database name "RuntimeMigrationTest" and removed EnsureDeleted() calls since CreateInitializedAsync already cleans the database.
CLI validation: Added validation that shows an error if -o or -n is used without --add.
Renamed to AddAndApplyMigration: Renamed CreateAndApplyMigration to AddAndApplyMigration and reordered parameters as suggested.
Extracted common validation: Created ValidateMigrationName() and ValidateMigrationNameNotContextName() helper methods that are shared by both AddMigration and AddAndApplyMigration.
Added EnsureMigrationsAssembly call: Now calling EnsureMigrationsAssembly in AddAndApplyMigration.
Removed unnecessary IDesignTimeModel registration: Confirmed there's no duplicate registration.
Merged IDynamicMigrationsAssembly into IMigrationsAssembly: Added AddMigrations(Assembly) method to IMigrationsAssembly and deleted IDynamicMigrationsAssembly, DynamicMigrationsAssembly, and their tests.
Replaced CompiledMigration with AddMigrations(Assembly): IMigrationCompiler.CompileMigration now returns Assembly directly, and RuntimeMigrationService uses _migrationsAssembly.AddMigrations() to register compiled migrations. Removed the CompiledMigration class.

All tests pass locally (SQLite: 26 tests, SQL Server: 7 tests, CSharpMigrationCompiler: 4 tests, MigrationsOperations: 2 tests).

src/ef/Commands/DatabaseUpdateCommand.cs

src/EFCore.Relational/Extensions/RelationalDatabaseFacadeExtensions.cs

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

src/EFCore.Design/Migrations/Design/RuntimeMigrationService.cs

test/EFCore.Design.Tests/Design/Internal/MigrationsOperationsTest.cs

src/ef/Commands/DatabaseUpdateCommand.cs

Properly restore connection state after cleaning database tables. The connection is closed after cleanup only if it wasn't already open before, preventing "connection was not closed" errors in tests that expect to open the connection themselves.

1. AddAndApplyMigration error handling: - Add try-catch around scaffold-compile-apply chain - Clean up saved files on failure to prevent orphans - Add TryDeleteFile helper for best-effort cleanup - Add AddAndApplyMigrationFailed resource string 2. Context disposal in PrepareForMigration: - Wrap context usage in try-catch - Dispose context if validation or service building fails - Prevents context leaks on validation exceptions 3. Thread safety in MigrationsAssembly: - Add lock protection around _additionalAssemblies, _migrations, and _modelSnapshot - Protect Migrations property getter, ModelSnapshot property getter, and AddMigrations method - Prevents race conditions in multi-threaded scenarios

The snapshot file may have overwritten an existing snapshot during Save(). Deleting it on failure would leave the project without a snapshot, breaking future migrations. Only delete migration and metadata files which are always newly created.

- Remove file deletion on failure (keep files for debugging) - Inline validation methods into PrepareForMigration - Remove DisableParallelization from test classes - Refactor tests to use SharedStoreFixtureBase pattern - Use NonCapturingLazyInitializer for MigrationsAssembly.Migrations - Convert to using declarations to reduce nesting - Make CleanDatabase virtual for provider overrides - Fix thread safety with lock-based ModelSnapshot caching

- Move migration name validation before context creation in AddMigration and AddAndApplyMigration to ensure proper error messages when name is empty - Use Single() instead of First() in Migration_preserves_existing_data test to avoid FirstWithoutOrderByAndFilterWarning

Replace First() with Single() to avoid FirstWithoutOrderByAndFilterWarning

Close connection before migrator.Migrate("0") call and reopen after, since the migrator manages its own connection state internally.

AndriySvyryd · 2026-01-09T00:12:00Z

Practical Risk Assessment

The race is unlikely because:

AddMigrations is only called during design-time database update --auto operations

It's not expected to be called concurrently with Migrations access

The scenario requires very specific timing

I agree that it's very unlikely, keep NonCapturingLazyInitializer and remove the lock from AddMigrations

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

test/EFCore.Relational.Specification.Tests/RuntimeMigrationTestBase.cs

AndriySvyryd · 2026-01-09T00:59:54Z

test/EFCore.Relational.Specification.Tests/RuntimeMigrationTestBase.cs

+    public void Can_scaffold_migration()
+    {
+        using var context = CreateContext();
+        CleanDatabase(context);


You should be able to call Fixture.ReseedAsync() instead.

You can also call it from IAsyncLifetime.InitializeAsync to avoid repeating the call in each method

Sorry, I forgot that ReseedAsync also creates tables. Change InitializeAsync to

{ using var context = CreateContext(); return Fixture.TestStore.CleanAsync(context, createTables: false); }

Then add the bool createTables = true parameter to TestStore.CleanAsync, SqlServerDatabaseFacadeExtensions.EnsureClean, SqliteDatabaseFacadeTestExtensions.EnsureClean and RelationalDatabaseCleaner.Clean

AndriySvyryd · 2026-01-09T01:01:04Z

test/EFCore.Relational.Specification.Tests/RuntimeMigrationTestBase.cs

+            => "RuntimeMigration";
+
+        protected override bool UsePooling
+            => false;


Why did you have to specify false here?

Re: UsePooling => false in RuntimeMigrationFixtureBase

Connection pooling is disabled because these tests dynamically alter the database schema (creating/dropping tables, applying/reverting migrations). With pooling enabled, pooled connections might hold stale schema information or cached state that conflicts with the schema changes made during migration operations.
However, if you think pooling should work fine for these tests, I can remove the override and test it.

This property controls DbContext pooling, not connection pooling, so leaving the default value (true) should be fine for these tests

I see now, if you use context pooling then the tests will get a context instance that still contains the runtime migrations assembly from the previous test, making it fail in most cases. So in this case, disabling DbContext pooling is the right choice

Per review: race condition is very unlikely since AddMigrations is only called during design-time operations, not concurrently with Migrations access.

thromel · 2026-01-09T03:01:30Z

Re: UsePooling => false in RuntimeMigrationFixtureBase

Connection pooling is disabled because these tests dynamically alter the database schema (creating/dropping tables, applying/reverting migrations). With pooling enabled, pooled connections might hold stale schema information or cached state that conflicts with the schema changes made during migration operations.

However, if you think pooling should work fine for these tests, I can remove the override and test it.

- Implement IAsyncLifetime and call Fixture.ReseedAsync() in InitializeAsync instead of manually calling CleanDatabase in each test - Use context.Database.OpenConnection/CloseConnection instead of direct connection.Open/Close calls - Move database cleanup logic to fixture's CleanAsync override - Add GetTableNamesAsync to fixtures for async cleanup

- Simplify CSharpMigrationCompiler.GetMetadataReferences to use cached references plus context assembly, removing explicit Assembly.Load calls - Remove duplicate name validation from AddMigration/AddAndApplyMigration since PrepareForMigration already validates - Remove UsePooling override (controls DbContext pooling, not connection pooling)

thromel · 2026-01-09T03:18:09Z

Addressed the remaining unresolved comments:

CSharpMigrationCompiler - Simplified GetMetadataReferences to use cached references from loaded assemblies plus context assembly. Removed explicit Assembly.Load() calls since assemblies should already be loaded by that point.
MigrationsOperations - Removed duplicate name validation from AddMigration and AddAndApplyMigration since PrepareForMigration already validates.
UsePooling - Removed the UsePooling => false override since it controls DbContext pooling, not connection pooling.

Commit: e7b6329

AndriySvyryd · 2026-01-09T23:00:44Z

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

+        if (string.IsNullOrWhiteSpace(name))
+        {
+            throw new OperationException(DesignStrings.MigrationNameRequired);
+        }


Ok, then remove this check since it's already checked before this call

Copilot

Pull request overview

This PR implements runtime migration creation and application to support scenarios like .NET Aspire and containerized applications where recompiling is not possible. It extends the existing dotnet ef database update and Update-Database commands with a new --add option that scaffolds, compiles (using Roslyn), registers, and applies a migration in one atomic operation.

Changes:

Adds IMigrationCompiler interface and CSharpMigrationCompiler implementation for runtime Roslyn-based compilation of scaffolded migrations
Extends IMigrationsAssembly with AddMigrations(Assembly) method to register dynamically compiled migrations
Adds AddAndApplyMigration operation to MigrationsOperations that orchestrates the scaffold → compile → register → apply workflow
Updates CLI and PowerShell commands to support --add, --output-dir, --namespace, and --json options with appropriate validation

Reviewed changes

Copilot reviewed 26 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`src/EFCore.Design/Migrations/Design/IMigrationCompiler.cs`	New internal interface for runtime migration compilation
`src/EFCore.Design/Migrations/Design/CSharpMigrationCompiler.cs`	Roslyn-based implementation with assembly reference caching
`src/EFCore.Relational/Migrations/IMigrationsAssembly.cs`	Adds AddMigrations method to public interface
`src/EFCore.Relational/Migrations/Internal/MigrationsAssembly.cs`	Implements dynamic migration registration with thread-safety concerns
`src/EFCore.Design/Design/Internal/MigrationsOperations.cs`	Core AddAndApplyMigration operation with error handling
`src/EFCore.Design/Design/OperationExecutor.cs`	Operation executor for AddAndApplyMigration command
`src/ef/Commands/DatabaseUpdateCommand*.cs`	CLI command extensions with validation logic
`src/EFCore.Tools/tools/EntityFrameworkCore.psm1`	PowerShell Update-Database function enhancements
`test/EFCore.Relational.Specification.Tests/RuntimeMigrationTestBase.cs`	Comprehensive test base with 20+ test scenarios
`test/EFCore..FunctionalTests/RuntimeMigrationTest.cs`	Provider-specific test implementations
Resource files (`.resx`, `.Designer.cs`)	New localized strings for errors and messages

Files not reviewed (3)

src/EFCore.Design/Properties/DesignStrings.Designer.cs: Language not supported
src/dotnet-ef/Properties/Resources.Designer.cs: Language not supported
src/ef/Properties/Resources.Designer.cs: Language not supported

src/EFCore.Relational/Migrations/Internal/MigrationsAssembly.cs

Copilot · 2026-01-11T03:47:32Z

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

+        MigrationFiles? files = null;
+        try
+        {
+            files = scaffolder.Save(_projectDir, migration, resolvedOutputDir, dryRun: false);
+
+            var compiledAssembly = compiler.CompileMigration(migration, context.GetType());
+            migrationsAssembly.AddMigrations(compiledAssembly);
+
+            migrator.Migrate(migration.MigrationId);
+
+            _reporter.WriteInformation(DesignStrings.MigrationCreatedAndApplied(migration.MigrationId));
+
+            return files;
+        }
+        catch (Exception ex)
+        {
+            throw new OperationException(
+                DesignStrings.AddAndApplyMigrationFailed(name, ex.Message), ex);
+        }
+    }


The error handling in AddAndApplyMigration does not clean up the saved migration files on failure, even though the PR description mentions "Error handling with cleanup" as a robustness feature. If compilation or migration application fails after line 311, the files remain on disk which can cause issues. The catch block should delete the files that were successfully saved to prevent orphaned migrations.

This is an intentional design decision. Files are kept on disk after failure to aid debugging - developers can inspect the generated migration code to understand what went wrong. Deleting files on failure would make troubleshooting more difficult. This follows the pattern of dotnet ef migrations add which also keeps files on failure.

Copilot · 2026-01-11T03:47:33Z

src/ef/Commands/DatabaseUpdateCommand.Configure.cs

+            if (_namespace!.HasValue())
+            {
+                throw new CommandException(Resources.NamespaceRequiresAdd);
+            }


The validation logic does not check if the --json option is used without --add. According to the PR description, the --json option requires --add to be specified, similar to --output-dir and --namespace. This validation should be added to ensure consistent behavior.

Suggested change

}

}

if (_json!.HasValue())

{

throw new CommandException(Resources.JsonRequiresAdd);

}

Fixed in commit a5b2659 - added validation that --json requires --add to be specified, consistent with --output-dir and --namespace.

Copilot · 2026-01-11T03:47:33Z

src/EFCore.Design/Design/DesignTimeServiceCollectionExtensions.cs

            .TryAdd(_ => context.GetService<IMigrationsModelDiffer>())
            .TryAdd(_ => context.GetService<IMigrator>())
+            .TryAdd(_ => context.GetService<IMigrationsAssembly>())


The order of service registration was changed: IMigrationsAssembly was moved from line 101 (before IMigrationsModelDiffer, IMigrator) to line 104 (after them). This reordering doesn't appear to have a clear justification and could potentially impact service resolution order or dependency chains. If the reordering is intentional for the new functionality, it should be explained, otherwise it should remain in its original position.

Suggested change

.TryAdd(_ => context.GetService<IMigrationsModelDiffer>())

.TryAdd(_ => context.GetService<IMigrator>())

.TryAdd(_ => context.GetService<IMigrationsAssembly>())

.TryAdd(_ => context.GetService<IMigrationsAssembly>())

.TryAdd(_ => context.GetService<IMigrationsModelDiffer>())

.TryAdd(_ => context.GetService<IMigrator>())

Fixed in commit a5b2659 - restored the original service registration order. The reordering was unintentional.

test/EFCore.Relational.Specification.Tests/RuntimeMigrationTestBase.cs

Copilot · 2026-01-11T03:47:34Z

src/EFCore.Tools/tools/EntityFrameworkCore.psm1

+    if ($Add)
+    {
+        $params += '--add'
+    }
+
+    if ($OutputDir)
+    {
+        $params += '--output-dir', $OutputDir
+    }
+
+    if ($Namespace)
+    {
+        $params += '--namespace', $Namespace
+    }


The PowerShell Update-Database function does not validate that OutputDir and Namespace parameters are only used with the Add switch. The underlying command will reject these combinations, but it would be better to validate in PowerShell and provide clearer error messages to the user before invoking the command.

Fixed in commit a5b2659 - added PowerShell-level validation that -OutputDir and -Namespace require the -Add switch.

Copilot · 2026-01-11T03:47:34Z

src/EFCore.Design/Design/Internal/MigrationsOperations.cs

+        {
+            _reporter.WriteInformation(DesignStrings.NoPendingModelChanges);
+            migrator.Migrate(null);
+            _reporter.WriteInformation(DesignStrings.Done);


When there are no pending model changes, the method returns an empty MigrationFiles object (line 298). However, the calling code in DatabaseUpdateCommand will attempt to output JSON from this empty object when --json is specified. This will result in null values for all file paths in the JSON output, which may be confusing to users. Consider returning null or a special indicator, or documenting this behavior clearly.

Suggested change

_reporter.WriteInformation(DesignStrings.Done);

_reporter.WriteInformation(DesignStrings.Done);

// Note: Returning an empty MigrationFiles instance here indicates that no migration was created.

// DatabaseUpdateCommand will serialize this instance when --json is specified, which results in null

// values for all file path properties in the JSON output.

Fixed in commit a5b2659 - added a comment documenting that an empty MigrationFiles is returned when there are no pending changes, and that JSON serialization will produce null values for all file paths.

…ions Per reviewer feedback, added a `createTables` parameter (default true) to the test store clean methods. Runtime migration tests use `createTables: false` to get an empty database without tables, allowing migrations to create them. Changes: - Add `bool createTables = true` to RelationalDatabaseCleaner.Clean() - Propagate parameter through SqliteDatabaseCleaner, SqlServerDatabaseCleaner - Add parameter to EnsureClean extension methods - Add parameter to TestStore.CleanAsync and provider implementations - Simplify RuntimeMigrationTestBase to use TestStore.CleanAsync directly - Add UsePooling => false to fixture (pooled contexts retain migration assemblies) - Remove custom CleanAsync overrides from runtime migration fixtures

- Add validation that --json requires --add in database update command - Restore original service registration order in DesignTimeServiceCollectionExtensions - Add PowerShell validation for -OutputDir/-Namespace requiring -Add - Add comment documenting empty MigrationFiles JSON behavior

Instead of adding a new custom resource JsonRequiresAdd, reuse the existing MissingConditionalOption resource which provides the same functionality. This avoids issues with T4 template regeneration in CI.

The AddAndApplyMigration tests require a database connection because Migrator.Migrate() calls _connection.Open(). Without a valid connection string, the tests fail in CI. Adding "Data Source=:memory:" provides an in-memory SQLite database that allows the migration operations to complete successfully.

The Migrator.Migrate method opens and closes the connection multiple times during migration (for CreateIfNotExists and MigrateImplementation). With Data Source=:memory:, the SQLite database is destroyed when the connection closes, which causes the migration history table to be lost between steps. Using an externally opened connection ensures EF Core won't close it during migration, keeping the in-memory database alive throughout the operation.

thromel force-pushed the feature/runtime-migrations branch from a15611a to 9a35a9b Compare December 23, 2025 20:45

AndriySvyryd added the community-contribution label Dec 23, 2025

AndriySvyryd self-assigned this Dec 23, 2025

thromel marked this pull request as ready for review December 24, 2025 06:22

thromel requested a review from a team as a code owner December 24, 2025 06:22

thromel marked this pull request as draft December 25, 2025 02:03

thromel force-pushed the feature/runtime-migrations branch from 62018f1 to 5c41f2f Compare December 25, 2025 03:51

thromel marked this pull request as ready for review December 25, 2025 08:13

thromel marked this pull request as draft December 25, 2025 16:43

thromel marked this pull request as ready for review December 25, 2025 21:07

AndriySvyryd reviewed Dec 30, 2025

View reviewed changes

test/EFCore.SqlServer.FunctionalTests/Migrations/RuntimeMigrationSqlServerTest.cs Outdated Show resolved Hide resolved

AndriySvyryd reviewed Dec 30, 2025

View reviewed changes

src/ef/Commands/DatabaseUpdateCommand.Configure.cs Show resolved Hide resolved

AndriySvyryd reviewed Dec 30, 2025

View reviewed changes

src/ef/IOperationExecutor.cs Outdated Show resolved Hide resolved

AndriySvyryd reviewed Dec 30, 2025

View reviewed changes

src/EFCore.Design/Design/Internal/MigrationsOperations.cs Show resolved Hide resolved

AndriySvyryd reviewed Dec 30, 2025

View reviewed changes

src/EFCore.Design/Design/Internal/MigrationsOperations.cs Show resolved Hide resolved

AndriySvyryd reviewed Dec 31, 2025

View reviewed changes

src/EFCore.Design/Design/DesignTimeServiceCollectionExtensions.cs Outdated Show resolved Hide resolved

AndriySvyryd reviewed Dec 31, 2025

View reviewed changes

src/EFCore.Design/Migrations/Design/IDynamicMigrationsAssembly.cs Outdated Show resolved Hide resolved

AndriySvyryd reviewed Dec 31, 2025

View reviewed changes

src/EFCore.Design/Migrations/Design/IDynamicMigrationsAssembly.cs Outdated Show resolved Hide resolved

thromel requested a review from AndriySvyryd December 31, 2025 05:21