Dev server doesn't release port when terminated via SIGTERM on Linux

[tmat]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

After dotnet-watch restarts web server on Linux by sending SIGTERM the new instance fails to start because the port has not been released.

A couple of repros:

1. Failing dotnet-watch test:
   Enable test on Linux #50951

2. Customer reported dotnet watch run completely broken for F# #44908 (comment):
   Unzip FSWeb.zip and run dotnet watch --no-hot-reload --verbose from the project directory.

The following customer reported output is from earlier version of dotnet-watch, but the issue is the same:

    ~/repos/dotnetwatchtest  dotnet watch run --no-hot-reload --verbose                                                                                                              ✔ 
dotnet watch ⌚ Working directory: '/home/oliver/repos/dotnetwatchtest'
dotnet watch ⌚ Hot Reload disabled by command line switch.
dotnet watch ⌚ Evaluating dotnet-watch file set.
dotnet watch ⌚ Running MSBuild target 'GenerateWatchList' on '/home/oliver/repos/dotnetwatchtest/dotnetwatchtest.fsproj'
dotnet watch 🚀 Launched '/usr/share/dotnet/dotnet' with arguments 'msbuild /restore /nologo /v:m /home/oliver/repos/dotnetwatchtest/dotnetwatchtest.fsproj /t:GenerateWatchList --property:NuGetInteractive=true /p:_DotNetWatchListFile=/tmp/tmpG7p55R.tmp /p:DotNetWatchBuild=true /p:DesignTimeBuild=true /p:CustomAfterMicrosoftCommonTargets=/usr/share/dotnet/sdk/10.0.100-preview.7.25380.108/DotnetTools/dotnet-watch/10.0.100-preview.7.25380.108/tools/net10.0/any/DotNetWatch.targets /p:CustomAfterMicrosoftCommonCrossTargetingTargets=/usr/share/dotnet/sdk/10.0.100-preview.7.25380.108/DotnetTools/dotnet-watch/10.0.100-preview.7.25380.108/tools/net10.0/any/DotNetWatch.targets': process id 724385
dotnet watch ⌚ Process id 724385 ran for 553ms and exited with exit code 0.
dotnet watch 🔨   Determining projects to restore...
dotnet watch 🔨   All projects are up-to-date for restore.
dotnet watch ⌚ Watching 2 file(s) for changes
dotnet watch ⌚ Configuring the app to use browser-refresh middleware
dotnet watch ⌚ dotnet-watch is configured to launch a browser on ASP.NET Core application startup.
dotnet watch ⌚ Refresh server running at wss://localhost:40309,ws://localhost:40133.
dotnet watch 🚀 Launched '/usr/share/dotnet/dotnet' with arguments 'run': process id 724479
Building...
...
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BlazorWasmHotReloadMiddleware[0]
      Middleware loaded
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserScriptMiddleware[0]
      Middleware loaded. Script /_framework/aspnetcore-browser-refresh.js (16091 B).
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserScriptMiddleware[0]
      Middleware loaded. Script /_framework/blazor-hotreload.js (776 B).
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserRefreshMiddleware[0]
      Middleware loaded: DOTNET_MODIFIABLE_ASSEMBLIES=(null), __ASPNETCORE_BROWSER_TOOLS=true
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5172
dotnet watch ⌚ Launching browser: http://localhost:5172 
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/oliver/repos/dotnetwatchtest
Opening in existing browser session.
dotnet watch ⌚ [FW] Updated '/home/oliver/repos/dotnetwatchtest/Program.fs'.
dotnet watch ⌚ [FW] Updated '/home/oliver/repos/dotnetwatchtest/Program.fs'.
dotnet watch ⌚ Terminating process 724479 (SIGTERM).
dotnet watch ⌚ Terminating process 724479 (SIGKILL).
dotnet watch ⌚ Waiting for process 724479 to exit (1).
dotnet watch ⌚ Process id 724479 ran for 17112ms and exited with exit code 137.
dotnet watch ❌ Exited with error code 137
dotnet watch ⌚ File changed: /home/oliver/repos/dotnetwatchtest/Program.fs
dotnet watch ⌚ Modifying command to use --no-restore
dotnet watch ⌚ dotnet-watch is configured to launch a browser on ASP.NET Core application startup.
dotnet watch 🚀 Launched '/usr/share/dotnet/dotnet' with arguments 'run --no-restore': process id 724850
Building...
...
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BlazorWasmHotReloadMiddleware[0]
      Middleware loaded
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserScriptMiddleware[0]
      Middleware loaded. Script /_framework/aspnetcore-browser-refresh.js (16091 B).
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserScriptMiddleware[0]
      Middleware loaded. Script /_framework/blazor-hotreload.js (776 B).
dbug: Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserRefreshMiddleware[0]
      Middleware loaded: DOTNET_MODIFIABLE_ASSEMBLIES=(null), __ASPNETCORE_BROWSER_TOOLS=true
fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.IO.IOException: Failed to bind to address http://127.0.0.1:5172: address already in use.
       ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
       ---> System.Net.Sockets.SocketException (98): Address already in use
         at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
         at System.Net.Sockets.Socket.Bind(EndPoint localEP)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
Unhandled exception. System.IO.IOException: Failed to bind to address http://127.0.0.1:5172: address already in use.
 ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
 ---> System.Net.Sockets.SocketException (98): Address already in use
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.main(String[] args) in /home/oliver/repos/dotnetwatchtest/Program.fs:line 12
dotnet watch ⌚ Process id 724850 ran for 1760ms and exited with exit code 134.
dotnet watch ❌ Exited with error code 134
dotnet watch ⏳ Waiting for a file to change before restarting ...
dotnet watch 🛑 Shutdown requested. Press Ctrl+C again to force exit.
dotnet watch ⌚ Evaluating dotnet-watch file set.

Related:

dotnet/aspnetcore#62528

[danroth27]

@danmoseley @joperezr Which area should this one be assigned to? Can we get this prioritized for .NET 10 as part of the Hot Reload related investments?

[javiercn]

This is not related to the blazor dev server, untagging it.

[maraf]

Possibly related to dotnet/docs#46226 and aspnet/Announcements#526

[CharlieDigital]

Also encountered this behavior on macOS.

[joperezr]

@MackinnonBuck can we check to see if this is a new regression in 10? If so, we should double check and make sure it is not related to the obsoletion of WebHostbuilder as suggested above.

[joperezr]

Setting milestone as RC2 to investigate by RC2.

[MackinnonBuck]

TL;DR: This seems to me like a bug in dotnet run.

Following are some findings from my investigation.

Terminating an ASP.NET Core application

ASP.NET Core applications appear to correctly handle the SIGTERM signal both in .NET 9 and .NET 10:

• The dotnet run process and application process both terminate
• The port is released

This is demonstrated below:

.NET 9

dotnet run output:

Application process termination:

.NET 10

dotnet run output:

Application process termination:

NOTE: ASP.NET Core has already reacted to the runtime breaking change in dotnet/aspnetcore#62019, which is probably why we're seeing this work correctly.

dotnet watch behavior

However, dotnet watch does not send SIGTERM to the application process. Instead, it sends the signal to the wrapper dotnet run process. You can see this in the following screenshots:

Processes launched by dotnet watch:

dotnet watch output:

So, if we now run the original experiment but terminate the dotnet run process instead of the application process, we now observe a difference between .NET 9 and .NET 10:

.NET 9

.NET 10

The ".NET 10" output above indicates that the application process is not getting terminated.

If we repeat the same experiment using the .NET 10 SDK against a .NET 9 ASP.NET Core application, we still observe that the application process gets orphaned. This further suggests that it's not an ASP.NET Core issue.

Recommended next steps

My guess would be that one of the following needs to happen:

1. dotnet watch needs to terminate the application process instead of terminating the dotnet run process
2. dotnet run needs to be updated to handle the SIGTERM signal

cc @tmat - based on these findings, I believe this issue should be transferred to https://github.com/dotnet/sdk

[tmat]

@MackinnonBuck thanks for the analysis. I did not realize there is a wrapper process in between.

[tmat]

@baronfel Seems like dotnet run should handle SIGTERM and forward it to the target process. Agreed?