NUnit-Console version 18.0.0 and up freezes/hangs when running tests on Mac via Terminal

[Moshex]

When I run this command in Mac via Terminal
mono ../../../.nuget/packages/NUnit.ConsoleRunner/3.18.3/tools/nunit3-console.exe --labels=Before --out=TestResult.txt "--result=TestResult.xml;format=nunit2" ./Project.UITests.csproj --trace=Verbose;
The Terminal just freezes not running the tests or seeming to do anything. I am not sure what the issue is and why this started to occur but if there is something I can do to fix this that would be helpful. My project is .net 8 and I wonder if this issue has to do with that since it worked on .net 6. I am not getting an error; if I did, I would share that with you. Please let me know what more information I can provide about my project to help fix this issue.
Thank you for your help!

This is what it looks like on my end.

[CharliePoole]

@Moshex Can you say exactly which package you installed for both the runner and the VS Project loader extension? Since you don't get an error for the csproj file type, it's clear you do have the extension installed. You can also try to run the output test assembly itself rather than the proj file and see what happens.

[Moshex]

Here is a screenshot of the packages I have installed:

for the output test assembly which dll is that per this screenshot that I can test that with?

[CharliePoole]

Here is a screenshot of the packages I have installed:

That looks fine

for the output test assembly which dll is that per this screenshot that I can test that with?

None of those. You should try replacing ./Project.UITests.csproj in your command line with the path to the whatever test assembly it produces. It might be something like bin/Debug/UITests.dll. This will tell us if the problem is in the interaction between the runner and the project loader extension or in the runner itself.

[Moshex]

These are all the assemblies that are being produced which one would I use?

[CharliePoole]

That's not something I can tell you. You have a project, referenced in your command line, which builds one or more test assemblies. You know their names - or somebody in your shop does. Pick one of those assemblies. From your list, I'm guessing it's SOMETHING.UITest.dll, but that's only a guess. If there are multiple test assemblies, just pick one of them, since this is only an experiment to find out what works and what doesn't.

In your list of files, I do see both TestResult.txt and TestResult.xml, so it seems that some output is being produced. Also, there are a number of log files, created by the --trace option you used. All those files should be examined.

[Moshex]

Thank you for your help. You were right about the DLL to use, and I was able to kick off a run with it. However, it did not work, just like the other way I did the command. I will post the log files to see if there is anything from those you can help with troubleshooting the issue.

This is the command I used:

username@Moshes-MacBook-Pro-4 WellRx.UITests % mono ../../../.nuget/packages/NUnit.ConsoleRunner/3.18.3/tools/nunit3-console.exe --labels=Before  --out=TestResult.txt "--result=TestResult.xml;format=nunit2" /Users/mmoadeb/Documents/WellRX_MAUI/WellRx.UITests/bin/Debug/net8.0/WellRx.UITests.dll --trace=Verbose;
NUnit Console Runner 3.18.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Friday, November 8, 2024 8:11:03 AM

Runtime Environment
   OS Version: MacOSX 23.6.0.0 
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    /Users/username/Documents/WellRX_MAUI/WellRx.UITests/bin/Debug/net8.0/WellRx.UITests.dll

Log files:

InternalTrace.76060.log
InternalTrace.76063.log
InternalTrace.76063.WellRx.UITests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.log

Please let me know if there is anything else I could provide to help fix this issue, Thank you!

[CharliePoole]

@Moshex Thanks for the info. We should keep working with this command-line because it narrows the problem down.

The traces show that your test run is launched and uses a net8.0 agent as expected. The agent run starts as expected but then reaches a point where nothing more happens.

Your second trace file ends with an apparently unsuccessful attempt to load an unmanaged assembly. You should check to see what is causing that assembly to load and where it is. You should also look at the text output file created by your --out option because it should indicate which tests were started before the failure. If you are running on your own machine, then you may want to remove the --out option so you can see the output.

@nunit As you know, I'm actually not working on the runner for the next two weeks, so feel free to jump in!

[Moshex]

Taking out the "out" all it does is hang on the 1st test, which looks like this. Remember that my tests work just fine when running them via an IDE such as Rider or VS for Mac. I am not sure when running them via the command it just hangs like this.

 mono ../../../.nuget/packages/NUnit.ConsoleRunner/3.18.3/tools/nunit3-console.exe --labels=Before  "--result=TestResult.xml;format=nunit2" /Users/mmoadeb/Documents/WellRX_MAUI/WellRx.UITests/bin/Debug/net8.0/WellRx.UITests.dll --trace=Verbose;
NUnit Console Runner 3.18.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Friday, November 8, 2024 2:41:16 PM

Runtime Environment
   OS Version: MacOSX 23.6.0.0 
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    /Users/username/Documents/WellRX_MAUI/WellRx.UITests/bin/Debug/net8.0/WellRx.UITests.dll

=> WellRx.UITests.Features.ChangePasswordFeature(iOS,"A490A6F8-1559-4405-BC5D-33DB2C8CD6C0").TC001

[CharliePoole]

@Moshex The benefit of the output (which is also in your TestResult.txt file when you use --out) is that it tells us instantly the same info that took me reading through various trace files... i.e. one test started but it didn't finish. The trace does let us know that running that test involved loading an unmanaged dll and that the dll was not loaded but also did not throw an exception. So this is a bug in one of two places: the runner or your test. Where you can help is by identifying (1) which unmanaged dll is being loaded and (2) where it is located. If it's somewhere where the engine should be able to find it, then it's a bug in the engine.

If nobody else takes this on, I'll work on it when I return to the project after Nov 25.

[Moshex]

@CharliePoole is this a bug on my end that I can fix or do I need to wait for a bug fix?

[CharliePoole]

I don't know. It needs to be worked on to find out, i.e. working in the NUnit code, debugging, etc. Since I'm not available to do it for a few weeks, I've invited other @nunit team members to take a look. If nobody does, I'll look at it on return but right now I'm only taking time to answer questions to the extent I can each day.

[Moshex]

That sounds good, thank you so much for all your help!

[Moshex]

@CharliePoole, I'm just checking in to see if there are any updates for the fix.

[mikkelbu]

Hi @Moshex
AFAIK there are no updates as we still don't know where the problem is (or what the problem are). I don't use Mac nor Mono, so I don't think I can help you much, but I've some comments/suggestions.

• Do the problem also occur if you make a minimal reproducible example - e.g. just a single test project with a single (empty) test and no external dependencies?
• I don't have much experience with reading the log files, but if the problem is the unmanaged file - libSystem.Native - then this part of the dotnet runtime. Can you see anything if you use a debugger or something like strace to see which files are read etc.

[Moshex]

@mikkelbu
is there no way for you to test this on a mac to see if you see the same issue?