My struggles with many DependsOn

[Artmos]

Hey, first of all great work on TUnit so far!

I've been trying out TUnit for a new API project where I'm using it for doing integration tests and I wanted to share my experience. I didn't want to create an issue and thought it would be better to create a discussion, as I don't know if my struggles are caused by one or more issues or some limitations to the platform and how I'm trying to structure my test project.

Platform:

• Windows 11
• Rider 2024.3.5
• TUnit 0.11.0
• Verify.Http 6.4.1
• Verify.Tunit 28.10.1

Most of my tests use an HttpClient to call a local endpoint and then using Verify on the response.

Tests are grouped in classes like, SomethingCreateTestsClass : AbstractCreateTestsClass where the class either inherits or overrides tests from the base class. The base class has a Create test and a Get test that depends on the Create test. This structure is repeated for Update, Delete etc.

The whole project is comprised of these classes that depend on the class before it so UsersCreateTestsClass depends on OrganizationsCreateTestsClass, ProductsCreateTestsClass depends on UsersCreateTestsClass etc. Some of these classes use a MethodDataSource on the Class with IEnumerable<Func<T>>.

Simplified structure:

1. AuthenticationTests * 3
2. OrganizationsCreateTests
3. UsersCreateTests * 3
4. OrganizationsUpdateTests
5. UsersUpdateTests * 3
6. ProductsCreateTests * 9
7. ProductsUpdateTests * 9
8. ProductsPatchTests * 9
9. ProductsDeleteTests * 9
10. UsersDeleteTests * 3
11. OrganizationsDeleteTests

So this all results in 100+ tests with a lot of dependencies and this is where things start to break. I would run into situations where later tests become really slow or hang, and the editor will eat up more and more CPU and RAM until it freezes completely. It would also cause failures to not propagate and so the test session would hang again.

I think many of these issues happen because DependsOn doesn't put tests into a Pending state where it executes them in order, but instead puts them into the Running state. So if a Delete test runs it would put all dependencies up the chain into Running. This also means an Assembly level ParallelLimit will break the test session, because dependencies never resolve.

At first I was able to resolve these issues by using ParallelGroups and limiting the amount of DependsOn. However as the project grew this wasn't enough and I needed to use NotInParallel(Order = x) in addition to the ParallelGroups and removing more DependsOn.

My ideal setup for a project like this would be to have ParallelGroups that can depend on other ParallelGroups and then within these groups I could have tests that depend on other tests like it works now. ParallelGroups with an Order property could probably also be useful.

I hope this is helpful.

[thomhurst]

I think many of these issues happen because DependsOn doesn't put tests into a Pending state where it executes them in order, but instead puts them into the Running state. So if a Delete test runs it would put all dependencies up the chain into Running. This also means an Assembly level ParallelLimit will break the test session, because dependencies never resolve.

I think you're probably right. Tests get blocked by a semaphore lock. I would've thought in those scenarios the threads would realise they're inactive, and release back to the thread pool to process other tests that aren't sitting behind a lock.

This might be a tricky one to sort. Open to any ideas 😅

[Artmos]

So I haven't really looked much at the code yet. But conceptuelly DependsOn chains could probably be turned into NotInParallel with the order set and a unique contraint key for each dependendy chain. Not sure how that would work with multiple DependsOn for the same test, or with shared dependencies, but since NotInParallel can take an array of contraint keys, maybe functionality already exists for these cases.

[thomhurst]

ParallelGroups with an Order property could probably also be useful

This could probably be a quick one if it'd help unblock you somewhat

[Artmos]

It would definitely help clean up my test project, and might unblock further progress for me.

[Artmos]

@thomhurst

I've been spending some time with release 0.12.17 and changing the arguments of my tests.

First of all the ParallelGroup with Order seems to work great, I can get rid of all my NotInParallel.

However I tried gradually removing my ParallelGroups and just using DependsOn and at some point the test discovery breaks. It's at about 10 levels of dependency and in my case around 70 tests that are no longer part of a ParallelGroup

[Artmos]

I tested v0.13.0 and now I'm back to the original behaviour from v0.12.17 where the .NET Host process just eats up memory

I recreated the problem in a new test project and the problem seems to be one of scale, as I add more tests using a MethodDataSource at some point it hangs. If the data source only has 3 elements, my tests work, but more than 3 and it hangs. If I just have the MethodDataSource on some of the classes it also works, but too many and it hangs.

Here are my tests:

public abstract class BaseClass
{
    [Test]
    public async Task Test1()
    {
        await Task.Delay(TimeSpan.FromSeconds(1));
    }

    [Test]
    [DependsOn(nameof(Test1))]
    public async Task Test2()
    {
        await Task.Delay(TimeSpan.FromSeconds(1));
    }
}

public class TestsDataSource
{
    public static IEnumerable<Func<string>> Data()
    {
        yield return () => "Data1";
        yield return () => "Data2";
        yield return () => "Data3";
        yield return () => "Data4";
        yield return () => "Data5";
        yield return () => "Data6";
        yield return () => "Data7";
        yield return () => "Data8";
        yield return () => "Data9";
    }
}

[InheritsTests]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class Tests(string data) : BaseClass;

[InheritsTests]
[DependsOn<Tests>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class ReadTests(string data) : BaseClass;

[InheritsTests]
[DependsOn<ReadTests>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class UpdateTests(string data) : BaseClass;

[InheritsTests]
[DependsOn<UpdateTests>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class DeleteTests(string data) : BaseClass;

[InheritsTests]
[DependsOn<DeleteTests>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class CreateTests2(string data) : BaseClass;

[InheritsTests]
[DependsOn<CreateTests2>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class ReadTests2(string data) : BaseClass;

[InheritsTests]
[DependsOn<ReadTests2>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class UpdateTests2(string data) : BaseClass;

[InheritsTests]
[DependsOn<UpdateTests2>]
[MethodDataSource<TestsDataSource>(nameof(TestsDataSource.Data))]
public class DeleteTest2(string data) : BaseClass;

[thomhurst]

Thanks! That repro was useful. I THINK (famous last words) I've fixed it: #1872

[thomhurst]

Version 0.13.3 is out

[Artmos]

Version 0.13.3 is out

Success!

Things are working great now, I could get rid of my ParallelGroup usages, do some more advanced stuff with ProceedOnFailure AND it's super fast now.

[thomhurst]

We got there in the end. Sorry that took a while haha but super glad it's working 😁

[Artmos]

Things are working great in v0.13.3.

@thomhurst Thanks a lot for the quick fixes. I'll continue using TUnit as the project grows and see how things are going, but so far I'm really happy with the results.