question about letting cake script to keep going

[minoseah629]

How do i make my cake script keep going if a step has an exit code other than zero?

I see there is a ToolSetting class that has HandleExitCode in it.

In my cake script, I am using xunit2 alias and settings.

But I want to know if there is a way to get a count of the failed tests and respond based on that number.

Or read in content from console?

[pascalberger]

We currently don't return the number of failing tests through the Cake API. Easiest option is probably to parse the output file yourself (similar to what has been done here for NUnit #2772 (comment)).

[devlead]

As @pascalberger writes, specifying that a test report should be created and parsing it is probably the best option.

Func<bool> HandleExitCode property could indeed be used to handle the result instead of the default exit code handling by creating a helper method.

It could look something like below

Task("Run-Unit-Tests")
    .Does(context => {
        FilePath xmlReportPath = context.MakeAbsolute(context.File("./artifacts/TestReport.xml"));
        XUnit2(
            "MyTests/bin/Debug/net472/MyTests.dll",
            new XUnit2Settings {
                XmlReport = true,
                ReportName =  xmlReportPath.GetFilenameWithoutExtension().FullPath,
                OutputDirectory = xmlReportPath.GetDirectory().FullPath,
                HandleExitCode = code => context.HandleXunitXmlResult(code, xmlReportPath)
            }
        );
    });

This will create a report in the artifacts folder called TestReport.xml, HandleXunitXmlResult extension method will then decide if exit code should be handled and if it should fail the build or not based on report information.

HandleXunitXmlResult could look something like below

public static bool HandleXunitXmlResult(this ICakeContext context, int exitCode, FilePath testResultsPath)
{
    // Exit code indicates success
    if (exitCode == 0)
    {
        return true;
    }

    if (!context.FileExists(testResultsPath))
    {
        throw new FileNotFoundException("Test report not found.", testResultsPath.FullPath);
    }

    var xml = System.Xml.Linq.XDocument.Load(testResultsPath.FullPath);

    var testCounters = (
        from testRun in xml.Elements("assemblies")
        from resultSummary in testRun.Elements("assembly")
        from counters in resultSummary.Elements("collection")
        select new {
            Total = counters.GetElementAttributeLongValue("total"),
            Passed = counters.GetElementAttributeLongValue("passed"),
            Failed = counters.GetElementAttributeLongValue("failed"),
            Skipped = counters.GetElementAttributeLongValue("skipped")
        }
    ).FirstOrDefault();

    if (testCounters == null)
    {
        throw new Exception($"Failed to parse test xml file {testResultsPath}.");
    }

    // allow one error
    if (testCounters.Failed < 2)
    {
        context.Information("Less than 2 ({0}) failures \"ok\"", testCounters.Failed);
        return true;
    }

    return false;
}

it parses the XML and the counters within it, GetElementAttributeLongValue is just a helper method that parses attribute string value to a long.

public static long GetElementAttributeLongValue(this System.Xml.Linq.XElement element, string key)
    => element
            .Attributes(key)
            .Select(
                value=>long.TryParse(
                value.Value,
                System.Globalization.NumberStyles.Integer,
                System.Globalization.CultureInfo.InvariantCulture,
                out var longValue
                ) ? longValue : null as Nullable<long>
            )
            .FirstOrDefault() ?? -1L;

resulting in that the build will continue if there are in this case less than 2 errors

So something similar to the above would let you make decisions on something different from XUnit default behavior for failed tests.

[minoseah629]

Hello @devlead,
I am trying to do this now.

Before I asked this question, my cake script executed my test projects run in parallel.

But for some reason, when it tries to compile the cake script, it tells me 'XUnit2Settings' does not contain a definition for 'HandleExitCode'

Here are relevant portions of my cake script since I know other parts work fine.

#addin nuget:?package=Cake.Npm&version=0.17.0
#addin nuget:?package=Cake.DoInDirectory&version=3.3.0

#tool "nuget:?package=xunit.runner.console&version=2.4.1"
#tool "nuget:?package=OctopusTools&version=7.4.4"

using Cake.Common.Tools.XUnit;

var xunitUnitSettings = new XUnit2Settings
{
    MaxThreads = 16,
    XmlReport = true, 
    HandleExitCode = code => context.HandleXunitXmlResult(code, xmlReportPath) 
    //tried here, but doesnt seem like the correct spot since i need each project 
    //to have a slightly different setting for each dll.
};

var unitTests = new[] { webApiString, serviceString, dataInterfaceString, mvcString, unitTestString };

Task("Run Unit Tests In Parallel")
    .Does((context) =>
    {
        Information("Starting Unit Test Executions");
        RunUnitTestsInParallel(unitTests, context);
        Information("Finished Unit Test Executions");
    });

public void RunUnitTestsInParallel(string[] filePaths, ICakeContext context)
{
    var actions = new List<Action>();
    foreach (var item in filePaths)
    {
        actions.Add(() =>
        {
            var split = item.Split('/');
            var file = split[split.Length - 1];
            var project = file.Substring(0, file.LastIndexOf('.'));
            Information("Starting Unit Test Executions: " + project);
            FilePath xmlReportPath = context.MakeAbsolute(context.File(project)); //not sure about this part yet?
            var settings = xunitUnitSettings;
            settings.HandleExitCode = code => context.HandleXunitXmlResult(code, xmlReportPath);
            /*var xunitUnitSettings1 = new XUnit2Settings
            {
                MaxThreads = 16,
                XmlReport = true,
                HandleExitCode = code => context.HandleXunitXmlResult(code, xmlReportPath)
            };
            FilePath xmlReportPath = context.MakeAbsolute(context.File(project));*/
                        
            XUnit2(item, xunitUnitSettings1);
            Information("Finished Unit Test Executions: " + project);
        });
    }

    var options = new ParallelOptions
    {
        MaxDegreeOfParallelism = filePaths.Length
    };

    Parallel.Invoke(options, actions.ToArray());
}

[augustoproiete]

@minoseah629 The HandleExitCode property is a new feature that was introduced with Cake v1.0.0-rc0001. I suspect you are using an older version of the Cake runner. Try installing the latest version (v1.0.0-rc0003 as of this writing).

• Docs on the different Cake Runners

[minoseah629]

Thanks. 🤦sorry didn’t read docs.

[minoseah629]

@augustoproiete and @devlead, I finally implemented it. Thank you. I finally got it to go after a couple hours of troubleshooting.

=== TEST EXECUTION SUMMARY ===
UnitTest Total: 76, Errors: 0, Failed: 4, Skipped: 0, Time: 6.589s

Less than 2 (0) failures "ok"
Finished Unit Test Executions: UnitTest
Finished Unit Test Executions

[augustoproiete]

@minoseah629 Good to hear!

Feel free to post the final solution in the "Show & Tell" if you like.