Missing time estimation for tests that have not been executed in the last build

[thomasrebele]

I have the following scenario: some tests for a project A is flaky on the CI but passes when executed locally. As executing the whole test suite takes several hours, I wanted to execute just the flaky tests. So I made some modifications to the pom and created a pull-request (PR1). Some time later I investigated a timeout for another pull-request (PR2) and discovered that it reached timeout because it executed 2 splits instead of the usual 20 splits.

After some investigation, I came to the following conclusion: PR2 used the build of PR1 as a reference to estimate the execution times of the tests (Using build <build-number-of-PR1> as reference in the logs). However, as PR1 didn't execute most of them, the Splitter called one of the Parallelism implementations with a much shorter list of tests, resulting in a much lower number of splits.

Suggestion
Which one of the following two approaches would be better? Or are there other alternatives?

• If the time for a test could not be estimated using the last build, the Splitter should look into the earlier builds and try to get a time estimation from them.

• It might be worthwhile to consider checking the previous build of the target branch first, to avoid that one PR influences the subsequent PRs:

  parallel-test-executor-plugin/src/main/java/org/jenkinsci/plugins/parallel_test_executor/Splitter.java

  Lines 199 to 201 in 23a9285

  	TestResult result = getTestResult(project, b.getPreviousBuild(), listener);
  	if (result == null) {
  	// Look for test results from the target branch builds if this is a change request.

Related tickets / PRs:

• estimateTestsFrom"Branch" #83
• [JENKINS-46028] Use primary job as fallback #26
• Estimate test split from file source #34

Addendum
Just for the record, to limit the execution to the test that I wanted to run, I've made changes to project A's pom.xml analogous to the following:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <test>TestToInvestigate</test>
          <excludes>
            <exclude>**/*.java</exclude>
          </excludes>
          <includes><include>**/TestToInvestigate.java</include></includes>
          <!-- ... -->
        </configuration>
      </plugin>