Tests are always run exhaustively

[jyrkka]

If I have understood correctly, looking at the CTL XML-file I see there should be possibility to run tests non-exhaustively. When running test via web application should there be a some form which would ask me if I want to test each Layer/CRS combination?

ets-wms13/src/main/scripts/ctl/main.xml

Line 271 in dc7025c

<xsl:variable name="exhaustive">

And when running test via rest interface, 'exhaustive' variable value is set to 'no', but despite that the test are still run exhaustively.

ets-wms13/src/main/scripts/ctl/main-auto.xml

Line 155 in dc7025c

<xsl:variable name="exhaustive">no</xsl:variable>

Have I misunderstood something or is this expected behviour?

[dstenger]

Thank you for reporting.

Following part of the code is skipped

ets-wms13/src/main/scripts/ctl/main.xml

Line 271 in dc7025c

<xsl:variable name="exhaustive">

when running it as a servlet.
This is evaluated by the method te:isWeb().

Thus, this variable has no function when using it with the Web Browser Interface or the REST Interface.
Using the CLI will probably consider this variable.

Is there a specific reason you need this functionality?

[jyrkka]

The reason why I would need that is because our system contains thousands of different supported CRS and the time to run the tests is over half an hour. And also at the end of the test suite at the moment i get OutOfMemory error when it tries to create the final report. The OOM can be surely fixed somehow but, when running these test as a part of a CI-pipe, i would like to run these test non-exhaustively also.

I also noticed this:

ets-wms13/src/main/scripts/ctl/getmap.xml

Line 75 in dc7025c

<xsl:if test="$exhaustive != 'exhaustive'">

Should that condition be other way around? That is the main reason why test are taking so long when the tests are run via rest.
As I said in the previous message, the 'exhaustive' variable value is set to 'no' in the main-auto.xml. And with that if condition, suite will call 'each-crs' test`

[dstenger]

I agree that this line might introduce a bug.
However, in the recent years we did not actively use or modify this parameter/ functionality.
So, it seems there should be a complete rework regarding the exhaustive parameter.

Did you set up TEAM Engine locally? If that is the case, can you please try to modify this line and check if it solves your problem?

[jyrkka]

Yes, I tried it with local TEAM Engine installation and changing that line solved the problem.