Some guidance on running the tests please

[Roaders]

Hi

I am at the stage where we're testing our desktop agent against the conformance tests. At the moment we have several failures and I guess there will be more problems in the future as we fix issues. What is the best way to get help with this? Is it ok to raise issues here asking for help with this?

At the moment I have the following issues and questions:

• At the moment when the tests are running we have app resolver UIs opening in the root container (if our agent gets an intent request with no app it opens the UI in its own window at the moment). What should happen in this case? I assume if the iframe ui is implemented the ui would open in the conformance test window. Should we be selecting apps in the resolver UI that opens or is this just a failure case as our agent does not open an iframe UI?
• No more tests open after progress gets to about 82%. Any suggestions why this might be the case (I understand it will be hard to debug this remotely!) The last test that runs is:

(RaiseIntentFailTargetedAppResolve4) Should throw an IntentDeliveryFailed error when raising intent with targeted app intent-i, context 'testContextY', intent 'sharedTestingIntent2'

• We get quite a few failures:
  Error: App didn't return close context within .5 secs
  What does this mean?

[kriswest]

@robmoffat is your best bet for some questions.

At the moment when the tests are running we have app resolver UIs opening in the root container (if our agent gets an intent request with no app it opens the UI in its own window at the moment). What should happen in this case? I assume if the iframe ui is implemented the ui would open in the conformance test window. Should we be selecting apps in the resolver UI that opens or is this just a failure case as our agent does not open an iframe UI?

This shouldn't happen unless you are running the manual tests. The tests are constructed such that the automated ones should never pop up the resolver as there is either only a single option available (starting an instance of a specific appId) or because the call was targeted at a specific instance. See https://fdc3.finos.org/docs/api/spec#intent-resolution and https://fdc3.finos.org/docs/api/ref/DesktopAgent#raiseintentfor confirmation that the resolver shouldn't show up if there is only a single option (intent was unambiguous, in the language used in the standard). The tests assume that the test apps are the only ones available (makes it easier to manage the complex set of different apps needed in all the intent tests - although they do also (mostly) use made-up context types to avoid conflicts.

You are also in control of whether you show your own resolver somewhere or use an injected one in the window. You do so by sending your raiseIntentResponse containing a RaiseIntentNeedsResolutionResponsePayload if you want the frame to show the resolver OR you just show your own resolver and then send a RaiseIntentSuccessResponsePayload when the user is done with it. The conformance tests don't know which you are doing - and it only matters during the manual cases where its supposed to show up. If it happens during the automated cases then something else is wrong.

No more tests open after progress gets to about 82%. Any suggestions why this might be the case (I understand it will be hard to debug this remotely!) The last test that runs is:
(RaiseIntentFailTargetedAppResolve4) Should throw an IntentDeliveryFailed error when raising intent with targeted app intent-i, context 'testContextY', intent 'sharedTestingIntent2'

That test has a 121 second timeout as the app never adds the intent listener that is being waited for. If your implementation uses a longer timeout (the standard doesn't specify one, only that there should be one) then extend NoListenerTimeout to the value you use (the tests will add 1 second to that, but feel free to extend longer).

We get quite a few failures:
Error: App didn't return close context within .5 secs
What does this mean?

The test framework tries to close apps using window.close, triggered via an intent or broadcast (I forget which) and waits for confirmation from the app that it will indeed close. Rob has changed this at some point and introduced a timeout (not on the main branch): https://github.com/finos/FDC3-conformance-framework/blame/61c85450821305b31b8c57ca57703a429e43d185/src/test/v2.0/fdc3-2_0-utils.ts#L34-L38

typical usage: https://github.com/finos/FDC3-conformance-framework/blob/2_0_fdc3_for_the_web/src/test/v2.0/fdc3-2_0-utils.ts/#L10-L11
typical handling (with window.close() call): https://github.com/finos/FDC3-conformance-framework/blob/2_0_fdc3_for_the_web/src/mock/v2.0/mock-functions.ts/#L17-L32

I was getting random failures of that too - possibly due to chrome throttling tabs that weren't selected. Went away when I bumped it to 2 seconds (in teh first code ref above).

However, there is an additional problem in that there is no good way to close apps hosted as iframes in a parent window - window.close() does nothing. I mentioned this at the FDC3 for Web implementors meeting the other day I think - it feels like we need an fdc3.close() function for apps to request that the DA close them...

[Roaders]

Many thanks again @kriswest

I certainly need to change our app resolution to not appear if there is only 1 option. I can also try changing the timeout to change the inconsistent close context issue. All of the apps are opening in new windows rather than iframes so they should all close correctly.