Using AET for web endpoints contract check

[malaskowski]

Description

Writing web endpoint contract tests is not complicated but requires some work like establishing the template (contract), running the tests, creating report. Test template should be maintained and when the API changes, also the contract should be updated (usually manually).

Motivation and Context

AET could speed up things with testing endpoints contracts thanks to its easy-tests-maintenance nature.
The idea here is to create a special AET module that will be able to check the web endpoint and state if it meets the contract or not.
But AET wouldn't require passing endpoint template directly but could establish it automatically with the first run of API check (the sam way AET setup the pattern for the screenshot comparison).

What I'm exactly thinking of is to make possible to pass to AET list of endpoints URLs and execute contract-check. The result will be a report (maybe similar to the existing AET report) that says which endpoint responses changed and which didn't. If the AET execute call for the first time to the endpoint, it saves the response as the template/contract for the future usage. If the response didn't change* url will be presented as not changed on the report, otherwise it will be presented in the similar split view as e.g.
source comparison. If the change is desired user may accept the change (similar as how it is working in AET patterns now) and the new response will be used in the next checks as the contract.

_{*) What does mean that response did or did not change is the subject to define here, e.g. maybe new or missing field. Probably values of the response should not be taken into account.}

What needs to be done

There are couple of ways how it could be achieved. Some part of the functionality described above is very similar of usage of Source comparison but it is not smart enough for that case (because of the nature of the JSON type which will be the most common case to check) e.g.:

• probably values of the response fields should be ignored, only the template should be checked,
• the status code is important here,
• there should be more request methods than GET supported to check,
• order of fields in the JSON response could differ between requests but shouldn't be treated as a change

Example usage

<suite name="web-check" ... >
   <endpointCheck name="my-test" endpoint="https://some-rest-api.com">
      <url href="/api/books" />
      <url href="/api/books/123" />
      <url href="/api/books/123/authors" />
   </endpointCheck >
</suite>

And https://some-rest-api.com/api/books/123 returns:

{
  "id": "123",
  "title": "AET are awesome",
  "isbn13": "9781592432172",
  "authors": [
    "John Doe",
    "Jim Smith"
  ],
  "pageCount": 1,
  "categories": ["Automated Testing"],
  "imageLinks": {
    "thumbnail": "http://some-dam.com/books/content?id=123&img=1&zoom=1&edge=curl"
  },
  "language": "en"
}

Running such suite for the first time in AET will:

• create patterns for those 3 endpoints
• produce report with no changes noticed

Lets now say that endpoint https://some-rest-api.com/api/books/123 changed and returns now:

{
  "id": "123",
  "title": "AET are awesome",
  "isbn13": "9781592432172",
  "authors": [
    "John Doe",
    "Jim Smith"
  ],
  "pageCount": 5,
  "imageLinks": {
    "thumbnail": "http://some-dam.com/books/content?id=123&img=1&zoom=1&edge=curl"
  },
  "ratingsCount": 4,
}

Running suite once again will notice change in /api/books/123 endpoint and produce report with information:

• changes detected on 1 of 3 endpoints
• detailed info of /api/books/123 would be:
  • language property missing
  • categories property missing
  • new property ratingsCount

Change of the value at pageCount (1 -> 5) will not be detected.

Extensions and ideas for the future

• Example extension would be some kind of check-modifiers like ignore new fields or ignore field XXX etc. - idea similar to the existing filters in AET.
• This feature could be executed periodically and it is natural idea to record the results, so that we have some kind of web api availability history.
• This simple example shows only checking GET requests, there should be option to check other request methods.