[FEATURE] - E2E tests for the Python codebase

[SecKatie]

@mrbrandao I am thinking that we should develop E2E tests that exercise the Lola python CLI to ensure that the inputs lead to the expected output and side effects. I was thinking we could develop these e2e tests in Go and then we could point the same test suite at the new go code and use it for test driven development.

Do you think that is valuable? If so I can work on a PR to add it and we can include it as part of CI. This way when we write our PR to add the Go version we can gate the PR on the E2E testing and only merge once parity is provably achieved.

[rjeffman]

@SecKatie I'd better have the description of the tests is something like Gherkin, and use some cucumber-compatible tool to run these. In Python there's behave. I haven't look at this in Go.

[rjeffman]

It looks like Godog is the official tool for Go.

[SecKatie]

@rjeffman that's a great idea! I haven't ever used anything like that. I will do some research. It is Day of Learning after all!

[rjeffman]

@SecKatie I'm writing (well... my friend Claude is) a plan for that and will open a PR soon to start a discussion on the design.

[trevor-vaughan]

I'd like to propose the use of Ginkgo (and optionally Gomega).

It scratched my itch for good BDD in Golang (coming from a rspec-world). I've started putting together a library that will provide nice, human readable output. Hopefully will have it posted by the end of the week.

[trevor-vaughan]

Newly minted ginkgoleaf for Ginkgo formatting.

[rjeffman]

I don't like the idea of describing what to do/achieve with code. Natural language is much easier for someone not versed on the programming language and using the system.

The idea is not to only have "tests" with BDD, but use it as specification language for features and behaviors.

[trevor-vaughan]

After using Cucumber for quite some time, I found that it became excessively difficult for performing complex tests, particularly around e2e integration test suites.

I ended up switching back to pure rspec because it was overall easier to handle at scale and the human readable version could be extracted from the reports.

I'll freely admit that there may be a better way of using it so hopefully I'll discover what I was missing.

[trevor-vaughan]

I forgot to mention, we ended up having issues with contributions when using Cucumber because it was "too magic". It's not natural language, but more of a macro processor where all positions and exact wording matter.

Everyone ended up having to read the code to figure out which magic keywords were required, in what order, to create a valid test. This is another reason we simply went back to code.