Property/Fuzz Testing

[jamesadarich]

Goal

Make writing input testing easier, readable and more robust

How

Simplifying syntax for supplying many arguments to Test functions

Currently you'd have to pass in multiple TestCase decorators which is fine but takes time and lines often resulting in not enough tests being run.

Instead if we use generators this could ensure a load of tests get run against functions testing them with an assortment of inputs ensuring they are robust.

@TestProperties(TrueAndFalse)

@TestProperties(Numbers.random().generate(100))

@TestProperties(Strings.ofLength({ over: 42 }).generate(500))

@TestProperties(Integers.random({ between: { lowerLimit: 0, upperLimit: 42000000 } }).generate(1000))

@TestProperties(Arrays.ofLength(6).fill(() => "some string").generate(42))

@TestProperties(Objects.with(o => o.property = "something").generate(24))

Further to this we could supply fuzz style generators to ensure edge cases are caught.

@TestProperties(Numbers.edgeCases)

@TestProperties(internationalNames)

@TestProperties(internationalPhoneNumbers)

Why not TestCases?

The TestCases decorator was a great start on this idea but there are a few things we want to improve upon.

1. it takes in more than generators - we want to stick to generators for a clearer way to achieve this sort of testing
2. it can be confusing how to pass in multiple arguments, now we'd write @TestProperties(Numbers.random().generate(4), Numbers.random().generate(4)) (Might want to think about this how we can ensure generate(4) is called on both. Maybe a wrapper of some sort?)
3. it is too close to TestCase want it to be named sufficiently different to ensure no confusion
4. it draws all cases into memory immediately, we want to move to generating values when we need them at runtime

What will we provide?

• TestProperties decorator in core to enable usage of generators
• a set of predefined generators to get people started in a separate package - alsatian-property-testing
• helper functions to create custom generators

[jamesadarich]

Rules for arguments of decorator

• generators don't have to resolve to same number of arguments
• all generators given to property resolve out to a matrix of values
• all random generators must be seedable