simulator: allow failure assertions

[alpaylan]

• add creating the same table two times to the list of checked properties as a failure property example

[pereman2]

why is this a different property? I imagined under create table we would have a probability, though small one, to try to create the same table.

[pereman2]

I think this flat frequency thing will have not scale when have have a bunch of properties because we would have to scale the percentages and the manually change them. I like the approach of hierarchy where for example: write 50%, create table 2% of write -> overwrite 10% of create table. At the end this is a write and should be included with that workload.

[alpaylan]

I had a different perspective, let me explain it, I think both of them have merits so I'm open to using either.

One thing I was planning for the future is extendable properties, you can imagine generating some stuff in the middle that abides to certain constraints. For example;

CREATE TABLE t1 (int id);
...(no drop(t1), no alter(t1))
CREATE TABLE t1 (char id);

Currently, the middle portion is just empty, we consecutively generate two creates, ideally, that middle part would have a constrained generation that we know will not invalidate the property.

I think this approach works better with a flat property structure, because properties would be rather standalone, a new contributor could add a new property, test it separately without breaking existing properties in any way.

We can push the rule 10% of the writes should be creates to the configuration level, and control the frequency as we did in other examples.

(the 1 here is just me being lazy, sorry about that, I'll fix into the 10% of writes probabilities)

I think this approach would potentially scale to 100s of different properties, which could be added incrementally, especially considering existing bugs in the code. Whenever someone discovers a bug, we could add a property that would catch it, and that property would be the part of the testing setup forever.

The hierarchical approach, in my opinion, would scale to maybe 20-30 properties, but it would be hard to maintain after that, at least that's my intuition.

What do you think?

[alpaylan]

Adding onto this @pereman2,

I even have a plan to turn the properties into data structures in near future, so the probabilities wouldn't be hard-coded as they are right now, but we would just count the interactions and dynamically compute the frequency. I can write up a concept implementation to make it more concrete if you would like.

[jussisaurio]

i really like the concept of imposing constraints on execution while we want certain things to hold like "during t1...t2 table T exists with the schema it was created". excited to see where this goes. tbh I would be in favor of just tossing this to main and seeing where it goes -- you seem to have a lot of ideas and exp in this area, so i wouldn't want to keep that train from going while it has momentum

[jussisaurio]

i really like the concept of imposing constraints on execution while we want certain things to hold like "during t1...t2 table T exists with the schema it was created". excited to see where this goes. tbh I would be in favor of just tossing this to main and seeing where it goes -- you seem to have a lot of ideas and exp in this area, so i wouldn't want to keep that train from going while it has momentum

[alpaylan]

The latest changes fixes a bug in the simulator, adds an option to set the minimum size, and the maximum time for the simulator.