Proposal: how we could describe an individual test case#4
Conversation
| "uuid": "4fa03a8d-2e39-4866-9cd3-69b77bd78a6b", | ||
| "testName": "t test produces calibrated p values", | ||
| "keywords": ["t-test", "p value", "statistics"], | ||
| "description": "This test checks that the p values produced by stats::t.test do not deviate substantially from the expected uniform distribution.", |
There was a problem hiding this comment.
GAMP and CSA suggest a Test Objective, which ist linked to N Test cases.
| "keywords": ["t-test", "p value", "statistics"], | ||
| "description": "This test checks that the p values produced by stats::t.test do not deviate substantially from the expected uniform distribution.", | ||
| "references:" "", | ||
| "code": "set.seed(42)\nm <- 100\nfor (n in c(5, 50, 500)) {\n# repeatedly sample data under null and record p value\nres <- numeric(m)\nfor (i in 1:m) {\nres[i] <- t.test(rnorm(n))$p.value\n}\n# expect non significant result\nexpect_true(\nks.test(res, 'punif')$p.value > 0.05\n)\n}", |
There was a problem hiding this comment.
should be executable as markdown r chunk
| "references:" "", | ||
| "code": "set.seed(42)\nm <- 100\nfor (n in c(5, 50, 500)) {\n# repeatedly sample data under null and record p value\nres <- numeric(m)\nfor (i in 1:m) {\nres[i] <- t.test(rnorm(n))$p.value\n}\n# expect non significant result\nexpect_true(\nks.test(res, 'punif')$p.value > 0.05\n)\n}", | ||
| "output": "", | ||
| "environment": { |
There was a problem hiding this comment.
specify a reference environment to run the test in
|
@kkmann - thanks for proposing! I really like how this is coming together. One topic that kept coming up in the various stakeholder interviews was the desire to have a formal requirement/use case/spec that the test is addressing. I think it is worth discussing the addition of a (potentially optional) requirement field to the test specification. |
| "testName": "t test produces calibrated p values", | ||
| "keywords": ["t-test", "p value", "statistics"], | ||
| "description": "This test checks that the p values produced by stats::t.test do not deviate substantially from the expected uniform distribution.", | ||
| "references:" "", |
There was a problem hiding this comment.
| "references:" "", | |
| "references": "", |
|
In General, we should consider to standardize to the CSA guidance. Maybe the structure for "Assurance Activity |
4 participants
This PR explores how we could describe an individual test case. The idea would be to then aggregate multiple such tests to assess a function / package and build human-readable documentation.
Besides test cases, we could also define formats for sharing bugs or discrepancies from other implementaitons -> different topic though.
Would be cool to see whether we can run these atomic test cases in GH actions.