feat: quick 3 run scheme

[mdjastrzebski]

Summary

This experimental scheme of test runs will run the test 3 times (plus any additional warmup runs to be discarded). This will result in three test results: min, mid & max. We then modify the run table to contain the following content: [min, mid, mid, mid, mid, max], effectively quadrupling the middle measurement.

The idea is based on PERT distribution. Since I am not a statistician, I propose this PR as an engineering exercise when we would verify its usefulness against some stability test. The main benefit being in reducing the default 11 runs (10 real + 1 warmup) to just 4 runs (3 runs + 1 warmpu).

Test plan

This method should undergo stability testing to compare how well or badly it fares against the default run: 10 scheme.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 50d2200

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

Performance Comparison Report

• Current: ea22ba8 - 2023-12-03 22:32:34Z
• Baseline: main (ce92794) - 2023-12-03 22:31:45Z

Significant Changes To Duration

Name	Type	Duration	Count
fib 30	function	84.8 ms → 90.8 ms (+6.0 ms, +7.0%) 🔴	1 → 1
Other Component 10 legacy scenario	render	122.1 ms → 114.6 ms (-7.5 ms, -6.1%) 🟢	4 → 4

Show details

Name	Type	Duration	Count
fib 30	function	Baseline Mean: 84.8 ms Stdev: 0.2 ms (0.3%) Runs: 85.3 84.9 84.9 84.9 84.8 84.8 84.7 84.7 84.5 84.5 Current Mean: 90.8 ms Stdev: 0.1 ms (0.1%) Runs: 90.9 90.8 90.8 90.8 90.8 90.6	Baseline Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1 1 1 1 1 Current Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1
Other Component 10 legacy scenario	render	Baseline Mean: 122.1 ms Stdev: 5.2 ms (4.3%) Runs: 134 127 124 122 122 120 119 118 118 117 Current Mean: 114.6 ms Stdev: 3.9 ms (3.4%) Runs: 121 121 115 114 114 114 113 113 113 108	Baseline Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4 Current Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4

Meaningless Changes To Duration

Show entries

Name	Type	Duration	Count
Async Component	render	199.1 ms → 199.2 ms (+0.1 ms, ±0.0%)	7 → 7
fib 31	function	158.9 ms → 162.8 ms (+3.9 ms, +2.4%)	1 → 1
fib 32	function	244.6 ms → 243.9 ms (-0.7 ms, ±0.0%)	1 → 1
Other Component 10	render	117.9 ms → 118.8 ms (+0.9 ms, +0.8%)	4 → 4
Other Component 20	render	118.7 ms → 117.9 ms (-0.8 ms, -0.6%)	4 → 4

Show details

Name	Type	Duration	Count
Async Component	render	Baseline Mean: 199.1 ms Stdev: 4.5 ms (2.3%) Runs: 209 201 201 200 200 200 196 196 195 193 Current Mean: 199.2 ms Stdev: 11.0 ms (5.5%) Runs: 221 196 196 196 196 190	Baseline Mean: 7 Stdev: 0 (0.0%) Runs: 7 7 7 7 7 7 7 7 7 7 Current Mean: 7 Stdev: 0 (0.0%) Runs: 7 7 7 7 7 7
fib 31	function	Baseline Mean: 158.9 ms Stdev: 4.7 ms (3.0%) Runs: 172.0 159.8 158.3 157.3 157.1 157.0 157.0 156.8 156.8 156.7 Current Mean: 162.8 ms Stdev: 0.2 ms (0.1%) Runs: 163.2 162.7 162.7 162.7 162.7 162.6	Baseline Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1 1 1 1 1 Current Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1
fib 32	function	Baseline Mean: 244.6 ms Stdev: 1.7 ms (0.7%) Runs: 249.2 245.4 244.5 244.2 243.9 243.9 243.8 243.8 243.6 243.5 Current Mean: 243.9 ms Stdev: 0.4 ms (0.2%) Runs: 244.6 243.9 243.9 243.9 243.9 243.3	Baseline Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1 1 1 1 1 Current Mean: 1 Stdev: 0 (0.0%) Runs: 1 1 1 1 1 1
Other Component 10	render	Baseline Mean: 117.9 ms Stdev: 3.4 ms (2.9%) Runs: 121 121 120 120 119 119 118 116 115 110 Current Mean: 118.8 ms Stdev: 3.7 ms (3.1%) Runs: 123 122 121 121 120 118 118 118 117 110	Baseline Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4 Current Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4
Other Component 20	render	Baseline Mean: 118.7 ms Stdev: 5.4 ms (4.5%) Runs: 129 125 124 123 123 122 122 120 120 119 119 119 118 117 116 114 114 112 109 108 Current Mean: 117.9 ms Stdev: 4.9 ms (4.1%) Runs: 127 126 123 122 122 121 121 120 119 118 117 117 115 115 115 114 114 111 111 110	Baseline Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 Current Mean: 4 Stdev: 0 (0.0%) Runs: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

Changes To Count

There are no entries

Added Scenarios

There are no entries

Removed Scenarios

There are no entries

Generated by 🚫 dangerJS against 50d2200