Add support for stress tests to nextest

[sunshowers]

Add support to run a single test multiple times in parallel. This is a somewhat different mode of operation from nextest as usual (the runner should be fed the same test multiple times, and should maybe not run other tests at the same time), but is worth doing as a future improvement.

[howardjohn]

Have done quite a bit of this in the past with Go. Some notes that may help:

In Go, tests are fairly similar; there are a few test binaries that get compiled then you can execute them like ./test and optionally add some filters. So all of our stress testing we do is on these binaries.

We use a tool https://github.com/howardjohn/golang-tools/tree/master/cmd/stress - there is an official Go variant but the fork has a lot more features.

Running it will output something like 129909 runs so far, 48 failures (99.96% pass rate). 34.237162ms avg, 15.518448036s max, 18.07186ms min every second or so. Failure outputs get written to a tempfile for later review (the command just shows a few lines of it as a preview to avoid spam). It also does some (very unsophisticated) logic to detect if an execution is running behind (logging something like process still running after 13.527344113s); this is useful since often failures end up happening in some test logic that is like retry for 10min { check something }, which can mask errors as we won't see any failures until 10min in.

Typically this is used on a single test or test package, focusing on a specific area, but we also use a wrapper that allows running on ~all tests, with some fixed test count or duration. This allows doing something like "run every test for 1000 executions or 1 min (whichever is fastest)" to analyze the entire codebase for flaky tests.

So overall, I think nice features are:

• Run all or a subset of tests many times in a row, as fast as possible, in parallel
• Report current status (execution count, failures, and runtime) in realtime.
• Report slow executions, dynamically detecting what 'slow' is
• Report each failure to a unique log for later analysis

In the meantime, I also created a bash helper that does this. Its a bit overcomplex and poor UX, but usage is like rust-flaker test-binary optional-regex-filter:

function rust-flaker() {
  relpath () {
    python -c "import os.path; print(os.path.relpath('$1','${2:-$PWD}'))"
  }
  clr='\e[0m'
  blue='\e[0;34m'

  executables="$(cargo test --no-run --message-format json | jq -R 'fromjson? | select(type == "object") | select(.executable != null) | select(.profile.test == true) | select(.target.kind | contains(["bin"]) == false)| .executable' -r)"
  want_name="${1}"
  local bin=""
  while read -r line; do
    name="$(basename $line | cut -d- -f1)"
    if [[ "${want_name}" == "${name}" ]]; then
      echo "${blue}Found test binary: $(relpath $line)${clr}"
      bin=${line}
      break
    fi
  done <<< "${executables}"
  if [[ "${bin}" == "" ]]; then
    "Test binary '${want_name}' not found, have:"
    while read -r line; do
      echo "  $(basename $line | cut -d- -f1)"
    done <<< "${executables}"
    return 1
  fi

  echo "${blue}Running test once...${clr}"
  $bin "${2:-}" || return 1
  echo "${blue}Running test repeatedly...${clr}"
  filter="${2:-}"
  shift; shift
  stress "$@" $bin "${filter}"
}

[sunshowers]

Thanks for the detailed overview @howardjohn!

[iamazy]

Hi, @sunshowers Is this feature supported now?

[sunshowers]

Hi @iamazy. No, the feature isn't yet supported. I don't expect to get to it any time at least in 2023 either, I'm too busy with other work.

However, I'd be interested in both design and engineering contributions towards this. I think the list of concerns mentioned in @howardjohn's post is a good start. I know @bmwill expressed some interest in implementing it but I believe he's been super busy as well.

[esemeniuc]

Would love to see this implemented

[baxterjo]

Adding my two cents on the proposed design. Would like to see parallelization and repeats as separate configs. Or ensure the { max-threads = 1 } config for certain tests plays nice with this feature.

[sunshowers]

The need for stress testing at my workplace keeps going up, so I'm hoping to carve out some time for this within the next 3 months.