Added PISTACHE_ENABLE_FLAKY_TESTS option, defaulting to True, to toggle unit tests that are known to be flaky... #1051

kiplingw · 2022-04-19T04:00:58Z

Added PISTACHE_ENABLE_FLAKY_TESTS option, defaulting to True, to toggle unit tests that are known to be flaky. This is a temporary workaround for failing CI created by #1023 and #1033.

…le unit tests that are known to be flaky...

kiplingw · 2022-04-19T13:31:15Z

@Tachi107 it looks like all tests either explicitly failed or were false negatives. The explicitly failed tests, like on amd64 are claiming:

../meson.build:5:0: ERROR: Unknown options: "PISTACHE_ENABLE_FLAKY_TESTS"

I don't know why because that's been added to CMakeLists.txt, meson_options.txt, and tests/meson.build. Is it possible not everything intended in the PR was used in the CI test bed?

As for those that ought to have failed, as an example on ppc64el it looks like none of the tests are run because of a broken pipe:

autopkgtest [13:10:41]: starting date: 2022-04-19
autopkgtest [13:10:41]: version 5.21
autopkgtest [13:10:41]: host fv-az136-332; command line: /usr/bin/autopkgtest ../pistache_0.0.3+git20220329.cacd640-0ubuntu1.dsc -- autopkgtest-virt-qemu --dpkg-architecture=ppc64el --timeout-reboot=120 --ram-size=3072 /home/runner/autopkgtest-testing.img
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-cfpc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-sbbc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-ibs=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-cfpc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-sbbc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-ibs=workaround
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [13:12:41]: ERROR: testbed failure: cannot send to testbed: [Errno 32] Broken pipe
Warning: autopkgtest failed, but it's not your fault

Thoughts?

Tachi107 · 2022-04-19T13:39:06Z

@Tachi107 it looks like all tests either explicitly failed or were false negatives. The explicitly failed tests, like on amd64 are claiming:

../meson.build:5:0: ERROR: Unknown options: "PISTACHE_ENABLE_FLAKY_TESTS"

I don't know why because that's been added to CMakeLists.txt, meson_options.txt, and tests/meson.build. Is it possible not everything intended in the PR was used in the CI test bed?

That's probably because you opened a PR from a branch under the pistacheio namespace and the checkout CI action is cloning the master branch instead of yours. I've never tested this, as I usually branch in my personal repo instead, but the issue should go away when merging this into master.

As for those that ought to have failed, as an example on ppc64el it looks like none of the tests are run because of a broken pipe:

Unfortunately that's an issue I've never been able to solve. Something's wrong in either GitHub's Ubuntu image or autopkgtest, and virtualized runs often fail for no apparent reason.

kiplingw · 2022-04-19T13:48:15Z

Got it. Ok, let's wait and see what the Launchpad builders do now.

kiplingw · 2022-04-19T14:07:52Z

All builds ok!

eli-schwartz · 2022-04-19T14:36:23Z

tests/meson.build

+
+if get_option('PISTACHE_ENABLE_FLAKY_TESTS')
+    pistache_test_files += 'streaming_test'
+    pistache_test_files += 'rest_swagger_server_test'
+endif
+


I would have instead suggested defining these as test(..., suite: ['flaky']), so that people can select the tests at will with meson test --no-suite flaky. It is also possible to make meson test without arguments default to not running a suite:

add_test_setup( 'noflaky', exclude_suites: ['flaky'], is_default: true, # run this one without `meson test --setup noflaky` )

Oh that's cool, and certainly better than having to reconfigure your build to skip flaky tests.

@kiplingw for this to work in the Ubuntu PPA we would need to bump debhelper-compat to version 13, as version 12 and below run ninja test instead of meson test

add_test_setup() works well with ninja test too. :)

(No idea how debian packaging works, so no clue whether bumping debhelper can cause compat issues.)

EDIT: though unfortunately it has the opposite problem -- it raises the minimum meson to 0.57 for exclude_suite.

But it would force us to skip flaky tests by default, right?

@eli-schwartz, I think your solution is much more elegant. If you'd like to submit a PR, I'm totally fine with that.

@Tachi107, yes, I recall we'd have to bump the debhelper-compat version. I'm fine with that, but will that break all unit tests?

@Tachi107, yes, I recall we'd have to bump the debhelper-compat version. I'm fine with that, but will that break all unit tests?

No, but it might break compatibility with old Ubuntu releases. I don't know if the build tools of older Ubuntu versions get upgraded to support newer debhelper-compat versions. But we could at least try

Do you want to give it a try?

I also think the approach of flagging flaky tests as flaky in the build environment is much better than what I proposed. That way they still run, but the entire build environment doesn't break in the interim while we wait for fixes.

I also think the approach of flagging flaky tests as flaky in the build environment is much better than what I proposed. That way they still run, but the entire build environment doesn't break in the interim while we wait for fixes.

No, they wouldn't run, but you would be able to easily skip them with --no-suite flaky. An alternative that would run still run them but wouldn't mark their failure as such would be to mark them with should_fail: true.

I've submitted a PR that implements what Eli suggested as #1052

Ok perfect. If you could add to your PR to override override_dh_auto_test in d/rules so that --no-suite flaky is added, then it should work.

Added PISTACHE_ENABLE_FLAKY_TESTS option, defaulting to True, to togg…

1a09d84

…le unit tests that are known to be flaky...

kiplingw requested a review from Tachi107 April 19, 2022 04:00

kiplingw self-assigned this Apr 19, 2022

kiplingw requested review from a team and dennisjenkins75 April 19, 2022 04:01

dennisjenkins75 approved these changes Apr 19, 2022

View reviewed changes

Tachi107 approved these changes Apr 19, 2022

View reviewed changes

kiplingw merged commit 3089485 into master Apr 19, 2022

kiplingw deleted the kip-flaky-unit-tests-workaround branch April 19, 2022 13:47

eli-schwartz reviewed Apr 19, 2022

View reviewed changes

Added PISTACHE_ENABLE_FLAKY_TESTS option, defaulting to True, to toggle unit tests that are known to be flaky... #1051

Added PISTACHE_ENABLE_FLAKY_TESTS option, defaulting to True, to toggle unit tests that are known to be flaky... #1051

Uh oh!

Conversation

kiplingw commented Apr 19, 2022

Uh oh!

kiplingw commented Apr 19, 2022

Uh oh!

Tachi107 commented Apr 19, 2022

Uh oh!

kiplingw commented Apr 19, 2022

Uh oh!

kiplingw commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kiplingw commented Apr 19, 2022 •

edited

Loading

eli-schwartz Apr 19, 2022 •

edited

Loading