Skip to content

test: add cross-node invariant checks#3055

Draft
shuowang12 wants to merge 1 commit intomainfrom
shuo/antithesis_test_invariant_check
Draft

test: add cross-node invariant checks#3055
shuowang12 wants to merge 1 commit intomainfrom
shuo/antithesis_test_invariant_check

Conversation

@shuowang12
Copy link
Collaborator

Description

Describe the changes or additions included in this PR.

Test plan

How did you test the new or updated feature?


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)

  • Storage node:
  • Aggregator:
  • Publisher:
  • CLI:

@wbbradley
Copy link
Contributor

wbbradley commented Mar 4, 2026

Claude says: Soft-invariant streak counters are not reset on skipped rounds in Antithesis observer. When a node
is unreachable the round is skipped via continue, but event_source_streak and fully_stored_streak retain their previous values. This means non-consecutive violations separated by skipped rounds still accumulate toward the patience threshold, causing false-positive crashes in the exact fault-injection environment (Antithesis) where the script runs. Fix: reset both streak counters to 0 inside the all_ok=false branch before continue. (docker/walrus-antithesis/build-test-config-image/files/run-observer.sh:170-185)

@shuowang12
Copy link
Collaborator Author

Claude says: Soft-invariant streak counters are not reset on skipped rounds in Antithesis observer. When a node is unreachable the round is skipped via continue, but event_source_streak and fully_stored_streak retain their previous values. This means non-consecutive violations separated by skipped rounds still accumulate toward the patience threshold, causing false-positive crashes in the exact fault-injection environment (Antithesis) where the script runs. Fix: reset both streak counters to 0 inside the all_ok=false branch before continue. (docker/walrus-antithesis/build-test-config-image/files/run-observer.sh:170-185)

Thanks for the notes. I am still trying to figure out how to run the Antithesis tests against a branch/PR and thus created this draft PR. Will address the issue later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants