Another case of state reset in State Resolution v2

[wj25czxj47bu6q]

---

Update: See #15987 (comment). This original comment contains grossly incorrect conclusions.

---

Description

Synapse is responding over the client–server API with a different set of state events depending on whether the requesting user has registered an account or is a guest.

Steps to reproduce

This bug was discovered in room !PfltsPSvJVImWNRDyj:matrix.org. Here is a series of commands to demonstrate the problem:

# Access token for a registered user account
export USER_TOKEN=''
# Access token for a guest user account
export GUEST_TOKEN=''

export ROOMID='!PfltsPSvJVImWNRDyj:matrix.org'
export HS='https://matrix-client.matrix.org'

export FILTER='[ .[] | walk(if type == "object" then delpaths([["age"],["age_ts"],["replaces_state"],["prev_content"],["prev_sender"],["transaction_id"],["unsigned"],["user_id"]]) else . end | if type=="array" then sort else . end) ] | sort_by(.type)'

curl --no-progress-meter -H 'Accept:application/json' -H "Authorization: Bearer $USER_TOKEN" "$HS/_matrix/client/v3/rooms/$ROOMID/state" | jq -rS "$FILTER" > user_state.json
curl --no-progress-meter -H 'Accept:application/json' -H "Authorization: Bearer $GUEST_TOKEN" "$HS/_matrix/client/v3/rooms/$ROOMID/state" | jq -rS "$FILTER" > guest_state.json

Inspection of the output files user_state.json and guest_state.json reveals wildly different sets of state events, even though both come from the same homeserver. For example:

user_state.json

{
  "content": {
    "join_rule": "public"
  },
  "event_id": "$W8ilz2XuqXYU3LilZkU3BLlbGvQgxGMBn5BY81SmBP0",
  "origin_server_ts": 1687249386275,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@wj25czxj47bu6q:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules"
}

guest_state.json

{
  "content": {
    "join_rule": "invite"
  },
  "event_id": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
  "origin_server_ts": 1686004259002,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@moderation-tools:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules"
}

In this case, the m.room.join_rules reported to registered users is the correct one, while the m.room.join_rules reported to guest users is an old event which has been replaced.

This also does not appear to have any relation with m.room.guest_access, which was set to "guest_access": "forbidden" long before either of the m.room.join_rules state events displayed above were sent.

Homeserver

matrix.org

Synapse Version

1.88.0

Installation Method

I don't know

Database

I don't know

Workers

I don't know

Platform

I don't know

Configuration

No response

Relevant log output

No response

Anything else that would be useful to know?

This issue seems to have cropped up very recently, perhaps in Synapse v1.88.0 or v1.88.0rc1, though it is also entirely possible that it just went unnoticed until now.

[DMRobertson]

Thanks for the thorough report. I think this is going to need more investigation and for us to double check the rules about what state a guest is supposed to see. (It'd also be good to rule out something like e.g. a response cache giving the guest stale data).

[wj25czxj47bu6q]

After further investigation… this issue turns out to be completely different from what I originally thought.

This issue is actually unrelated to whether a user is registered or a guest — that was a premature conclusion on my part. A brand new registered user appears to exhibit the exact same behavior described for the guest account.

A critical confounding variable that I overlooked is that the account I was using for the registered user case had previously joined and left the room, without forgetting the room (it is a space, and Element does not expose the ability to forget spaces in the UI).

I have successfully replicated the discrepancy between brand new user and joined-and-left-but-not-forgotten user on a different homeserver federated to the room (also Synapse v1.88.0).

Attempting to join the room fails, whether as the joined-and-left-but-not-forgotten user or as the brand new user. So it appears that the 'true' state of m.room.join_rules is indeed invite ($lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo).

All relevant events in the timeline, ordered:

$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo — "join_rule": "invite"

{
  "content": {
    "join_rule": "invite"
  },
  "origin_server_ts": 1686004259002,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@moderation-tools:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "replaces_state": "$ywrjmtTuZ7ql5oBGBS466LzpF2AJAP8fGzuBOBUwGN8",
    "prev_content": {
      "join_rule": "public"
    },
    "prev_sender": "@wj25czxj47bu6q:jameskitt616.one",
    "age": 4534812248
  },
  "event_id": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
  "user_id": "@moderation-tools:matrix.org",
  "age": 4534812248,
  "replaces_state": "$ywrjmtTuZ7ql5oBGBS466LzpF2AJAP8fGzuBOBUwGN8",
  "prev_content": {
    "join_rule": "public"
  }
}

$_-e4eQlLxLoCBtaEt5NMQazFz3SxiHtn7EIkuLljU54 — "join_rule": "public"

{
  "content": {
    "join_rule": "public"
  },
  "origin_server_ts": 1686021693065,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@wj25czxj47bu6q:arcticfoxes.net",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "replaces_state": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
    "prev_content": {
      "join_rule": "invite"
    },
    "prev_sender": "@moderation-tools:matrix.org"
  },
  "event_id": "$_-e4eQlLxLoCBtaEt5NMQazFz3SxiHtn7EIkuLljU54",
  "user_id": "@wj25czxj47bu6q:arcticfoxes.net",
  "replaces_state": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
  "prev_content": {
    "join_rule": "invite"
  }
}

Users successfully join

$YYlrJkikizYv4FAZFgm-Y_8Ga4ANJVOZ_UsnWifvKzE — "join_rule": "public"

{
  "content": {
    "join_rule": "public"
  },
  "origin_server_ts": 1686386491812,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@wj25czxj47bu6q:arcticfoxes.net",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "replaces_state": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
    "prev_content": {
      "join_rule": "invite"
    },
    "prev_sender": "@moderation-tools:matrix.org"
  },
  "event_id": "$YYlrJkikizYv4FAZFgm-Y_8Ga4ANJVOZ_UsnWifvKzE",
  "user_id": "@wj25czxj47bu6q:arcticfoxes.net",
  "replaces_state": "$lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo",
  "prev_content": {
    "join_rule": "invite"
  }
}

Users successfully join

$6Frvbfcw2NsQSP7ClnKYU34m8cVGt10uKKyf1tRuHPY — "join_rule": "invite"

{
  "content": {
    "join_rule": "invite"
  },
  "origin_server_ts": 1687249381786,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@wj25czxj47bu6q:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "replaces_state": "$YYlrJkikizYv4FAZFgm-Y_8Ga4ANJVOZ_UsnWifvKzE",
    "prev_content": {
      "join_rule": "public"
    },
    "prev_sender": "@wj25czxj47bu6q:arcticfoxes.net",
    "age": 3289678902
  },
  "event_id": "$6Frvbfcw2NsQSP7ClnKYU34m8cVGt10uKKyf1tRuHPY",
  "user_id": "@wj25czxj47bu6q:matrix.org",
  "age": 3289678902,
  "replaces_state": "$YYlrJkikizYv4FAZFgm-Y_8Ga4ANJVOZ_UsnWifvKzE",
  "prev_content": {
    "join_rule": "public"
  }
}

$W8ilz2XuqXYU3LilZkU3BLlbGvQgxGMBn5BY81SmBP0 — "join_rule": "public"

{
  "content": {
    "join_rule": "public"
  },
  "origin_server_ts": 1687249386275,
  "room_id": "!PfltsPSvJVImWNRDyj:matrix.org",
  "sender": "@wj25czxj47bu6q:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "replaces_state": "$6Frvbfcw2NsQSP7ClnKYU34m8cVGt10uKKyf1tRuHPY",
    "prev_content": {
      "join_rule": "invite"
    },
    "prev_sender": "@wj25czxj47bu6q:matrix.org",
    "age": 3289674413
  },
  "event_id": "$W8ilz2XuqXYU3LilZkU3BLlbGvQgxGMBn5BY81SmBP0",
  "user_id": "@wj25czxj47bu6q:matrix.org",
  "age": 3289674413,
  "replaces_state": "$6Frvbfcw2NsQSP7ClnKYU34m8cVGt10uKKyf1tRuHPY",
  "prev_content": {
    "join_rule": "invite"
  }
}

Users successfully joined, with the last successful join on 21 July 2023 at 12:22:50 UTC

Note that all the aforementioned timeline events are visible to brand new accounts on both homeservers. Yet both servers still believe $lZoXigGI2_plf_Hdj47IDmYPeoV00te_jgkVQGToJJo is the current state.

The timing of the last successful join suggests that the release of Synapse v1.88.0 may have something to do with this issue. Regardless, this is very clearly a state resolution issue, in particular a state reset (#8629) — completely different from my initial conclusion.