Skip to content

feat(matrix): Matrix rooms as Talk conversations #19214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
karlitschek wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(matrix): Matrix rooms as Talk conversations #19214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion appinfo/info.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
<screenshot>https://raw.githubusercontent.com/nextcloud/spreed/main/docs/video-verfication.png</screenshot>

<dependencies>
<nextcloud min-version="36" max-version="36" />
<nextcloud min-version="34" max-version="36" />
</dependencies>

<background-jobs>
Expand All @@ -79,6 +79,8 @@
<job>OCA\Talk\BackgroundJob\ResetAssignedSignalingServer</job>
<job>OCA\Talk\BackgroundJob\RetryNotificationsJob</job>
<job>OCA\Talk\BackgroundJob\SendScheduledMessages</job>
<job>OCA\Talk\Matrix\BackgroundJob\MatrixSync</job>
<job>OCA\Talk\Matrix\BackgroundJob\MatrixCleanup</job>
</background-jobs>

<repair-steps>
Expand Down Expand Up @@ -106,6 +108,11 @@
<command>OCA\Talk\Command\Developer\GenerateChats</command>
<command>OCA\Talk\Command\Developer\UpdateDocs</command>

<command>OCA\Talk\Command\Matrix\Homeserver</command>
<command>OCA\Talk\Command\Matrix\Account</command>
<command>OCA\Talk\Command\Matrix\Status</command>
<command>OCA\Talk\Command\Matrix\Rekey</command>

<command>OCA\Talk\Command\Monitor\Calls</command>
<command>OCA\Talk\Command\Monitor\HasActiveCalls</command>
<command>OCA\Talk\Command\Monitor\Room</command>
Expand Down
3 changes: 2 additions & 1 deletion composer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
{
"autoload": {
"psr-4": {
"OCA\\Talk\\": "lib/"
"OCA\\Talk\\": "lib/",
"Nextcloud\\Matrix\\": "matrix-client/src/"
}
},
"config": {
Expand Down
3 changes: 3 additions & 0 deletions docs/capabilities.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,3 +237,6 @@
* `config => call => external-call-service` (local) - The target URL for an external call service if one is configured
* `bot-features-api` (local) - Whether bots can fetch their own enabled features using their shared secret
* `promote-demote-owner` - Whether owners can promote other participants to owner and demote other owners again, by sending the `participantType` parameter when promoting or demoting a participant

## 25
* `matrix-rooms` (local) - Matrix rooms can be shown as conversations; `config => matrix => enabled|can-link|linked|homeservers|e2ee-enabled|upload-enabled|typing-enabled`; conversations of Matrix rooms carry `matrixRoomId` and `matrixCapabilities`
10 changes: 10 additions & 0 deletions docs/chat.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,3 +569,13 @@ See [OCP\RichObjectStrings\Definitions](https://github.com/nextcloud/server/blob
* `avatar_removed` - {actor} removed the conversation avatar
* `federated_user_added` - {actor} invited {federated_user} / {federated_user} accepted the invitation
* `federated_user_removed` - {actor} removed {federated_user} / {federated_user} declined the invitation

### Matrix system messages

* `matrix_user_added` - {user} joined the Matrix room (Matrix-only member)
* `matrix_user_removed` - {user} left the Matrix room
* `matrix_room_upgraded` - The Matrix room was upgraded, the conversation is read-only
* `matrix_call_unsupported` - A call was started in Matrix (calls are not supported in Talk for Matrix rooms)
* `matrix_encryption_enabled` - Encryption was enabled for the Matrix room

Rich object `matrix-media` (`{object}`): `id` (Matrix event id), `name`, `mimetype`, `size`, `msgtype`, `mxc`, `encrypted`, `width`, `height`, `link` (download through Talk).
3 changes: 3 additions & 0 deletions docs/constants.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
| `sample` | No | Room is a sample conversation | User ID the sample |
| `event` | Yes | Event conversation created via the calendar | Start and end unix timestamp of the event concatenated by pound sign: `start#end` |
| `extended_conversation` | Yes | Room is created from another conversation (e.g. adding a participant to 1-1) | Token of previous conversation |
| `matrix` | No | Conversation mirrors a Matrix room (`matrix-rooms` capability) | Matrix room id |
| `external_call` | Yes | Room whose calls are handled by an external video service (see [External Call Service](external-call-service.md)) | Meeting/room identifier on the external service side |
| `classified` | No | Classified conversation queued for automatic deletion after a call (retention `retention_classified_rooms`) | Unix timestamp of the call that queued the deletion |
| `classified_persist` | No | Classified conversation a moderator kept (unbound), so it is no longer auto-deleted | Unix timestamp of when it was kept |
Expand Down Expand Up @@ -122,6 +123,7 @@ Required capability: `conversation-presets`
### Attendee types
* `users` - Logged-in users
* `federated_users` - Federated users invited by their CloudID
* `matrix` - Matrix users without a Nextcloud account, the actor id is the Matrix user id (`matrix-rooms` capability)
* `groups` - Groups
* `circles` - [Circle from the Circles app](https://github.com/nextcloud/circles)
* `guests` - Guest without a login
Expand Down Expand Up @@ -151,6 +153,7 @@ Required capability: `conversation-presets`
* `bridged` - Users whose messages are bridged in by the [Matterbridge integration](matterbridge.md)
* `deleted_users` - Former logged-in users that got deleted (actor id is hardcoded to `deleted_users` and the display name is empty)
* `federated_users` - Federated users
* `matrix` - Matrix users without a Nextcloud account (`matrix-rooms` capability)

### Session states
* `0` - Inactive (Notifications should still be sent, even though the user has this session in the room)
Expand Down
2 changes: 2 additions & 0 deletions docs/conversation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@
| `lastCommonReadMessage` | int | v3 | | ID of the last message read by every user that has read privacy set to public in a room. When the user themself has it set to private the value is `0` (only available with `chat-read-status` capability) |
| `lastMessage` | message | v1 | | Last message in a conversation if available, otherwise empty. **Note:** Even when given the message will not contain the `parent` or `reactionsSelf` attribute due to performance reasons |
| `objectType` | string | v1 | | The type of object that the conversation is associated with (See [Object types](constants.md#object-types)) |
| `matrixRoomId` | string | v4 | `matrix-rooms` | Matrix room id, only for conversations with `objectType` = `matrix` |
| `matrixCapabilities` | object | v4 | `matrix-rooms` | What the Matrix room and the current user in it support (`canSend`, `canInvite`, `canRename`, `isModerator`, `isDirect`, `encrypted`, `calls`, …), only for `objectType` = `matrix` |
| `objectId` | string | v1 | | See [Object types](constants.md#object-types) documentation for explanation |
| `breakoutRoomMode` | string | v4 | | Breakout room configuration mode (see [constants list](constants.md#breakout-room-modes)) (only available with `breakout-rooms-v1` capability) |
| `breakoutRoomStatus` | string | v4 | | Breakout room status (see [constants list](constants.md#breakout-room-status)) (only available with `breakout-rooms-v1` capability) |
Expand Down
142 changes: 142 additions & 0 deletions docs/matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Matrix rooms in Talk

Talk can show and use rooms of a [Matrix](https://matrix.org) account as conversations. Talk's
server acts as a Matrix *client* (one device per linked Nextcloud user): it syncs the rooms of the
linked account into ordinary Talk conversations, so the web interface as well as the unchanged
mobile and desktop apps work with them.

Status: **phase 4** – everything below plus: create Matrix rooms and direct chats from Talk,
join rooms by address or from the public directory (*Matrix room …* in the conversation-list
menu), restore encrypted history from the key backup (verified device or recovery key),
mentions of Matrix-only members, `occ talk:matrix:rekey`. Phase 3 – everything from phases 1 and 2 plus reactions, edits, deletions and
threads in both directions, read receipts to Matrix, inline media with thumbnails, "Save to
Nextcloud" and uploads of shared files (also into encrypted rooms). Phase 2 – everything from phase 1 (account linking, rooms read/write, replies,
invitations, membership, power levels, room name/topic) plus **end-to-end encrypted rooms**:
Talk is a real Matrix E2EE device (Olm/Megolm implemented in PHP, interoperable with
Element/vodozemac), decrypts incoming messages when they arrive, encrypts outgoing ones and
shares room keys with the members' devices, decrypts encrypted attachments, and can be
verified from another client via emoji (SAS). Media upload, reactions/edits/threads and room
creation follow in later phases. See `docs/matrix/SPEC.md` for the full design.

## Administration

1. In *Administration settings → Talk → Matrix rooms* enable the feature.
2. Add the homeserver(s) people may link accounts on. Talk resolves the server name via
`.well-known/matrix/client` and validates `/_matrix/client/versions`; a client API URL can be set
manually. Only these servers are ever contacted.
3. Optionally restrict linking to groups and tune the synchronisation settings.

Synchronisation runs in the background job `MatrixSync` every time cron runs (interval 10 s, budget
25 s per run, per-account locks). Real latency is bounded by how often cron runs – **run cron every
minute**. When a person looks at a Matrix conversation in a client, Talk additionally syncs inline if
the account's data is older than *foreground sync age* (default 5 s), so the web UI feels near-real-time.

`occ` commands:

```
occ talk:matrix:homeserver add example.org [--name=Label] [--base-url=https://matrix.example.org]
occ talk:matrix:homeserver list|remove|test <server-name>
occ talk:matrix:account list
occ talk:matrix:account sync --user=<uid> [--budget=20]
occ talk:matrix:account unlink --user=<uid>
occ talk:matrix:status [--output=json] # exit code 2 when accounts need a re-login
occ talk:matrix:account request-keys --user=<uid> # ask the other devices for missing room keys again
occ talk:matrix:rekey --old-secret=<previous config.php secret> [--dry-run] # after rotating "secret"
```

App config keys (`occ config:app:set spreed <key>`): `matrix_enabled`, `matrix_allowed_groups`
(JSON list), `matrix_sync_interval` (10–300 s, default 30), `matrix_idle_sync_interval` (default
120), `matrix_max_parallel_syncs` (4), `matrix_foreground_sync_age` (5), `matrix_history_events`
(200), `matrix_history_days` (30), `matrix_max_upload` (bytes), `matrix_typing_in`, `matrix_typing_out`,
`matrix_e2ee_shared_lookup`, `matrix_e2ee_verified_only`.

## Linking an account

In Talk → Settings → *Matrix account* a person picks a homeserver and enters their Matrix username
and password. Talk performs a `m.login.password` login once, stores only the access token (encrypted
with the instance secret) and device id, and registers as a device named `Nextcloud Talk (<instance>)`.
Rooms are imported immediately (bounded) and then kept in sync. If the homeserver invalidates the
token, the person gets a notification and can log in again; the device id is reused.

Unlinking logs the device out on the homeserver, deletes the stored token and removes the person from
their Matrix conversations. Conversations without any remaining linked Nextcloud user are deleted.

## End-to-end encryption

* On linking, Talk creates an Olm account for the new device and uploads its device keys plus
one-time keys. Key material and tokens are stored encrypted with the instance secret in the
`talk_matrix_*` crypto tables; **decrypted messages are stored like any other Talk message**
(this is what makes search, previews and all clients work — the Nextcloud server is inside
the trust boundary, see the spec).
* Incoming encrypted events are decrypted during sync. When the key is missing the message is
shown as "🔒 Encrypted message – waiting for the key", a key request is sent to the person's
other devices and to the sender, and the placeholder is replaced in place once the key arrives.
* Sending creates/rotates a Megolm session per room (rotation by age/message count/membership
change) and shares it via Olm with every device of every member (unless the device is blocked,
or the admin enabled *only trusted devices*).
* **Verify the Talk device** in Talk → Settings → Matrix account → *Verify this device*: the
request appears on the other Matrix clients (Element etc.), both sides compare seven emoji,
then the other client cross-signs the Talk device and shares keys for rooms it knows. Until
a device is verified, other clients may refuse to share keys with it.
* Key requests from the person's *own* devices are answered only when that device is
cross-signed or was verified. Forwarded keys are accepted only from such devices.
* Admin policy per homeserver: *Allow encrypted rooms* off → Talk neither decrypts nor sends
into encrypted rooms (placeholders only). `matrix_e2ee_shared_lookup` (default on) lets a
message be decrypted with another linked Nextcloud user's copy of the key when they are in the
same room; `matrix_e2ee_verified_only` restricts key sharing to cross-signed/verified devices.
* **Encrypted history from before the link**: Talk → Settings → Matrix account → *Restore
encrypted history*. After a verification the other client usually hands over the backup key
automatically; otherwise paste the recovery key (`EsTc …`). Talk downloads the server-side key
backup, imports the sessions and replaces the placeholders.
* Not yet: verifying *other* users, creating cross-signing keys.

## How rooms map to conversations

| Matrix | Talk |
|---|---|
| joined room | group conversation with `objectType = "matrix"`, `objectId = <room id>` |
| room name / canonical alias / heroes | conversation name (Matrix room-name algorithm) |
| topic | description |
| `m.room.avatar` (or the peer's avatar in a direct chat) | conversation avatar |
| linked Nextcloud users | `users` attendees |
| other members | attendees with actor type `matrix` (actor id = Matrix user id, display name and avatar from Matrix; avatar served by `/apps/spreed/matrix/avatar/{token}/{size}/{mxid}`) |
| power level ≥ 50 / creator | moderator / owner |
| `m.room.message` text/notice/emote | chat message (HTML → Markdown, pills → mentions) |
| `m.image`/`m.file`/… | `matrix-media` rich object (link served by `/apps/spreed/matrix/media/{eventId}`, `?thumbnail=1` for previews, `POST …/save` copies it into the user's files) |
| `m.in_reply_to` | reply |
| `m.thread` | Talk thread (created on first use) |
| `m.reaction` / redaction of one | reaction added / removed |
| `m.replace` (edit by the original sender) | message edited |
| `m.room.redaction` | message deleted (Talk moderator rules do not apply – Matrix power levels do) |
| Nextcloud file shared into the conversation | uploaded to the homeserver and posted as `m.file`/`m.image`/… (encrypted attachment in encrypted rooms) |
| membership changes | system messages (`matrix_user_added`, `matrix_user_removed` for Matrix-only members) |
| `m.room.tombstone` | conversation becomes read-only, `matrix_room_upgraded` |
| calls | `matrix_call_unsupported` system message |
| invite | pending invitation in the federation invitation API (`remoteServerUrl` = `matrix:<server>`) |

Calls, lobby, SIP, listable, password, read-only toggle, message expiration, recording consent and
permission changes are rejected with HTTP 405 `matrix-unsupported`. Renaming, topic, invites, kicks,
promotions, leaving, reactions, edits, deletions and thread replies are executed on the Matrix side
first (the homeserver enforces power levels); the Talk read marker is mirrored as an `m.read` receipt.

## API additions

* Capabilities: feature `matrix-rooms` (local), `config.matrix` with `enabled`, `can-link`, `linked`,
`homeservers`, `e2ee-enabled`, `upload-enabled`, `typing-enabled`.
* Conversation objects of Matrix rooms carry `matrixRoomId` and `matrixCapabilities` (room-level
capabilities plus the current user's power-level dependent ones, e.g. `canSend`, `canInvite`,
`canRename`, `isModerator`, `isDirect`, `encrypted`).
* `GET|POST|PUT|DELETE /ocs/v2.php/apps/spreed/api/v1/matrix/account`, `POST …/matrix/account/sync`
* Verification: `POST|GET|PUT|DELETE …/matrix/account/verification` (start, poll, confirm/reject, cancel)
* Key backup: `POST …/matrix/account/backup` (`recoveryKey` optional)
* Rooms: `POST …/matrix/room` (create/DM), `POST …/matrix/room/join` (`reference`), `GET …/matrix/room/directory`
* Administration: `…/matrix/admin/homeserver[/{id}[/test]]`, `PUT …/matrix/admin/settings`, `GET …/matrix/admin/status`
* Adding a participant to a Matrix conversation accepts `source=users` (linked Nextcloud user) or
`source=matrix` with a Matrix user id.

## The Matrix client library

The protocol implementation lives in `matrix-client/` as the standalone, framework-free package
`nextcloud/matrix-client-php` (namespace `Nextcloud\Matrix`). It has no Nextcloud dependency and
its own test-suite; `examples/echo-bot.php` runs against a plain Synapse. Talk-specific code lives
in `lib/Matrix/` (`OCA\Talk\Matrix`).
Empty file added ...vue src/components/SettingsDialog/MatrixAccountSettings.vue src/services/matrixService.ts
View file
Open in desktop
Empty file.
Loading
Loading