Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,20 @@ All notable changes to msgvault, grouped by release.
cursors, and CardDAV sync runs are recorded and recoverable after a
daemon restart.

- Chat media collection now skips attachments from conversations with more
than 20 participants by default on Beeper, Slack, Discord, and Teams. Direct
chats and small groups keep their media; skipped occurrences carry a typed
`participant_threshold` marker instead of a retry marker. Set
`media_max_participants = 0` in the provider table to remove the cap, or
raise it to taste. With large-room volume gone, the per-attachment size
default for Beeper, Slack, and Teams moves from 100 MiB to 250 MiB so long
voice notes, screen recordings, and phone video from direct chats are kept;
Discord stays at 50 MiB, and an explicit `max_media_mb` is unchanged.
Previously over-cap files under 250 MiB are retried by the next
`backfill-*-media` run because the cap changed. The `media_scope`,
`media_max_participants`, `max_media_mb`, and `accounts_config` keys are now
documented for every chat provider.

- Starting in v0.20.0, remote deletion remains permanently opt-in. The
invoking CLI can grant durable consent with
`[deletion] remote_enabled = true`; `MSGVAULT_ENABLE_REMOTE_DELETE=1`
Expand Down
108 changes: 102 additions & 6 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
last_edited: 2026-08-30
last_edited: 2026-09-03
title: Configuration
description: Configuration file reference, environment variables, and file locations.
---
Expand Down Expand Up @@ -53,6 +53,9 @@ auto_confirm_identities = false
[discord]
# Per-attachment download cap (default: 50 MiB)
max_media_bytes = 52428800
# Skip attachments from rooms with more than this many participants
# (default: 20; 0 = no cap). Shared by [beeper], [slack], and [teams].
media_max_participants = 20
# Trailing edit/delete/reaction repair window (default: seven days)
edit_rescan_window = "168h"

Expand Down Expand Up @@ -463,14 +466,22 @@ add-discord`; tokens and binding labels do not belong in `config.toml`.
| Key | Default | Description |
|---|---|---|
| `max_media_bytes` | `52428800` (50 MiB) | Maximum size of one Discord attachment downloaded during sync or backfill |
| `max_media_mb` | — | Same cap in MiB; when set it takes precedence over `max_media_bytes` |
| `media` | `true` | Download attachment bytes at all |
| `media_scope` | `all` | Which conversations collect media: `all`, `direct` (direct and group chats only), or `none` |
| `media_max_participants` | `20` | Skip media from conversations with more participants than this; `0` disables the cap. See [Media policy](#media-policy) |
| `edit_rescan_window` | `168h` (seven days) | Trailing per-channel/thread window refreshed for edits, deletions, and reaction summaries |

Use an exact guild ID for a per-guild filter block:
Use an exact guild ID for a per-guild filter block. The same block also takes
the per-account media overrides (`media`, `max_media_mb`) that the other chat
providers put under `accounts_config`:

```toml
[discord.guilds."123456789012345678"]
include = ["456789012345678901"]
exclude = ["567890123456789012"]
# media = false
# max_media_mb = 25
```

An empty `include` means every accessible text or announcement channel, thread,
Expand All @@ -480,6 +491,48 @@ can override an excluded parent; an explicit child exclude can override an
included parent. `exclude` wins when the same ID is in both lists. See
[Discord](/usage/discord/#configure-media-repairs-and-channel-filters).

### Media policy

`[beeper]`, `[slack]`, `[discord]`, and `[teams]` share one attachment policy
vocabulary. It decides which chat media is downloaded during sync and backfill;
message text is always archived.

| Key | Default | Description |
|---|---|---|
| `media` | `true` | Download attachment bytes. `false` archives messages without their media and records a `policy_scope` skip marker |
| `media_scope` | `all` | `all` collects from every conversation; `direct` collects only from direct and group chats (not channels, rooms, or guild channels); `none` collects nothing |
| `media_max_participants` | `20` | Skip media from conversations with more participants than this. Omitting the key applies the default; an explicit `0` removes the cap |
| `max_media_mb` | `250` (Discord `50`) | Per-attachment size cap in MiB. Sized for long voice notes, screen recordings, and phone video from direct chats now that the participant cap keeps large-room volume out |
| `accounts_config` | — | Per-account overrides of `media` and `max_media_mb`, keyed by Beeper accountID, Slack team ID, or Teams account email. Discord uses `[discord.guilds."<id>"]` instead |

The participant cap exists because most attachment bytes in a real chat
archive come from large rooms whose forwarded videos nobody wants kept. Direct
chats and small groups keep their photos, voice notes, and files. A skipped
occurrence is recorded with a typed marker (`participant_threshold`,
`policy_scope`, `account_policy`, or `size_cap`) that distinguishes a
deliberate skip from a failed download, so the `backfill-*-media` commands do
not retry it unless the policy changes.

```toml
[beeper]
media_scope = "all"
media_max_participants = 20
max_media_mb = 250

# Keep everything from one account regardless of room size or size cap.
[beeper.accounts_config.signal]
media = true
max_media_mb = 500

# Never download from another account.
[beeper.accounts_config.telegram]
media = false
```

Policy changes apply to future downloads. Media already stored under an
earlier policy stays until you run `msgvault purge-excluded-media`, which
removes attachment bytes the current policy would no longer collect.

### `[log]`

Structured file logging. Disabled by default. Enable it to get persistent, machine-readable logs for troubleshooting. Every CLI invocation writes a unique `run_id` on every log line so you can trace a single run across shared daily log files.
Expand Down Expand Up @@ -706,7 +759,13 @@ accounts = [] # accountID include filter (empty = all)
exclude_accounts = [] # skip networks archived natively, e.g. ["whatsapp"]
rate_limit_qps = 20 # request rate against the local API
media = true # download attachment bytes
max_media_mb = 100 # per-attachment download cap (MiB)
media_scope = "all" # all, direct, or none
media_max_participants = 20 # skip media from larger rooms; 0 = no cap
max_media_mb = 250 # per-attachment download cap (MiB)

# [beeper.accounts_config.signal] # per-account override, keyed by accountID
# media = true
# max_media_mb = 500
```

| Key | Default | Description |
Expand All @@ -718,7 +777,10 @@ max_media_mb = 100 # per-attachment download cap (MiB)
| `exclude_accounts` | — | Beeper accountIDs to skip (wins over `accounts`) |
| `rate_limit_qps` | `20` | Request rate limit against the local API |
| `media` | `true` | Download attachment bytes (failed downloads retry via `backfill-beeper-media`) |
| `max_media_mb` | `100` | Per-attachment download cap in MiB (over-cap media leaves a retry marker) |
| `media_scope` | `all` | `all`, `direct`, or `none`; see [Media policy](#media-policy) |
| `media_max_participants` | `20` | Skip media from conversations above this many participants; `0` = no cap |
| `max_media_mb` | `250` | Per-attachment download cap in MiB (over-cap media is recorded as a `size_cap` skip and retried only after the cap changes) |
| `accounts_config` | — | Per-accountID `media` and `max_media_mb` overrides |

### `[slack]`

Expand All @@ -733,7 +795,12 @@ schedule = "*/30 * * * *" # 5-field cron; empty = manual sync only
channels = [] # channel-name include filter (empty = all memberships)
exclude_channels = [] # channel names to skip, e.g. ["noise"]
media = true # download shared-file bytes
max_media_mb = 100 # per-file download cap (MiB)
media_scope = "all" # all, direct, or none
media_max_participants = 20 # skip files from larger channels; 0 = no cap
max_media_mb = 250 # per-file download cap (MiB)

# [slack.accounts_config.T0123456] # per-workspace override, keyed by team ID
# media = false
```

| Key | Default | Description |
Expand All @@ -743,7 +810,36 @@ max_media_mb = 100 # per-file download cap (MiB)
| `channels` | all | Channel names to sync (include filter; DMs are never filtered) |
| `exclude_channels` | — | Channel names to skip (wins over `channels`) |
| `media` | `true` | Download shared-file bytes (failed downloads retry via `backfill-slack-media`) |
| `max_media_mb` | `100` | Per-file download cap in MiB (over-cap files leave a retry marker) |
| `media_scope` | `all` | `all`, `direct` (DMs and group DMs only), or `none`; see [Media policy](#media-policy) |
| `media_max_participants` | `20` | Skip files from conversations above this many members; `0` = no cap |
| `max_media_mb` | `250` | Per-file download cap in MiB (over-cap files are recorded as a `size_cap` skip and retried only after the cap changes) |
| `accounts_config` | — | Per-team-ID `media` and `max_media_mb` overrides |

### `[teams]`

Media policy for [Microsoft Teams](/usage/teams/) chats and channels. Teams
sync itself is scheduled through `[[accounts]]`; this table only decides which
attachments are downloaded.

```toml
[teams]
media = true
media_scope = "all"
media_max_participants = 20
max_media_mb = 250

[teams.accounts_config."user@example.com"]
media = true
max_media_mb = 500
```

| Key | Default | Description |
|---|---|---|
| `media` | `true` | Download attachment and inline hosted-content bytes (failed downloads retry via `backfill-teams-media`) |
| `media_scope` | `all` | `all`, `direct` (chats only, not channels), or `none`; see [Media policy](#media-policy) |
| `media_max_participants` | `20` | Skip media from chats and channels above this many members; `0` = no cap |
| `max_media_mb` | `250` | Per-attachment download cap in MiB |
| `accounts_config` | — | Per-account overrides of `media` and `max_media_mb`, keyed by the Teams account email |

### Granola Sources

Expand Down
25 changes: 21 additions & 4 deletions docs/usage/beeper.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
last_edited: 2026-09-03
title: Beeper
description: Archive every chat network connected to Beeper Desktop via its local API.
---
Expand Down Expand Up @@ -90,9 +91,14 @@ are only picked up by `--full` runs.
- Voice-note transcriptions (when Beeper has them) are appended to the message
body so they are searchable.
- Attachments (photos, videos, voice notes, files) are downloaded during sync
into msgvault's content-addressed attachment store. Downloads that fail (or
exceed `max_media_mb`) leave a pending marker and the message is archived
anyway; retry them later with `msgvault backfill-beeper-media`. Use
into msgvault's content-addressed attachment store. By default media from
conversations with more than 20 participants is skipped with a typed
`participant_threshold` marker, so direct chats and small groups keep their
media while large rooms do not fill the disk; set
`media_max_participants = 0` to collect from every room. Downloads that fail
leave a pending marker and the message is archived anyway; retry them later
with `msgvault backfill-beeper-media`. Over-cap files (`max_media_mb`) are
recorded as a `size_cap` skip and retried only after the cap changes. Use
`--no-media` or `media = false` to skip downloads — note that skipped-by-flag
downloads leave no pending markers, so `backfill-beeper-media` will not fetch
them later; re-enable media and run `sync-beeper --full` instead.
Expand Down Expand Up @@ -167,9 +173,20 @@ accounts = [] # accountID include filter (empty = all)
exclude_accounts = [] # e.g. ["whatsapp"] — see below
rate_limit_qps = 20 # request rate against the local API
media = true # download attachment bytes
max_media_mb = 100 # per-attachment size cap
media_scope = "all" # all, direct, or none
media_max_participants = 20 # skip media from larger rooms; 0 = no cap
max_media_mb = 250 # per-attachment size cap

# [beeper.accounts_config.signal] # per-account override, keyed by accountID
# media = true
# max_media_mb = 500
```

See [Media policy](/configuration/#media-policy) for how the scope, participant
cap, size cap, and per-account overrides combine, and
`msgvault purge-excluded-media` for removing media a changed policy would no
longer collect.

### Overlap with native importers

If you already archive a network natively (e.g. `import-whatsapp` or
Expand Down
16 changes: 13 additions & 3 deletions docs/usage/discord.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
last_edited: 2026-09-03
title: Discord
description: Archive Discord guild channels, threads, and attachments through a read-only bot.
---
Expand Down Expand Up @@ -84,19 +85,28 @@ ambiguous fails instead of guessing.
## Configure media, repairs, and channel filters

Discord settings are optional. The defaults download attachments up to 50 MiB
and re-scan the trailing seven days for edits, deletions, and changed reaction
counts:
from channels with at most 20 members, and re-scan the trailing seven days for
edits, deletions, and changed reaction counts:

```toml
[discord]
max_media_bytes = 52428800
media_scope = "all" # all, direct, or none
media_max_participants = 20 # skip media from larger channels; 0 = no cap
edit_rescan_window = "168h"

[discord.guilds."123456789012345678"]
include = ["456789012345678901"]
exclude = ["567890123456789012"]
# media = false # per-guild media override
# max_media_mb = 25
```

Most guild channels have more than 20 members, so the default participant cap
keeps guild media out of the archive unless you raise the cap or set it to `0`.
Skipped attachments carry a typed `participant_threshold` marker rather than a
retry marker; see [Media policy](/configuration/#media-policy).

`include` and `exclude` contain Discord channel, thread, or forum-post IDs. An
empty `include` means every accessible message container. Top-level channels
match their own IDs. Threads and forum posts inherit their parent's state, but
Expand Down Expand Up @@ -239,7 +249,7 @@ credentials or call the Discord API.
## Attachment backfill and limits

Retry attachment downloads after a transient failure or after raising
`max_media_bytes`:
`max_media_bytes` or `media_max_participants`:

```bash
# Scan all archived Discord messages that have attachments.
Expand Down
17 changes: 12 additions & 5 deletions docs/usage/slack.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
last_edited: 2026-08-30
last_edited: 2026-09-03
title: Slack
description: Archive Slack workspaces through the Web API or a Slackdump export.
---
Expand Down Expand Up @@ -150,9 +150,14 @@ the archived body, raw JSON, attachments, or reactions.
### Files

Files are downloaded into content-addressed attachment storage, capped at
`max_media_mb` per file. Files hosted outside `files.slack.com` (external
links, connected drives) are recorded as metadata + permalink only. Failed
downloads leave pending markers:
`max_media_mb` per file. By default files shared in conversations with more
than 20 members are skipped with a typed `participant_threshold` marker; DMs,
group DMs, and small channels keep theirs. Set `media_max_participants = 0`
under `[slack]` to collect from every channel, or `media_scope = "direct"` to
collect only from DMs and group DMs (see
[Media policy](/configuration/#media-policy)). Files hosted outside
`files.slack.com` (external links, connected drives) are recorded as metadata +
permalink only. Failed downloads leave pending markers:

```bash
msgvault backfill-slack-media
Expand All @@ -172,11 +177,13 @@ deleting the row or retrying an unreachable file forever.
[slack]
enabled = true
schedule = "*/30 * * * *"
media_max_participants = 20 # default; 0 = collect files from every channel
```

The daemon then syncs every registered workspace on the schedule. See
[Configuration](/configuration/#slack) for the full option list
(channel include/exclude filters, media caps).
(channel include/exclude filters, media scope, participant and size caps,
per-workspace `accounts_config` overrides).

## Identity unification

Expand Down
23 changes: 23 additions & 0 deletions docs/usage/teams.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
last_edited: 2026-09-03
title: Microsoft Teams
description: Archive Microsoft Teams chats and channels through delegated Microsoft Graph sync.
---
Expand Down Expand Up @@ -112,6 +113,28 @@ The backfill scans stored Teams HTML bodies for `hostedContents` URLs and
downloads those images into the attachment store. It is idempotent because
attachment storage is content-addressed.

## Media Policy

Attachment downloads follow the shared chat media policy. By default media
from chats and channels with more than 20 members is skipped with a typed
`participant_threshold` marker, while one-to-one and small group chats keep
theirs. Adjust it under `[teams]`:

```toml
[teams]
media = true
media_scope = "all" # all, direct (chats only), or none
media_max_participants = 20 # 0 = no cap
max_media_mb = 250

[teams.accounts_config."user@example.com"]
max_media_mb = 500
```

See [Media policy](/configuration/#media-policy) for the full vocabulary and
`msgvault purge-excluded-media` for removing media a changed policy would no
longer collect.

## Scheduled Sync

`msgvault serve` can schedule Teams syncs through the normal `[[accounts]]`
Expand Down
7 changes: 7 additions & 0 deletions internal/attachmentpolicy/policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@ package attachmentpolicy

import "fmt"

// DefaultChatMaxBytes is the per-attachment size cap chat providers (Beeper,
// Slack, Teams) apply when max_media_mb is unset. It is sized for the media
// worth keeping from direct chats and small groups: long voice notes, screen
// recordings, and phone video routinely clear 100 MiB, and the participant
// cap already keeps large-room volume out. Discord keeps its own lower cap.
const DefaultChatMaxBytes int64 = 250 << 20

// Scope limits media downloads by conversation kind.
type Scope string

Expand Down
4 changes: 2 additions & 2 deletions internal/beeper/media.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ import (
)

// defaultMaxMediaBytes caps individual attachment downloads (config
// max_media_mb overrides).
// max_media_mb overrides). It mirrors the config-layer default exactly.
const (
defaultMaxMediaBytes = int64(100 << 20)
defaultMaxMediaBytes = attachmentpolicy.DefaultChatMaxBytes
beeperAttachmentTypeImage = "img"
)

Expand Down
Loading