Skip to content

Commit fa34402

Browse files
committed
docs: add metapang cache/list
1 parent 1adac77 commit fa34402

3 files changed

Lines changed: 179 additions & 43 deletions

File tree

‎docs/data.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Managing pangenome data
2+
3+
`MetaPanG` profiles against `PanGBank` collections. Two commands let you see what
4+
is available and manage the files kept locally: `metapang list` to browse the
5+
catalog, and `metapang cache` to inspect, pre-download, and remove cached data.
6+
7+
## `metapang list`
8+
9+
Browse the `PanGBank` catalog and check what this `MetaPanG` release supports.
10+
11+
With no argument it lists every collection and release:
12+
13+
```
14+
metapang list
15+
```
16+
17+
| Column | Meaning |
18+
| --- | --- |
19+
| `collection` | The collection name. |
20+
| `version` | A release of that collection. |
21+
| `pangenomes` | Number of species pangenomes in the release. |
22+
| `supported` | Whether this `MetaPanG` release can profile against it (see [Collection compatibility](#collection-compatibility)). |
23+
| `discarded` | How many species are discarded by default in that release. |
24+
| `link` | The web page for the release. |
25+
26+
Passing a `collection[@version]` shows that release in detail, including the
27+
species discarded by default and their reasons, plus a link to browse every
28+
species on the web:
29+
30+
```
31+
metapang list GTDB_refseq@2.0.0
32+
```
33+
34+
The version defaults to `latest` when omitted.
35+
36+
## `metapang cache`
37+
38+
The files needed for a run (the collection index, and the graph and pangenome of
39+
each species) are downloaded from `PanGBank` on demand and cached locally, so each
40+
resource is retrieved only once and reused by later runs.
41+
42+
### Where the cache lives
43+
44+
By default the cache is `.metapang-cache` in the working directory. It can be
45+
relocated, in increasing order of precedence:
46+
47+
| Method | Scope |
48+
| --- | --- |
49+
| `cache_directory` in a config file (see [Configuration](configuration.md)) | Persistent |
50+
| `METAPANG_PANGBANK_CACHE_DIRECTORY` environment variable | Per shell |
51+
| `metapang cache --path DIR ...` | Per invocation |
52+
53+
```
54+
metapang cache --path /data/metapang-cache list
55+
```
56+
57+
`metapang cache path` prints the resolved directory.
58+
59+
### Inspecting the cache
60+
61+
`metapang cache list` shows the cached collections and their contents, one row per
62+
item, with its kind (`index`, `pangenome`, or `dbg`), completeness status, and
63+
size, followed by the cache total.
64+
65+
```
66+
metapang cache list
67+
```
68+
69+
### Pre-downloading with `fetch`
70+
71+
`metapang cache fetch` populates the cache ahead of time, so a later `profile` run
72+
has everything it needs offline.
73+
74+
```
75+
metapang cache fetch TARGET [OPTIONS]
76+
```
77+
78+
`TARGET` is `collection[@version][:species[,species,...]]`.
79+
80+
- Without `:species`, only the release bank index is fetched. This is what species
81+
detection needs.
82+
- With `:species`, each named species graph (and its pangenome) is fetched as well.
83+
This is what read mapping needs.
84+
85+
| Option | Default | Description |
86+
| --- | --- | --- |
87+
| `--index` / `--no-index` | on | Fetch the release bank index. |
88+
| `--dbg` / `--no-dbg` | on | Fetch the de Bruijn graph of each named species. |
89+
| `--pangenome` / `--no-pangenome` | on | Fetch the `.h5` pangenome of each named species. |
90+
| `-f`, `--force` | off | Re-download even if already cached. |
91+
92+
```
93+
# prepare for detection: the index only
94+
metapang cache fetch GTDB_refseq@2.0.0
95+
96+
# prepare to profile two species end to end: index plus their graphs
97+
metapang cache fetch GTDB_refseq@2.0.0:s__Klebsiella_pneumoniae,s__Abiotrophia_defectiva
98+
```
99+
100+
An unsupported `collection@version` is rejected up front, and unknown species
101+
names are reported before anything is downloaded. Already-cached items are kept
102+
and reported as present unless `--force` is given.
103+
104+
### Clearing the cache
105+
106+
`metapang cache clear` removes the whole cache, or a subset given as
107+
`collection[@version][:name]`.
108+
109+
```
110+
# everything (asks for confirmation)
111+
metapang cache clear
112+
113+
# one collection release
114+
metapang cache clear GTDB_refseq@2.0.0
115+
116+
# a single species within a release
117+
metapang cache clear GTDB_refseq@2.0.0:s__Abiotrophia_defectiva
118+
```
119+
120+
Add `-y` to skip the confirmation prompt.
121+
122+
## Collection compatibility
123+
124+
Each `MetaPanG` release ships a compatibility policy that decides which
125+
`PanGBank` collections it can profile against. Run [`metapang list`](#metapang-list)
126+
to see, per release, whether it is supported and how many species are discarded.
127+
128+
**Supported versions.** A collection version this release was not validated
129+
against is rejected up front, before any work is done:
130+
131+
```
132+
metapang profile -q reads.fastq.gz -b GTDB_refseq@1.0.0
133+
# Error: 'GTDB_refseq@1.0.0' is not compatible with this MetaPanG version
134+
# (supported versions: 2.0.0).
135+
```
136+
137+
This is a hard check and cannot be overridden; upgrade or downgrade `MetaPanG`
138+
to match the collection.
139+
140+
**Discarded species.** Within a supported version, some species may be
141+
discarded by default (for example, unreliable pangenomes).
142+
143+
When any are excluded the `profile` run log lists them:
144+
145+
```
146+
Excluding 1 discarded species from detection (use --include-discarded to include them):
147+
- s__Some_species: low quality pangenome
148+
```
149+
150+
Unlike the version check, discards are overridable with `--include-discarded`:
151+
152+
```
153+
# include every discarded species
154+
metapang profile -q reads.fastq.gz -b GTDB_refseq@2.0.0 --include-discarded
155+
156+
# include only specific ones (comma-separated, note the '=')
157+
metapang profile -q reads.fastq.gz -b GTDB_refseq@2.0.0 \
158+
--include-discarded=s__Some_species,s__Other_species
159+
```
160+
161+
Naming a discarded species explicitly through the `:pangenomes` suffix of `-b`
162+
also profiles it, since that skips detection entirely.

‎docs/profile.md‎

Lines changed: 5 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -34,44 +34,11 @@ command in the same output directory continues where it stopped.
3434

3535
## Collection compatibility
3636

37-
Each `MetaPanG` release ships a compatibility policy that decides which
38-
`PanGBank` collections it can profile against.
39-
40-
**Supported versions.** A collection version this release was not validated
41-
against is rejected up front, before any work is done:
42-
43-
```
44-
metapang profile -q reads.fastq.gz -b GTDB_refseq@1.0.0
45-
# Error: 'GTDB_refseq@1.0.0' is not compatible with this MetaPanG version
46-
# (supported versions: 2.0.0).
47-
```
48-
49-
This is a hard check and cannot be overridden; upgrade or downgrade `MetaPanG`
50-
to match the collection.
51-
52-
**Discarded species.** Within a supported version, some species may be
53-
discarded by default (for example, unreliable pangenomes).
54-
55-
When any are excluded the run log lists them:
56-
57-
```
58-
Excluding 1 discarded species from detection (use --include-discarded to include them):
59-
- s__Some_species: low quality pangenome
60-
```
61-
62-
Unlike the version check, discards are overridable with `--include-discarded`:
63-
64-
```
65-
# include every discarded species
66-
metapang profile -q reads.fastq.gz -b GTDB_refseq@2.0.0 --include-discarded
67-
68-
# include only specific ones (comma-separated, note the '=')
69-
metapang profile -q reads.fastq.gz -b GTDB_refseq@2.0.0 \
70-
--include-discarded=s__Some_species,s__Other_species
71-
```
72-
73-
Naming a discarded species explicitly through the `:pangenomes` suffix of `-b`
74-
also profiles it, since that skips detection entirely.
37+
`MetaPanG` only profiles against collection versions it supports, and discards a
38+
few unreliable species by default. Naming a discarded species explicitly (through
39+
the `:pangenomes` suffix of `-b`, or `--include-discarded`) profiles it anyway. The
40+
full policy and how to override it are described in
41+
[Managing pangenome data](data.md#collection-compatibility).
7542

7643
## Inputs
7744

‎docs/usage.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# Usage
22

33
:::{warning}
4-
Only the `GTDB_refseq@2.0.0` collection is supported at present. Other
5-
collections and versions are not yet compatible with `MetaPanG`.
4+
Only the `GTDB_refseq@2.0.0` collection is supported at present. `MetaPanG` rejects
5+
an unsupported `collection@version`, and a few species are discarded by default
6+
within a supported release (see [Collection compatibility](data.md#collection-compatibility)).
67
:::
78

8-
`MetaPanG` exposes three user-facing commands. Tha main one is `profile`. The two
9-
`search` commands are lower-level tools that expose the matching steps `profile`
10-
performs internally.
9+
`profile` is the main command. Around it, `list` and `cache` browse and manage the
10+
pangenome data a run needs, and the `search` commands expose the matching steps
11+
`profile` performs internally. Run `metapang --help` to see every command.
1112

1213
## Global options
1314

@@ -42,6 +43,11 @@ Putting these after the command, for example `metapang profile -v debug ...`, is
4243
error: `-v`, `-c`, and `-l` are options of `metapang`, not of `profile`.
4344
:::
4445

46+
`metapang list`, `metapang cache`
47+
: Browse the `PanGBank` catalog and manage the locally cached data, including
48+
pre-downloading a collection before a run. See
49+
[Managing pangenome data](data.md).
50+
4551
`metapang profile`
4652
: The main command. Given metagenomic reads and a pangenome collection, it detects
4753
the species present, maps the reads onto each species pangenome, and resolves the
@@ -62,6 +68,7 @@ error: `-v`, `-c`, and `-l` are options of `metapang`, not of `profile`.
6268
:maxdepth: 1
6369
:hidden:
6470

71+
data
6572
profile
6673
search-bank
6774
search-pangenome

0 commit comments

Comments
 (0)