Skip to content

Commit 46331cb

Browse files
authored
Merge pull request #2 from labgem/local_pangenome
Local pangenome
2 parents aec2d17 + 826a0f4 commit 46331cb

9 files changed

Lines changed: 429 additions & 119 deletions

File tree

‎docs/data.md‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,8 @@ metapang cache --path /data/metapang-cache list
6767
### Inspecting the cache
6868

6969
`metapang cache list` shows the cached collections and their contents, one row per
70-
item, with its kind (`index`, `pangenome`, or `dbg`), completeness status, and
71-
size, followed by the cache total.
70+
item, with its kind (`index`, `pangenome`, `dbg`, or `local`), completeness status,
71+
and size, followed by the cache total.
7272

7373
```
7474
metapang cache list
@@ -127,6 +127,35 @@ metapang cache clear GTDB_refseq@2.0.0:s__Abiotrophia_defectiva
127127

128128
Add `-y` to skip the confirmation prompt.
129129

130+
## Local pangenomes
131+
132+
Besides `PanGBank` collections, you can profile and search against your own
133+
pangenome: a `ppanggolin` `.h5` file. `metapang cache add` builds required indexes
134+
and put them into the cache under a name you choose. It is then referenced as `local:<name>` anywhere a collection is expected.
135+
136+
The result is stored under `<cache>/local/<name>`.
137+
138+
```
139+
# build the graphs once, from the .h5
140+
metapang cache add my_species.h5 --name my_species
141+
142+
# profile reads against it
143+
metapang profile reads.fastq.gz -b local:my_species
144+
145+
```
146+
147+
148+
| Option | Default | Description |
149+
| --- | --- | --- |
150+
| `-n`, `--name` | required | Name to register the pangenome under (`local:<name>`). |
151+
| `-k`, `--kmer-size` | `25` | K-mer size for the de Bruijn graph. |
152+
| `-t`, `--threads` | `1` | Build threads. |
153+
| `--metagraph-path` | `metagraph` | Path to the `metagraph` binary. |
154+
| `-f`, `--force` | off | Rebuild even if already present. |
155+
156+
Local pangenomes appear in `metapang cache list` under the `local` collection, and
157+
are removed with `metapang cache clear local:<name>`.
158+
130159
## Collection compatibility
131160

132161
Each `MetaPanG` release ships a compatibility policy that decides which
@@ -142,9 +171,6 @@ metapang profile reads.fastq.gz -b GTDB_refseq@1.0.0
142171
# (supported versions: 2.0.0).
143172
```
144173

145-
This is a hard check and cannot be overridden; upgrade or downgrade `MetaPanG`
146-
to match the collection.
147-
148174
**Discarded species.** Within a supported version, some species may be
149175
discarded by default (for example, unreliable pangenomes).
150176

‎docs/profile.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,15 @@ full policy and how to override it are described in
4848
`*.fastq.gz`. Stem of the first file is used as the sample name.
4949

5050
`-b`, `--pangbank` (default `GTDB_refseq`)
51-
: The collection to profile against, as `collection[@version][:pangenomes]`.
51+
: The collection to profile against, as `collection[@version][:pangenomes]`, or a
52+
local pangenome as `local:<name>` (see
53+
[Local pangenomes](data.md#local-pangenomes)).
5254

5355
- `version` is optional and defaults to `latest`.
5456
- `pangenomes` is optional: a comma-separated list, each item either a species
5557
name or a numeric `PanGBank` id. Give one or more to profile exactly those and
5658
skip species detection. Omit it to profile every detected species.
59+
- `local:<name>` profiles against a pangenome built with `metapang cache add`.
5760

5861
```
5962
# detect and profile every species found in the sample

‎docs/search-pangenome.md‎

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,25 @@ matches.
1212
## Synopsis
1313

1414
```
15-
metapang search pangenome -q query.fasta.gz -p GTDB_refseq@2.0.0:s__Abiotrophia_defectiva [OPTIONS]
16-
metapang search pangenome -q query.fasta.gz -g pangenome.dbg -a pangenome.annodbg [OPTIONS]
15+
metapang search pangenome -q query.fasta.gz -b GTDB_refseq@2.0.0:s__Abiotrophia_defectiva [OPTIONS]
16+
metapang search pangenome -q query.fasta.gz -b local:my_pangenome [OPTIONS]
17+
metapang search pangenome -q query.fasta.gz -g pangenome.dbg -a pangenome.annodbg [OPTIONS]
1718
```
1819

1920
## Inputs
2021

21-
The query is required, and the pangenome is given in exactly one of two ways: from
22-
`PanGBank`, or as a local de Bruijn graph plus its annotations.
22+
The query is required, and the pangenome is given in one of three ways: from
23+
`PanGBank`, a local pangenome built with `metapang cache add`, or an explicit local
24+
de Bruijn graph plus its annotations.
2325

2426
`-q`, `--query` (required)
2527
: The query file (fasta/fastq, gzipped accepted), a genome or reads.
2628

27-
`-p`, `--pangbank`
28-
: Use a pangenome from `PanGBank`, as `collection[@version]:name`, for example
29-
`GTDB_refseq:s__Abiotrophia_defectiva` or
30-
`GTDB_refseq@v1.0.0:s__Abiotrophia_defectiva`. The graph is downloaded and cached
31-
on first use.
29+
`-b`, `--pangbank`
30+
: Use a pangenome from `PanGBank`, as `collection[@version]:name` (for example
31+
`GTDB_refseq@2.0.0:s__Abiotrophia_defectiva`, downloaded and cached on first use),
32+
or a local pangenome as `local:<name>` built with `metapang cache add` (see
33+
[Local pangenomes](data.md#local-pangenomes)).
3234

3335
`-g`, `--dbg`
3436
: A local pangenome de Bruijn graph (the `.dbg` file).
@@ -37,8 +39,9 @@ The query is required, and the pangenome is given in exactly one of two ways: fr
3739
: The annotations for that graph (the `.annodbg` file).
3840

3941
:::{note}
40-
Give either `--pangbank`, or both `--dbg` and `--annotations`. The `--dbg` /
41-
`--annotations` pair is the per-pangenome output of `metapang index bank`.
42+
Give either `--pangbank` (a `PanGBank` or `local:` pangenome), or both `--dbg` and
43+
`--annotations`. The `--dbg` / `--annotations` pair is the per-pangenome output of
44+
`metapang index bank`.
4245
:::
4346

4447
## Options
@@ -64,20 +67,22 @@ with its read and k-mer counts, then saved to the given path. This is the same
6467
annotated graph `metapang profile` builds internally.
6568

6669
:::{note}
67-
`--annotate` requires `--pangbank`, because it needs the prebuilt `.gt` graph
68-
downloaded from `PanGBank` (the local `--dbg` / `--annotations` inputs do not include
69-
it). It also needs the query results in a file, so it cannot be combined with
70-
`--output stdout`.
70+
`--annotate` requires `--pangbank` (a `PanGBank` or `local:` pangenome)
71+
It also needs the query results in a file, so it cannot be combined with
72+
dout`.
7173
:::
7274

7375
## Examples
7476

7577
```
7678
# search a genome against one PanGBank pangenome
7779
metapang search pangenome -q genome.fasta.gz \
78-
-p GTDB_refseq@2.0.0:s__Abiotrophia_defectiva -t 8
80+
-b GTDB_refseq@2.0.0:s__Abiotrophia_defectiva -t 8
7981
80-
# search a locally indexed pangenome and print to stdout
82+
# search against a local pangenome (built with 'metapang cache add')
83+
metapang search pangenome -q genome.fasta.gz -b local:my_pangenome -o -
84+
85+
# search an explicit dbg / annotations pair and print to stdout
8186
metapang search pangenome -q genome.fasta.gz \
8287
-g s__Abiotrophia_defectiva.dbg -a s__Abiotrophia_defectiva.annodbg -o -
8388
```

‎metapang/app/cache.py‎

Lines changed: 113 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,13 @@
1010
from metapang.logger import mp_log
1111
from metapang.pg import compat
1212
from metapang.pg.api import PanGBank_Cache, parse_collection_name_version
13+
from metapang.pg.local import (
14+
LOCAL_COLLECTION,
15+
build_local_pangenome,
16+
is_built,
17+
local_dir,
18+
local_root,
19+
)
1320

1421

1522
def _cache(ctx) -> PanGBank_Cache:
@@ -67,6 +74,54 @@ def path(ctx) -> None:
6774
click.echo(str(_cache(ctx).directory))
6875

6976

77+
@cache.command()
78+
@click.argument(
79+
"pangenome", type=click.Path(exists=True, dir_okay=False, path_type=Path)
80+
)
81+
@click.option(
82+
"--name",
83+
"-n",
84+
required=True,
85+
help="Name to register the pangenome under (referenced as 'local:<name>').",
86+
)
87+
@click.option(
88+
"--kmer-size",
89+
"-k",
90+
type=int,
91+
help="K-mer size for the de Bruijn graph __placeholder__.",
92+
)
93+
@click.option("--threads", "-t", type=int, help="Build threads __placeholder__.")
94+
@click.option(
95+
"--metagraph-path",
96+
help="Path to the metagraph binary __placeholder__.",
97+
)
98+
@click.option("--force", "-f", is_flag=True, help="Rebuild even if already present.")
99+
@click.pass_context
100+
def add(ctx, pangenome, name, kmer_size, threads, metagraph_path, force) -> None:
101+
"""
102+
Build a local pangenome (.h5) into the cache as 'local:<name>'.
103+
104+
\b
105+
Profile against it with:
106+
metapang profile reads.fastq.gz -b local:<name>
107+
"""
108+
import tempfile
109+
110+
store = _cache(ctx)
111+
with tempfile.TemporaryDirectory(prefix="metapang-build-") as tmp:
112+
dbg_dir = build_local_pangenome(
113+
pangenome,
114+
name,
115+
kmer_size,
116+
store.directory,
117+
Path(tmp),
118+
threads,
119+
metagraph_path,
120+
force=force,
121+
)
122+
mp_log.info(f"Added local:{name}: {dbg_dir}")
123+
124+
70125
@cache.command(name="list")
71126
@click.pass_context
72127
def list_(ctx) -> None:
@@ -75,40 +130,55 @@ def list_(ctx) -> None:
75130
root = store.collection_directory
76131
console = Console()
77132

78-
if not root.exists() or not any(root.iterdir()):
79-
console.print(f"Cache is empty ({store.directory})")
80-
return
81-
82133
table = Table("collection", "version", "item", "kind", "status", "size")
83-
for coll in sorted(p for p in root.iterdir() if p.is_dir()):
84-
for ver in sorted(p for p in coll.iterdir() if p.is_dir()):
85-
if (ver / "bank_index").exists():
86-
table.add_row(
87-
coll.name,
88-
ver.name,
89-
"bank_index",
90-
"index",
91-
"-",
92-
_human(_dir_size(ver / "bank_index")),
93-
)
94-
pangenomes = ver / "pangenomes"
95-
if pangenomes.exists():
96-
for h5 in sorted(pangenomes.glob("*.h5")):
134+
if root.exists():
135+
for coll in sorted(p for p in root.iterdir() if p.is_dir()):
136+
for ver in sorted(p for p in coll.iterdir() if p.is_dir()):
137+
if (ver / "bank_index").exists():
97138
table.add_row(
98139
coll.name,
99140
ver.name,
100-
h5.stem,
101-
"pangenome",
141+
"bank_index",
142+
"index",
102143
"-",
103-
_human(h5.stat().st_size),
104-
)
105-
dbg = ver / "dbg"
106-
if dbg.exists():
107-
for d in sorted(p for p in dbg.iterdir() if p.is_dir()):
108-
status = "complete" if _dbg_complete(d) else "partial"
109-
table.add_row(
110-
coll.name, ver.name, d.name, "dbg", status, _human(_dir_size(d))
144+
_human(_dir_size(ver / "bank_index")),
111145
)
146+
pangenomes = ver / "pangenomes"
147+
if pangenomes.exists():
148+
for h5 in sorted(pangenomes.glob("*.h5")):
149+
table.add_row(
150+
coll.name,
151+
ver.name,
152+
h5.stem,
153+
"pangenome",
154+
"-",
155+
_human(h5.stat().st_size),
156+
)
157+
dbg = ver / "dbg"
158+
if dbg.exists():
159+
for d in sorted(p for p in dbg.iterdir() if p.is_dir()):
160+
status = "complete" if _dbg_complete(d) else "partial"
161+
table.add_row(
162+
coll.name,
163+
ver.name,
164+
d.name,
165+
"dbg",
166+
status,
167+
_human(_dir_size(d)),
168+
)
169+
170+
lroot = local_root(store.directory)
171+
if lroot.exists():
172+
for d in sorted(p for p in lroot.iterdir() if p.is_dir()):
173+
status = "complete" if is_built(store.directory, d.name) else "partial"
174+
table.add_row(
175+
LOCAL_COLLECTION, "-", d.name, "local", status, _human(_dir_size(d))
176+
)
177+
178+
if not table.rows:
179+
console.print(f"Cache is empty ({store.directory})")
180+
return
181+
112182
console.print(table)
113183
console.print(f"Total: {_human(_dir_size(store.directory))} ({store.directory})")
114184

@@ -127,12 +197,22 @@ def clear(ctx, target, yes) -> None:
127197
metapang cache clear GTDB_refseq
128198
metapang cache clear GTDB_refseq@2.0.0
129199
metapang cache clear GTDB_refseq@2.0.0:s__Abiotrophia_defectiva
200+
metapang cache clear local
201+
metapang cache clear local:my_pangenome
130202
"""
131203
store = _cache(ctx)
132204

133205
if target is None:
134206
to_remove = [store.directory]
135207
label = f"the entire cache ({store.directory})"
208+
elif (parsed := parse_collection_name_version(target))[0] == LOCAL_COLLECTION:
209+
_, _, pangenome = parsed
210+
if pangenome:
211+
to_remove = [local_dir(store.directory, pangenome)]
212+
label = f"local:{pangenome}"
213+
else:
214+
to_remove = [local_root(store.directory)]
215+
label = "all local pangenomes"
136216
else:
137217
name, version, pangenome = parse_collection_name_version(target)
138218
base = store.collection_directory / name
@@ -181,19 +261,19 @@ def _remove(path: Path) -> None:
181261
@cache.command()
182262
@click.argument("target")
183263
@click.option(
184-
"--index/--no-index", "do_index", default=True, help="Fetch the release bank index."
264+
"--index/--no-index",
265+
"do_index",
266+
help="Fetch the release bank index __placeholder__.",
185267
)
186268
@click.option(
187269
"--dbg/--no-dbg",
188270
"do_dbg",
189-
default=True,
190-
help="Fetch the de Bruijn graph of each named species.",
271+
help="Fetch the de Bruijn graph of each named species __placeholder__.",
191272
)
192273
@click.option(
193274
"--pangenome/--no-pangenome",
194275
"do_pangenome",
195-
default=True,
196-
help="Fetch the .h5 pangenome of each named species.",
276+
help="Fetch the .h5 pangenome of each named species __placeholder__.",
197277
)
198278
@click.option("--force", "-f", is_flag=True, help="Re-download even if already cached.")
199279
@click.pass_context

‎metapang/app/metapang.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ def metapang_epilog() -> str:
8787
"metapang cache": [
8888
{
8989
"name": "Commands",
90-
"commands": ["list", "path", "fetch", "clear"],
90+
"commands": ["list", "path", "add", "fetch", "clear"],
9191
},
9292
],
9393
}

0 commit comments

Comments
 (0)