Problem
build_index.sh scans /data recursively for every *.nt/*.ttl/*.n3 file (plus any extension a .qlever/converters.json converter handles) and indexes them all. Today the only way to keep a specific file out of the index is to make it not match those extensions, or to have the surrounding framework/deployment special-case it.
That means a data repository (a "chamber") cannot declare its own exclusions next to its data. Concretely: a large genomics dump (genetics.nt, ~24M triples) that belongs in a dedicated store had to be worked around by shipping it only gzipped (.gz, which the scan ignores), because there was no first-class way to say "skip this file".
Proposal
Support a .qleverignore file, following the existing .qlever/ convention (like .qlever/converters.json). It would list gitignore-style path globs, relative to the directory containing the .qleverignore, used to prune matching files from the scan.
Design sketch:
- Discover
.qleverignore files anywhere under /data (excluding .git/.qlever internals, as the scan already does).
- Each pattern is matched against paths within that file's directory subtree (nearest-wins semantics, consistent with how converters resolve).
- A matched file is skipped (optionally logged at info level).
This keeps exclusion knowledge with the data that owns it, instead of leaking chamber-specific filenames into the framework/deployment layer.
Why not the existing options
- Renaming/gzipping to dodge the extension filter is a fragile side effect, not an intent-revealing declaration.
- A framework-level env var (e.g. an exclude-globs list) works but forces the general-purpose runtime to know about specific chamber files, which is a layering violation.
Acceptance criteria
- A
.qleverignore with one or more glob patterns causes matching files under its subtree to be excluded from the built index.
- Files not matched are indexed exactly as before (no behavior change when no
.qleverignore is present).
- Documented in the README alongside the
.qlever/converters.json convention.
Problem
build_index.shscans/datarecursively for every*.nt/*.ttl/*.n3file (plus any extension a.qlever/converters.jsonconverter handles) and indexes them all. Today the only way to keep a specific file out of the index is to make it not match those extensions, or to have the surrounding framework/deployment special-case it.That means a data repository (a "chamber") cannot declare its own exclusions next to its data. Concretely: a large genomics dump (
genetics.nt, ~24M triples) that belongs in a dedicated store had to be worked around by shipping it only gzipped (.gz, which the scan ignores), because there was no first-class way to say "skip this file".Proposal
Support a
.qleverignorefile, following the existing.qlever/convention (like.qlever/converters.json). It would list gitignore-style path globs, relative to the directory containing the.qleverignore, used to prune matching files from the scan.Design sketch:
.qleverignorefiles anywhere under/data(excluding.git/.qleverinternals, as the scan already does).This keeps exclusion knowledge with the data that owns it, instead of leaking chamber-specific filenames into the framework/deployment layer.
Why not the existing options
Acceptance criteria
.qleverignorewith one or more glob patterns causes matching files under its subtree to be excluded from the built index..qleverignoreis present)..qlever/converters.jsonconvention.