How does the three-stage detection work, and which stage is slowest?

[amasen02]

The README mentions a three-stage detection pipeline (size, fast-hash, full-hash). I have ~500k files and scans are taking a while.

1. What does each stage do exactly?
2. At which stage does DupeSweep bail out early, and what percentage of files typically make it to stage 3?
3. Is there a way to skip stage 2 and go straight to full-hash for accuracy, or to only run stages 1-2 for a quick estimate?

[amasen02]

Here is how the three-stage pipeline works, with typical real-world percentages.

Stage 1 — Size grouping (instant)

DupeSweep groups all files by their exact byte size. Any file with a unique size is immediately excluded — it cannot have a duplicate. This is a pure in-memory sort with no I/O.

Typical reduction: 70–90% of files are eliminated here because most files have unique sizes (logs, photos with timestamps, etc.).

Stage 2 — Fast hash (partial read)

For the remaining size-matched groups, DupeSweep reads only the first 16 KB of each file and hashes it. Files where even the first 16 KB differ are split into separate groups and eliminated.

Typical reduction: another 80–90% of the stage-1 survivors are eliminated here. For 500k files, by this point you are usually looking at < 5k candidates.

Typical throughput: ~100k files/sec since only 16 KB is read per file.

Stage 3 — Full hash (complete read)

Only files that matched on both size and partial hash reach this stage. DupeSweep reads and hashes the complete content using BLAKE3 to produce a cryptographically reliable comparison.

Typical reduction: very few false positives remain from stage 2; stage 3 confirms or rejects them. For 500k files, usually < 1k files reach this stage.

Flags

• Skip stage 2 for maximum accuracy: use . This is slower but eliminates the (rare) possibility of two files that share the same first 16 KB but differ later.
• Stages 1-2 only for a quick estimate: use . This reports likely duplicates without reading full file contents and is very fast, but may have false positives.
• Progress output: shows per-stage counts in real time so you can see how many files are in each stage.