Introduce MinHash + LSH Prefilter

[tim-dickey]

Goal

Reduce O(N^2) pairwise comparisons by introducing MinHash signatures and locality-sensitive hashing (LSH) banding to prune candidate pairs before full Jaccard.

Scope

• Add MinHash signature generation (configurable number of permutations, e.g., 64).
• Implement banding (e.g., 16 bands of 4 hashes) to bucket potential duplicates.
• CLI flags: --minhash-perms, --lsh-bands, --prefilter (enable/disable).
• Short-circuit to full Jaccard only on candidate pairs from shared buckets.
• Benchmark comparison: with vs without prefilter (files/sec, candidate reduction rate).

Acceptance Criteria

• Prefilter reduces pairwise comparison count on synthetic large dataset (>2k files).
• Results identical (no false negatives) at chosen permutation/band config (validated on test corpus).
• Tests: candidate generation includes known duplicate pairs.

Non-Goals

• Adaptive parameter tuning (manual for now).

Future

• Dynamically adjust bands based on average signature sparsity.

[tim-dickey]

Implemented clustering output mode:

• Added cluster.py with build_clusters.
• CLI flag --clusters for table or JSON cluster output.
• JSON includes schema_version and cluster metadata (size, max_similarity, representative).
• Tests: test_clustering.py verifies cluster formation.
• README updated with cluster usage and format.
  Future improvements:
• Add intra-cluster similarity summary stats.
• Optional representative selection strategy (largest file, earliest mtime).
• Visual report (HTML) of cluster composition.