Goal
Improve scalability by parallelizing pairwise similarity comparisons for large file sets while maintaining deterministic output ordering.
Scope
- Strategy: partition signatures into blocks/chunks; compute pairwise within + across blocks using process pool.
- Deterministic ordering: gather results then sort globally before output.
- Config flag:
--compare-workers (distinct from signature --workers).
- Heuristic: enable automatically if
N > threshold (e.g., >1500 files) unless overridden.
- Measure overhead vs benefit; update benchmarking to include comparison phase metrics.
Acceptance Criteria
- Parallel mode yields identical result set to serial (order after final sort).
- Throughput improvement documented for large synthetic dataset (benchmarks updated).
- Tests ensure parity (serial vs parallel) and stable ordering.
Non-Goals
- Implement LSH pruning (separate future issue).
Future
- Adaptive threshold based on average shingle set size.
Goal
Improve scalability by parallelizing pairwise similarity comparisons for large file sets while maintaining deterministic output ordering.
Scope
--compare-workers(distinct from signature--workers).N > threshold(e.g., >1500 files) unless overridden.Acceptance Criteria
Non-Goals
Future