Skip to content

Optimise train u #2870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RobinL wants to merge 23 commits into
base: splink_5_dev
Choose a base branch
from
Open

Optimise train u #2870

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8d3d09b
one col at a time
RobinL Jan 6, 2026
9e3083b
don't materialise blockedp airs
RobinL Jan 6, 2026
4e1ec5a
chunking
RobinL Jan 6, 2026
5a1ed0c
use defaultdict
RobinL Jan 6, 2026
777f515
slight update
RobinL Jan 6, 2026
351837d
tidy before making chunking adaptive
RobinL Jan 6, 2026
c1eec72
early stop with min count
RobinL Jan 6, 2026
2e0fa06
handle count 0 properly
RobinL Jan 6, 2026
13d7999
100x probe
RobinL Jan 6, 2026
e636b43
tests and api
RobinL Jan 6, 2026
2d40d4f
fix tests
RobinL Jan 6, 2026
b347e00
allow user to skip early exit
RobinL Jan 6, 2026
3ebb0c4
update changelog
RobinL Jan 6, 2026
398dd21
add nice logging
RobinL Jan 7, 2026
4d3bfeb
add nice logging2
RobinL Jan 7, 2026
0a5bb79
slightly clarify interface
RobinL Jan 12, 2026
43cc6fd
simplify
RobinL Jan 12, 2026
ac21f3c
better comments
RobinL Jan 12, 2026
1c45e9d
simplify
RobinL Jan 12, 2026
54ba415
curry/bind early for more succinct and readable code
RobinL Jan 12, 2026
2eba5ec
name things better
RobinL Jan 12, 2026
50ec0ae
improve comment
RobinL Jan 12, 2026
1171090
update tests for clarity
RobinL Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `estimate_u_using_random_sampling()` now estimates u probabilities using chunking and can stop early once each comparison level has enough u observations (controlled by `min_count_per_level`). This makes u estimation a lot faster and less memory intensive
- Support for chunking to allow processing of very large datasets in blocking and prediction [#2850](https://github.com/moj-analytical-services/splink/pull/2850)
- New `table_management` functions to explicitly manage table caching [#2848](https://github.com/moj-analytical-services/splink/pull/2848)

Expand Down
Loading