improve blocking by exluding incorrect matches

[ericbuckley]

Summary

The blocking algorithm is too liberal in including all patients from overlapping person clusters. We should include those patients that don't match because of missing key values, but we should exclude patients with different values.

Acceptance Criteria

• re-implement the get_block_data method to exclude patients that don't match the blocking criteria
• new test cases to show that patients with incorrect matches are being excluded

Details / Tasks

There are likely two ways to solve this, a) modify the query b) keep the query as is and filter the dataset in memory. It's not obvious which strategy is better from a performance standpoint, please consider both when implementing and include some documentation / comments on why that strategy was taken.

Background / Context

Take a simple blocking example of finding all patients with a matching BIRTHDATE of '1999-01-01'. Currently, we find all patient records in the database with that matching BIRTHDATE in the blocking values table, we then reduce that to a unique set of person clusters (based on what clusters those records belong to). We then return all patients in those matching person clusters, this creates 3 groups of data in the result set.

• Patients that exactly match on '1999-01-01'
• Patients that have no BIRTHDATE value
• Patients that have a BIRTHDATE value, but does not match '1999-01-01'

The last group in the result set is problematic, it clearly doesn't block, and therefore shouldn't be present in our evaluation. Removing this group from our result set, should provide more accurate calculations on cluster membership.

Related Issues/PRs

#230

[ericbuckley]

@bamader @m-goggins two questions for you both

1. Does the description above match your expectations from our discussion this week?
2. Does this change benefit BOTH the belongingness ratio algorithm and them relative match score one we're working on? If it's only the latter, I only want to implement this work post our weighted confidence implementation.