Fix: Eliminate redundant full table scans in messages and events collection by PredictiveManish · Pull Request #3444 · augurlabs/augur

PredictiveManish · 2025-12-06T17:32:17Z

Description

Moved mapping queries outside batch loops and pass pre-built mappings as parameters to processing functions, following the pattern established in #3439.
Solves #3440

Changes Made

`augur/tasks/github/messages.py`

Built issue_url_to_id_map and pr_issue_url_to_id_map once in collect_github_messages() before any batch processing
Updated process_messages() to accept mappings as parameters instead of rebuilding them
Updated process_large_issue_and_pr_message_collection() to accept and pass mappings
Increased batch size from 20 to 1000 (reduces batch overhead)

`augur/tasks/github/events.py`

Built issue_url_to_id_map and pr_url_to_id_map once in BulkGithubEventCollection.collect() before the batch loop
Updated _process_events(), _process_issue_events(), and _process_pr_events() to accept mappings as parameters
Removed redundant _get_map_from_*() calls from batch processing methods

Performance Improvement

Before: 1,000 messages -> 50 full scans of issues AND PRs tables
After: 1,000 messages -> 1 full scan of each table (50x reduction)
Before: 10,000 events -> 40 full scans total
After: 10,000 events → 1 full scan of each table (40x reduction)

This PR fixes #3440

Notes for Reviewers

Signed commits

Yes, I signed my commits.

sgoggins

One question regarding the flip from 20 to 1,000 size batch minimums.

augur/tasks/github/messages.py

shlokgilda · 2025-12-07T06:05:06Z

I just have non-blocking code quality suggestion: use a named constant like MESSAGE_BATCH_SIZE = 500 instead of a magic number

MoralCode · 2025-12-07T18:22:31Z

I just have non-blocking code quality suggestion: use a named constant like MESSAGE_BATCH_SIZE = 500 instead of a magic number

Yep, agreed

PredictiveManish · 2025-12-08T16:04:54Z

I just have non-blocking code quality suggestion: use a named constant like MESSAGE_BATCH_SIZE = 500 instead of a magic number

What to give the size 500 or 200? as @MoralCode suggested for 200

sgoggins · 2025-12-09T22:46:23Z

I just have non-blocking code quality suggestion: use a named constant like MESSAGE_BATCH_SIZE = 500 instead of a magic number

What to give the size 500 or 200? as @MoralCode suggested for 200

200

sgoggins · 2025-12-09T22:47:16Z

@MoralCode / @PredictiveManish : I'm rerunning the failed end to end test. Sometimes GitHub gets overwhelmed and they just timeout.

sgoggins

@PredictiveManish : May we assume you ran this locally and collected data?

augur/tasks/github/messages.py

MoralCode · 2025-12-11T14:59:16Z

May we assume you ran this locally and collected data?

Lets not assume - would rather accidentally over-test than not test at all.

MoralCode · 2025-12-12T13:36:57Z

augur/tasks/github/messages.py

+        # Build mappings once before processing any messages
+        # create mapping from issue url to issue id of current issues
+        issue_url_to_id_map = {}
+        issues = augur_db.session.query(Issue).filter(Issue.repo_id == repo_id).all()
+        for issue in issues:
+            issue_url_to_id_map[issue.issue_url] = issue.issue_id
+
+        # create mapping from pr url to pr id of current pull requests
+        pr_issue_url_to_id_map = {}
+        prs = augur_db.session.query(PullRequest).filter(PullRequest.repo_id == repo_id).all()
+        for pr in prs:
+            pr_issue_url_to_id_map[pr.pr_issue_url] = pr.pull_request_id
+


Above you rely on some functions for these mappings, Is this code just a duplicate of the code in those functions? If those functions are useful elsewhere and not tied in anywhere maybe they can be useful utility functions that can be imported here too?

Overall id recommend splitting this PR so that the changes to events.py can be merged without being held up by the larger question of refactoring ( ref #3345) that this file brings up

@PredictiveManish : I agree with @MoralCode that splitting the events.py from the other refactoring would make this more straightforward to merge and test.

Covered changes in events.py in #3479

sounds good, will focus on that PR first. This one is still on hold until we figure out how best to share the same function that was proposed in events.py

yeah, for events.py I have created new PR as suggested for split.

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

MoralCode · 2026-01-08T14:32:28Z

augur/tasks/github/messages.py

+        # Build mappings once before processing any messages
+        # create mapping from issue url to issue id of current issues
+        issue_url_to_id_map = {}
+        issues = augur_db.session.query(Issue).filter(Issue.repo_id == repo_id).all()
+        for issue in issues:
+            issue_url_to_id_map[issue.issue_url] = issue.issue_id
+
+        # create mapping from pr url to pr id of current pull requests
+        pr_issue_url_to_id_map = {}
+        prs = augur_db.session.query(PullRequest).filter(PullRequest.repo_id == repo_id).all()
+        for pr in prs:
+            pr_issue_url_to_id_map[pr.pr_issue_url] = pr.pull_request_id
+


sounds good, will focus on that PR first. This one is still on hold until we figure out how best to share the same function that was proposed in events.py

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

…ection Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

Signed-off-by: Manish Tiwari <manish.tiwari.09@zohomail.in>

sgoggins self-assigned this Dec 6, 2025

sgoggins previously approved these changes Dec 6, 2025

View reviewed changes

augur/tasks/github/messages.py Outdated Show resolved Hide resolved

PredictiveManish dismissed sgoggins’s stale review via 7d975f3 December 7, 2025 17:55

sgoggins added the database Related to Augur's unifed data model label Dec 9, 2025

sgoggins previously approved these changes Dec 9, 2025

View reviewed changes

augur/tasks/github/messages.py Show resolved Hide resolved

MoralCode requested changes Dec 12, 2025

View reviewed changes

PredictiveManish added a commit to PredictiveManish/augur that referenced this pull request Dec 18, 2025

Fix: Splitting PR augurlabs#3444 only events.py changes included

d81173b

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

ABrain7710 previously approved these changes Jan 7, 2026

View reviewed changes

MoralCode requested changes Jan 8, 2026

View reviewed changes

MoralCode added discussion Seeking active feedback, usually for items under active development pending changes PRs that have review comments that have been added but havent been addressed in a while labels Jan 8, 2026

PredictiveManish added a commit to PredictiveManish/augur that referenced this pull request Jan 9, 2026

Fix: Splitting PR augurlabs#3444 only events.py changes included

1f128fc

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

PredictiveManish force-pushed the map-once-update branch from e699932 to 40ec167 Compare January 9, 2026 20:35

PredictiveManish added 3 commits January 21, 2026 13:15

Fix: Eliminate redundant full table scans in messages and events coll…

1b3e0d4

…ection Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

Reduced batch overhead

9824374

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

Removed magic number with constant

ac28a75

Signed-off-by: PredictiveManish <manish.tiwari.09@zohomail.in>

sgoggins force-pushed the map-once-update branch from 40ec167 to ac28a75 Compare January 21, 2026 19:16

sgoggins added this to the v0.93.0 milestone Jan 21, 2026

MoralCode added the waiting This change is waiting for some other changes to land first label Jan 27, 2026

MoralCode marked this pull request as draft January 27, 2026 03:08

MoralCode modified the milestones: v0.93.0, v0.94.0 Mar 6, 2026

Merge branch 'main' into map-once-update

81161b8

Signed-off-by: Manish Tiwari <manish.tiwari.09@zohomail.in>

PredictiveManish dismissed sgoggins’s stale review via 81161b8 March 7, 2026 16:39

PredictiveManish dismissed ABrain7710’s stale review via 81161b8 March 7, 2026 16:39

Conversation

PredictiveManish commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

augur/tasks/github/messages.py

augur/tasks/github/events.py

Performance Improvement

Uh oh!

sgoggins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shlokgilda commented Dec 7, 2025

Uh oh!

MoralCode commented Dec 7, 2025

Uh oh!

PredictiveManish commented Dec 8, 2025

Uh oh!

sgoggins commented Dec 9, 2025

Uh oh!

sgoggins commented Dec 9, 2025

Uh oh!

sgoggins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MoralCode commented Dec 11, 2025

Uh oh!

MoralCode Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

sgoggins Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

PredictiveManish Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

MoralCode Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

PredictiveManish Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

MoralCode Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PredictiveManish commented Dec 6, 2025 •

edited

Loading

`augur/tasks/github/messages.py`

`augur/tasks/github/events.py`