Heatmap PR2 - Refactoring of the heatmaps and addition of utils file.#1087
Heatmap PR2 - Refactoring of the heatmaps and addition of utils file.#1087EngCaioFonseca wants to merge 8 commits intodevfrom
Conversation
- Fix cache_facade.py: Add missing cache_conn.commit() call - Re-enable 3 codebase query tables in db_init.py - Re-enable codebase query imports in index_callbacks.py - Update queries to use materialized views for better performance: - cntrb_per_file_query: Use augur_data.explorer_cntrb_per_file - pr_files_query: Use augur_data.explorer_pr_files - repo_files_query: Use augur_data.explorer_repo_files
EngCaioFonseca
moved this from Backlog
to Testing | Waiting on PR response | Blocked
in Aspen Project Board
EngCaioFonseca
force-pushed
the
heatmap-pr2-refactor-with-utils
branch
from
33a7035 to
3255ec7
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| return data_array, repo_ids[0] | ||
| """Populate repository dropdown.""" | ||
| logging.debug(f"{VIZ_ID} - repo_dropdown called with {len(repo_ids) if repo_ids else 0} repos") | ||
| return hu.build_repo_dropdown_data(repo_ids, rfq) |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'build_repo_dropdown_data' member (no-member)
| """Populate directory dropdown based on selected repository.""" | ||
| logging.debug(f"{VIZ_ID} - Loading directories for repo_id={repo_id}") | ||
| if repo_id is None: | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| """Populate directory dropdown based on selected repository.""" | ||
| logging.debug(f"{VIZ_ID} - Loading directories for repo_id={repo_id}") | ||
| if repo_id is None: | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| # Wait for cache with timeout | ||
| if not hu.wait_for_cache(rfq.__name__, [repo_id]): | ||
| logging.error(f"{VIZ_ID} - Cache timeout for repo {repo_id}") | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| # Wait for cache with timeout | ||
| if not hu.wait_for_cache(rfq.__name__, [repo_id]): | ||
| logging.error(f"{VIZ_ID} - Cache timeout for repo {repo_id}") | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| return df_file | ||
|
|
||
| # Step 2: Aggregate by directory using new utility | ||
| df_dir = hu.aggregate_ids_by_directory(df_file, directory, "cntrb_ids") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'aggregate_ids_by_directory' member (no-member)
|
|
||
| df_file : Pandas Dataframe | ||
| Dataframe with file and related cntrb_id information | ||
| df_dir = hu.map_ids_to_dates(df_dir, df_last, "cntrb_ids", "cntrb_id", "created_at") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'map_ids_to_dates' member (no-member)
| # get all of the files in the directory or nested in folders in the directory | ||
| df_dynamic_directory = df_file[df_file["file_path"].str.startswith(directory)] | ||
| # Step 4: Create time matrix using new utility | ||
| result = hu.create_time_matrix(df_dir, df_actions, "created_at", filter_action="PR Opened") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'create_time_matrix' member (no-member)
| from pages.utils.graph_utils import get_graph_time_values, color_seq | ||
| import time | ||
|
|
||
| from app import augur |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused augur imported from app (unused-import)
| import time | ||
|
|
||
| from app import augur | ||
| import app |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused import app (unused-import)
| return data_array, repo_ids[0] | ||
| """Populate repository dropdown.""" | ||
| logging.debug(f"{VIZ_ID} - repo_dropdown called with {len(repo_ids) if repo_ids else 0} repos") | ||
| return hu.build_repo_dropdown_data(repo_ids, rfq) |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'build_repo_dropdown_data' member (no-member)
| """Populate directory dropdown based on selected repository.""" | ||
| logging.debug(f"{VIZ_ID} - Loading directories for repo_id={repo_id}") | ||
| if repo_id is None: | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| """Populate directory dropdown based on selected repository.""" | ||
| logging.debug(f"{VIZ_ID} - Loading directories for repo_id={repo_id}") | ||
| if repo_id is None: | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| # Wait for cache with timeout | ||
| if not hu.wait_for_cache(rfq.__name__, [repo_id]): | ||
| logging.error(f"{VIZ_ID} - Cache timeout for repo {repo_id}") | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| # Wait for cache with timeout | ||
| if not hu.wait_for_cache(rfq.__name__, [repo_id]): | ||
| logging.error(f"{VIZ_ID} - Cache timeout for repo {repo_id}") | ||
| return [hu.TOP_LEVEL_DIRECTORY], hu.TOP_LEVEL_DIRECTORY |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
| return df_file | ||
|
|
||
| # Step 2: Aggregate by directory using new utility | ||
| df_dir = hu.aggregate_ids_by_directory(df_file, directory, "reviewer_ids") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'aggregate_ids_by_directory' member (no-member)
|
|
||
| df_file : Pandas Dataframe | ||
| Dataframe with file and related reviewer_id information | ||
| df_dir = hu.map_ids_to_dates(df_dir, df_last, "reviewer_ids", "cntrb_id", "created_at") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'map_ids_to_dates' member (no-member)
| # get all of the files in the directory or nested in folders in the directory | ||
| df_dynamic_directory = df_file[df_file["file_path"].str.startswith(directory)] | ||
| # Step 4: Create time matrix using new utility | ||
| result = hu.create_time_matrix(df_dir, df_actions, "created_at", filter_action="PR Opened") |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'create_time_matrix' member (no-member)
| from pages.utils.graph_utils import get_graph_time_values, color_seq | ||
| import time | ||
|
|
||
| from app import augur |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused augur imported from app (unused-import)
| import time | ||
|
|
||
| from app import augur | ||
| import app |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused import app (unused-import)
| return data_array, repo_ids[0] | ||
| """Populate repository dropdown.""" | ||
| logging.warning(f"{VIZ_ID} - repo_dropdown called with repo_ids={repo_ids}") | ||
| result = hu.build_repo_dropdown_data(repo_ids, rfq) |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'build_repo_dropdown_data' member (no-member)
|
|
||
| return directories, "Top Level Directory" | ||
| logging.warning( | ||
| f"{VIZ_ID} - directory_dropdown returning {len(directories)} directories, default={hu.TOP_LEVEL_DIRECTORY}" |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
E1101: Module '8Knot.pages.codebase.visualizations.heatmap_utils' has no 'TOP_LEVEL_DIRECTORY' member (no-member)
- Add explorer_repo_files.sql with file path data query - Add explorer_cntrb_per_file.sql with contributor/reviewer per file query - Add explorer_pr_files.sql with pull request files query
EngCaioFonseca
force-pushed
the
heatmap-pr2-refactor-with-utils
branch
from
0c95a1e to
2d4a212
Compare
- Add Codebase navigation link to main topbar - Link points to /codebase route with heatmap visualizations - Enables access to contribution, contributor, and reviewer file heatmaps
Create heatmap_utils.py with 9 core helper functions to reduce code duplication across the three heatmap visualizations: - wait_for_cache(): Wait for cache with timeout - retrieve_cached_data(): Retrieve data from cache - clean_file_path(): Clean repo-specific path patterns - get_directories(): Extract unique directories from files - aggregate_by_directory(): Aggregate file data by directory - create_heatmap_figure(): Generate Plotly heatmap - prepare_file_df(): Prepare file dataframe - create_time_range_df(): Create date range dataframe - safe_merge(): Safe pandas merge Refactor all three heatmap visualization files to use these utilities: - cntrb_file_heatmap.py: Shows contributor activity per file/directory - contribution_file_heatmap.py: Shows PR activity per file/directory - reviewer_file_heatmap.py: Shows reviewer activity per file/directory Benefits: - Reduced code duplication - Consistent behavior across all heatmaps - Easier maintenance and testing - Improved UI labels and descriptions
- Add detailed logging to repo_dropdown and directory_dropdown to debug issue - Remove top-level docstrings from all three heatmap files - Restore original long-form PopoverBody descriptions for contributor/reviewer heatmaps - Restore original short-form description for contribution heatmap - Add back NOTE comment explaining query logic in repo_files_query.py
EngCaioFonseca
force-pushed
the
heatmap-pr2-refactor-with-utils
branch
from
2d4a212 to
dd76d94
Compare
1 participant
This PR refactors the heatmaps to make them work accordingly with mods made in PR1, and adds a heatmap_utils.py file in order to optimize the working of the heatmaps.
The PR also handles null data.
Generative AI disclosure