Skip to content

Improve Handling of Noise Points in Clustering Algorithms (Fixes #152)#200

Closed
minimalProviderAgentMarket wants to merge 2 commits intolucasimi:mainfrom
minimalProviderAgentMarket:24d80c91-a681-42f0-bac3-a48f0a59e6b7
Closed

Improve Handling of Noise Points in Clustering Algorithms (Fixes #152)#200
minimalProviderAgentMarket wants to merge 2 commits intolucasimi:mainfrom
minimalProviderAgentMarket:24d80c91-a681-42f0-bac3-a48f0a59e6b7

Conversation

@minimalProviderAgentMarket
Copy link
Copy Markdown

@minimalProviderAgentMarket minimalProviderAgentMarket commented Jan 11, 2025

Pull Request Description

Title: Improve Handling of Points Clustered as Noise

Related Issue: Fixes #152
Issue URL: Improve the handling of points clustered as noise

Summary

This pull request addresses issue #152, which focuses on enhancing the handling of noise points produced by clustering algorithms, specifically DBSCAN. Previously, noise points were labeled as -1 and removed from further analysis, which could lead to loss of potentially valuable information. This update introduces an improved mechanism for managing these noise points, allowing for various user-defined approaches.

Changes Made

  1. Created NoiseHandlingClustering Class:

    • This new class provides a wrapper around existing clustering algorithms with a focus on handling noise points. The class supports the following strategies:
      • Singleton: Noise points are treated as individual clusters.
      • Drop: Noise points remain labeled as -1 and are not considered in analysis.
      • Group: All noise points are grouped into a single cluster, simplifying further processing.

  2. Updated Functionality:

    • The mapper_connected_components function was modified to incorporate the new clustering strategies, while ensuring that the default behavior remains intact for backward compatibility.

  3. Testing and Validation:

    • A mock clustering algorithm was implemented to test the behavior of the new noise handling strategies under various scenarios:
      • Singleton Mode: Confirms that each noise point is assigned a unique cluster.
      • Group Mode: Validates that all noise points are consolidated into one consistent label.
      • Drop Mode: Ensures original -1 labels for noise points are preserved when selected.

Next Steps

Further testing will be conducted with larger datasets to ensure consistent performance and functionality across varied scenarios. Feedback from team members regarding additional test cases or potential edge cases is welcome.

Thank you for considering this enhancement to improve the handling of noise points in our clustering implementations.

Please let me know if there are any questions or if further adjustments are needed for this pull request!

@minimalProviderAgentMarket
Add flexible noise point handling in clustering
9714369 
Add NoiseHandlingClustering wrapper class to provide control over how 
noise points (labeled as -1) are handled during clustering. The wrapper 
supports three modes:
- 'singleton': Convert each noise point to its own cluster
- 'drop': Keep noise points labeled as -1
- 'group': Group all noise points into a single cluster

Update mapper_connected_components documentation to clarify noise point 
behavior and add comprehensive unit tests for the new functionality. 
This change enables more flexible handling of outliers and noise points 
in different Mapper applications.
@minimalProviderAgentMarket
Copy link
Copy Markdown
Author

minimalProviderAgentMarket commented Jan 11, 2025

Aider:

Update Summary: Enhancements to Noise Handling in Clustering

Context: Addressed issue #152 regarding the handling of points classified as noise by clustering algorithms, specifically in relation to the DBSCAN method.

Key Changes Implemented:

  1. Parameter Validation:

    • Introduced validation for the noise_handling parameter in the NoiseHandlingClustering class constructor to ensure only valid options ('singleton', 'drop', 'group') can be used.

  2. Documentation Improvements:

    • Updated the docstring of the NoiseHandlingClustering class to specify that 'singleton' is the default mode and to outline the performance implications of each noise handling option.

  3. Enhanced Noise Handling Logic:

    • Refined the logic for handling noise points in NoiseHandlingClustering:
      • 'drop' Mode: Preserves noise points with label -1.
      • 'group' Mode: Groups all noise points into a single cluster.
      • 'singleton' Mode: Assigns each noise point its own unique cluster label.
    • Ensured that the labels_ attribute is always a numpy array for consistent behavior across methods.

  4. Clustering Algorithm Adjustment:

    • Adjusted the parameters for the DBSCAN algorithm to ensure proper identification of noise points by setting min_samples=2.

  5. Comprehensive Testing:

    • Added tests to validate the behavior of the noise handling options and ensure correct functionality, including edge cases for purely noise datasets.

Test Results:

  • All tests are now passing successfully, confirming the effective implementation of the above changes.

These improvements enhance the flexibility and accuracy of noise handling in clustering, supporting better data analysis and representation.

@minimalProviderAgentMarket
Here's a git commit message based on the diff:
39a91d5 
Optimize noise handling in clustering implementation

- Improve NoiseHandlingClustering performance with numpy operations
- Add validation for noise_handling parameter values
- Add detailed performance implications documentation
- Enhance code readability with clearer variable names
- Add debug output and more thorough tests for noise handling
- Fix potential edge case when all points are noise
- Use array copying to preserve original cluster labels

The changes focus on making the noise handling more robust and
efficient while maintaining the same functionality. The use of
numpy operations replaces list comprehensions for better
performance with large datasets.
@lucasimi
Copy link
Copy Markdown
Owner

lucasimi commented Apr 17, 2025

This PR is generated by a bot. I'm closing this.

@lucasimi lucasimi closed this Apr 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve the handling of points clustered as noise

2 participants

@minimalProviderAgentMarket @lucasimi