feat: Refactor anomaly score computation (#111)

* refactor: Refactor anomaly score computation for training * refactor: Refactor inference code to use the new normalization, fix incorrect output save when images have the same name * build: Upgrade version, update documentation * build: Add support for newer typings for python 3.9 * build: Add support for newer typings for python 3.9 * fix: Fix type alias not being imported correctly with python 3.9 * refactor: Refactor default callback name
orobix · Apr 10, 2024 · 1c04008 · 1c04008
1 parent 8a40419
commit 1c04008
Show file tree

Hide file tree

Showing 25 changed files with 217 additions and 69 deletions.
diff --git a/BREAKING_CHANGES.md b/BREAKING_CHANGES.md
@@ -1,6 +1,13 @@
 # Breaking Changes
 All the breaking changes will be documented in this file.
 
+### [2.1.0]
+
+#### Changed
+
+- Change the way anomaly scores are normalized by default, instead of using a [0-1] range with a 0.5 threshold, the scores are now normalized to a [0-1000] range with a threshold of 100, the new score represents the distance from the selected threshold, for example, a score of 200 means that the anomaly score is 100% of the threshold above the threshold itself, a score of 50 means that the anomaly score is 50% of the threshold below. This change is intended to make the scores more interpretable and easier to understand, also it makes the score independent from the min and max
+scores in the dataset.
+
 ### [2.0.0]
 
 #### Changed

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,18 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+### [2.1.0]
+
+#### Updated
+
+- Change the way anomaly scores are normalized by default, instead of using a [0-1] range with a 0.5 threshold, the scores are now normalized to a [0-1000] range with a threshold of 100, the new score represents the distance from the selected threshold, for example, a score of 200 means that the anomaly score is 100% of the threshold above the threshold itself, a score of 50 means that the anomaly score is 50% of the threshold below. 
+- Change the default normalization config name for anomaly from `min_max_normalization` to `score_normalization`.
+
+#### Fixed
+
+- Fix the output heatmaps and preditions of anomaly inference tasks not being saved properly when images belonged to
+different classes but had the same name.
+
 ### [2.0.4]
 
 #### Fixed 

diff --git a/docs/tutorials/examples/anomaly_detection.md b/docs/tutorials/examples/anomaly_detection.md
@@ -105,8 +105,8 @@ What can be useful to customize are the default callbacks:
 ```yaml
 callbacks:
   # Anomalib specific callbacks
-  min_max_normalization:
-    _target_: anomalib.utils.callbacks.min_max_normalization.MinMaxNormalizationCallback
+  score_normalization:
+    _target_: quadra.utils.anomaly.ThresholdNormalizationCallback
     threshold_type: image
   post_processing_configuration:
     _target_: anomalib.utils.callbacks.post_processing_configuration.PostProcessingConfigurationCallback
@@ -122,7 +122,7 @@ callbacks:
     _target_: quadra.callbacks.anomalib.VisualizerCallback
     inputs_are_normalized: true
     output_path: anomaly_output
-    threshold_type: ${callbacks.min_max_normalization.threshold_type}
+    threshold_type: ${callbacks.score_normalization.threshold_type}
     disable: true
     plot_only_wrong: false
     plot_raw_outputs: false
@@ -140,13 +140,13 @@ callbacks:
 
     By default lightning batch_size_finder callback is disabled. This callback will automatically try to infer the maximum batch size that can be used for training without running out of memory. We've experimented runtime errors with this callback on some machines due to a Pytorch/CUDNN incompatibility so be careful when using it.
 
-The min_max_normalization callback is used to normalize the anomaly maps to the range [0, 1] such that the threshold will become 0.5. 
+The score_normalization callback is used to normalize the anomaly maps to the range [0, 1000] such that the threshold will become 100.
 
 The threshold_type can be either "image" or "pixel" and it indicates which threshold to use to normalize the pixel level threshold, if no masks are available for segmentation this should always be "image", otherwise the normalization will use the threshold computed without masks which would result in wrong segmentations.
 
 The post processing configuration allow to specify the method used to compute the threshold, methods and manual metrics are generally specified in the model configuration and should not be changed here.
 
-The visualizer callback is used to produce a visualization of the results on the test data, when the min_max_normalization callback is used the input_are_normalized flag must be set to true and the threshold_type should match the one used for normalization. By default it is disabled as it may take a while to compute, to enable just set `disable: false`.
+The visualizer callback is used to produce a visualization of the results on the test data, when the score_normalization callback is used the input_are_normalized flag must be set to true and the threshold_type should match the one used for normalization. By default it is disabled as it may take a while to compute, to enable just set `disable: false`.
 
 In the context where many images are supplied to our model, we may be more interested in restricting the output images that are generated to only the cases where the result is not correct. By default it is disabled, to enable just set `plot_only_wrong: true`.
 
@@ -224,7 +224,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
 
 print_config: false

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "quadra"
-version = "2.0.4"
+version = "2.1.0"
 description = "Deep Learning experiment orchestration library"
 authors = [
   "Federico Belotti <[email protected]>",
@@ -82,6 +82,7 @@ segmentation_models_pytorch = { git = "https://github.com/qubvel/segmentation_mo
 anomalib = { git = "https://github.com/orobix/anomalib.git", tag = "v0.7.0+obx.1.3.0" }
 xxhash = "~3.2"
 torchinfo = "~1.8"
+typing_extensions = { version = "4.11.0", python = "<3.10" }
 
 # ONNX dependencies
 onnx = { version = "1.15.0", optional = true }

diff --git a/quadra/__init__.py b/quadra/__init__.py
@@ -1,4 +1,4 @@
-__version__ = "2.0.4"
+__version__ = "2.1.0"
 
 
 def get_version():

diff --git a/quadra/callbacks/anomalib.py b/quadra/callbacks/anomalib.py
@@ -92,7 +92,7 @@ class VisualizerCallback(Callback):
     Args:
         task: either 'segmentation' or 'classification'
         output_path: location where the images will be saved.
-        inputs_are_normalized: whether the input images are normalized (i.e using MinMax callback).
+        inputs_are_normalized: whether the input images are normalized (like when using MinMax or Treshold callback).
         threshold_type: Either 'pixel' or 'image'. If 'pixel', the threshold is computed on the pixel-level.
         disable: whether to disable the callback.
         plot_only_wrong: whether to plot only the images that are not correctly predicted.

diff --git a/quadra/configs/callbacks/default_anomalib.yaml b/quadra/configs/callbacks/default_anomalib.yaml
@@ -1,6 +1,6 @@
 # Anomalib specific callbacks
-min_max_normalization:
-  _target_: anomalib.utils.callbacks.min_max_normalization.MinMaxNormalizationCallback
+score_normalization:
+  _target_: quadra.utils.anomaly.ThresholdNormalizationCallback
   threshold_type: image
 post_processing_configuration:
   _target_: anomalib.utils.callbacks.post_processing_configuration.PostProcessingConfigurationCallback
@@ -16,7 +16,7 @@ visualizer:
   _target_: quadra.callbacks.anomalib.VisualizerCallback
   inputs_are_normalized: true
   output_path: anomaly_output
-  threshold_type: ${callbacks.min_max_normalization.threshold_type}
+  threshold_type: ${callbacks.score_normalization.threshold_type}
   disable: true
   plot_only_wrong: false
   plot_raw_outputs: false

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/cfa.yaml b/quadra/configs/experiment/generic/mnist/anomaly/cfa.yaml
@@ -11,7 +11,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
   early_stopping:
     patience: 2

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/cflow.yaml b/quadra/configs/experiment/generic/mnist/anomaly/cflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
   early_stopping:
     patience: 2

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/csflow.yaml b/quadra/configs/experiment/generic/mnist/anomaly/csflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
   early_stopping:
     patience: 3

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/draem.yaml b/quadra/configs/experiment/generic/mnist/anomaly/draem.yaml
@@ -10,7 +10,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
   early_stopping:
     patience: 20

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/fastflow.yaml b/quadra/configs/experiment/generic/mnist/anomaly/fastflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
 print_config: false
 

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/padim.yaml b/quadra/configs/experiment/generic/mnist/anomaly/padim.yaml
@@ -15,7 +15,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
 
 print_config: false

diff --git a/quadra/configs/experiment/generic/mnist/anomaly/patchcore.yaml b/quadra/configs/experiment/generic/mnist/anomaly/patchcore.yaml
@@ -15,7 +15,7 @@ datamodule:
   good_number: 9
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "image"
 
 print_config: false

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/cfa.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/cfa.yaml
@@ -11,7 +11,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
   early_stopping:
     patience: 2

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/cflow.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/cflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
   early_stopping:
     patience: 2

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/csflow.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/csflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
   early_stopping:
     patience: 3

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/draem.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/draem.yaml
@@ -10,7 +10,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
   early_stopping:
     patience: 20

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/efficient_ad.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/efficient_ad.yaml
@@ -15,7 +15,7 @@ datamodule:
   category: hazelnut
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
 
 print_config: false

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/fastflow.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/fastflow.yaml
@@ -10,7 +10,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
 print_config: false
 

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/padim.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/padim.yaml
@@ -15,7 +15,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
 
 print_config: false

diff --git a/quadra/configs/experiment/generic/mvtec/anomaly/patchcore.yaml b/quadra/configs/experiment/generic/mvtec/anomaly/patchcore.yaml
@@ -15,7 +15,7 @@ datamodule:
   category: bottle
 
 callbacks:
-  min_max_normalization:
+  score_normalization:
     threshold_type: "pixel"
 
 print_config: false