Skip to content

Commit

Permalink
Updated all parity metrics to difference metrics
Browse files Browse the repository at this point in the history
  • Loading branch information
@denysgerasymuk799
denysgerasymuk799 committed Jan 29, 2024
1 parent 3e97130 commit ddf756d
Show file tree
Hide file tree
Showing 8 changed files with 152 additions and 49 deletions.
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Virny decouples model auditing into several stages, including: **subgroup metric
and **metric visualization**. This gives data scientists more control and flexibility to use the library
for model development and monitoring post-deployment.

For quickstart, look at our [Use Case Examples](https://dataresponsibly.github.io/Virny/examples/Multiple_Models_Interface_Use_Case/).
For quickstart, look at our [use case examples](https://dataresponsibly.github.io/Virny/examples/Multiple_Models_Interface_Use_Case/), [an interactive demo](https://huggingface.co/spaces/denys-herasymuk/virny-demo), and [a demonstrative Jupyter notebook](https://huggingface.co/spaces/denys-herasymuk/virny-demo/blob/main/notebooks/ACS_Income_Demo.ipynb).


## 🛠 Installation
Expand All @@ -61,6 +61,7 @@ pip install virny
* [Introduction](https://dataresponsibly.github.io/Virny/)
* [API Reference](https://dataresponsibly.github.io/Virny/api/overview/)
* [Use Case Examples](https://dataresponsibly.github.io/Virny/examples/Multiple_Models_Interface_Use_Case/)
* [Interactive Demo](https://huggingface.co/spaces/denys-herasymuk/virny-demo)


## 💡 Features
Expand Down
Binary file added docs/diagrams/RAI_Toolkit_Architecture_v8.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
101 changes: 101 additions & 0 deletions docs/diagrams/drawio_files/RAI_Toolkit_Architecture_v8.drawio
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
<mxfile host="app.diagrams.net" modified="2024-01-26T00:27:43.769Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" etag="tG84Hw-QYM0eef_zB49R" version="23.0.2" type="device">
<diagram id="hlvisfvuWRFe17uFSuDf" name="Page-1">
<mxGraphModel dx="1331" dy="1001" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="zDrE7s1ACpcQbbRIbE0_-3" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="166.25" y="282.5" width="112" height="48" as="geometry" />
</mxCell>
<mxCell id="zDrE7s1ACpcQbbRIbE0_-4" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="166" y="352.5" width="112" height="48" as="geometry" />
</mxCell>
<mxCell id="zDrE7s1ACpcQbbRIbE0_-1" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="166" y="422.5" width="112" height="48" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-27" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=13;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-7" target="JsNyH0pSIVGMAEf0iLmD-11" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-7" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#cce5ff;strokeColor=#7EA6E0;" parent="1" vertex="1">
<mxGeometry x="334.75" y="282.5" width="205.25" height="185" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-8" value="&lt;div style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b style=&quot;&quot;&gt;Subgroup Error &lt;br&gt;&lt;/b&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b style=&quot;&quot;&gt;Analyzer&lt;/b&gt;&lt;/font&gt;&lt;/div&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#F5F5F5;fontColor=#333333;strokeColor=#666666;" parent="1" vertex="1">
<mxGeometry x="350" y="407" width="175.5" height="45" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-9" value="&lt;div style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot; color=&quot;#333333&quot; face=&quot;Verdana&quot;&gt;&lt;b style=&quot;&quot;&gt;Subgroup Variance &lt;br&gt;&lt;/b&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot; color=&quot;#333333&quot; face=&quot;Verdana&quot;&gt;&lt;b style=&quot;&quot;&gt;Analyzer&lt;/b&gt;&lt;/font&gt;&lt;/div&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;" parent="1" vertex="1">
<mxGeometry x="349.62" y="349.5" width="175.5" height="45" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-28" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=13;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-11" target="JsNyH0pSIVGMAEf0iLmD-17" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-11" value="&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Overall&lt;br&gt;Metrics Matrix&lt;/b&gt;&lt;/font&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#D6C5F0;strokeColor=#9673A6;" parent="1" vertex="1">
<mxGeometry x="590" y="345" width="130" height="60" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-44" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=14;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-17" target="JsNyH0pSIVGMAEf0iLmD-19" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-45" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=14;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-17" target="JsNyH0pSIVGMAEf0iLmD-21" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-17" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#cce5ff;strokeColor=#7EA6E0;" parent="1" vertex="1">
<mxGeometry x="770" y="335" width="140" height="80" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-46" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=14;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-19" target="NrY6HeF-G9XXoupHhg0z-18" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-19" value="&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Overall&lt;br&gt;Metrics Matrix&lt;/b&gt;&lt;/font&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#D6C5F0;strokeColor=#9673A6;" parent="1" vertex="1">
<mxGeometry x="972" y="298" width="135" height="60" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-47" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fontSize=14;" parent="1" source="JsNyH0pSIVGMAEf0iLmD-21" target="NrY6HeF-G9XXoupHhg0z-18" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="JsNyH0pSIVGMAEf0iLmD-21" value="&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Disparity &lt;br&gt;Metrics Matrix&lt;/b&gt;&lt;/font&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#D6C5F0;strokeColor=#9673A6;" parent="1" vertex="1">
<mxGeometry x="972" y="396" width="135" height="60" as="geometry" />
</mxCell>
<mxCell id="eiP0nZE08H6xvK_XeUf1-1" style="rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" target="JsNyH0pSIVGMAEf0iLmD-7">
<mxGeometry relative="1" as="geometry">
<mxPoint x="270" y="320" as="sourcePoint" />
</mxGeometry>
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-6" value="&lt;font face=&quot;Verdana&quot;&gt;&lt;b style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot;&gt;Base Flow &lt;br&gt;Dataset &lt;/font&gt;&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=7;" parent="1" vertex="1">
<mxGeometry x="166.75" y="280.5" width="110" height="50" as="geometry" />
</mxCell>
<mxCell id="eiP0nZE08H6xvK_XeUf1-2" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" target="JsNyH0pSIVGMAEf0iLmD-7">
<mxGeometry relative="1" as="geometry">
<mxPoint x="280" y="375" as="sourcePoint" />
</mxGeometry>
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-11" value="&lt;font face=&quot;Verdana&quot;&gt;&lt;b style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot;&gt;Config Yaml&lt;/font&gt;&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=7;" parent="1" vertex="1">
<mxGeometry x="162" y="360" width="120" height="30" as="geometry" />
</mxCell>
<mxCell id="eiP0nZE08H6xvK_XeUf1-4" style="rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=-0.008;entryY=0.668;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" target="JsNyH0pSIVGMAEf0iLmD-7">
<mxGeometry relative="1" as="geometry">
<mxPoint x="270" y="430" as="sourcePoint" />
<mxPoint x="330" y="400" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-13" value="&lt;font face=&quot;Verdana&quot;&gt;&lt;b style=&quot;font-size: 15px;&quot;&gt;&lt;font style=&quot;font-size: 15px;&quot;&gt;Models &lt;br&gt;Config&lt;/font&gt;&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=1;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=7;movable=1;rotatable=1;deletable=1;editable=1;connectable=1;" parent="1" vertex="1">
<mxGeometry x="181.75" y="421.5" width="80" height="50" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-15" value="&lt;font style=&quot;font-size: 17px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Metric Computation&lt;br&gt;Interface&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=12;" parent="1" vertex="1">
<mxGeometry x="332.25" y="293.5" width="210" height="50" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-17" value="&lt;font style=&quot;font-size: 18px;&quot; face=&quot;Verdana&quot;&gt;&lt;b style=&quot;&quot;&gt;Metric &lt;br&gt;Composer&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=14;" parent="1" vertex="1">
<mxGeometry x="780" y="345" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="eiP0nZE08H6xvK_XeUf1-5" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="NrY6HeF-G9XXoupHhg0z-18" target="NrY6HeF-G9XXoupHhg0z-57">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-18" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#cce5ff;strokeColor=#7EA6E0;" parent="1" vertex="1">
<mxGeometry x="1168" y="335" width="140" height="80" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-19" value="&lt;font style=&quot;font-size: 18px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Metric &lt;br&gt;Visualizers&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;fontSize=14;" parent="1" vertex="1">
<mxGeometry x="1173" y="345" width="130" height="60" as="geometry" />
</mxCell>
<mxCell id="NrY6HeF-G9XXoupHhg0z-57" value="&lt;font style=&quot;font-size: 15px;&quot; face=&quot;Verdana&quot;&gt;&lt;b&gt;Static&lt;/b&gt;&amp;nbsp;&lt;b&gt;&amp;amp; Interactive&lt;/b&gt;&lt;br&gt;&lt;b&gt;Visualizations&lt;/b&gt;&lt;/font&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#D6C5F0;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="1361" y="342" width="140" height="65" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
14 changes: 7 additions & 7 deletions virny/configs/constants.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,20 @@ class ComputationMode(Enum):
EQUALIZED_ODDS_FNR = 'Equalized_Odds_FNR'
DISPARATE_IMPACT = 'Disparate_Impact'
STATISTICAL_PARITY_DIFFERENCE = 'Statistical_Parity_Difference'
ACCURACY_PARITY = 'Accuracy_Parity'
ACCURACY_DIFFERENCE = 'Accuracy_Difference'

# Stability disparity metrics
LABEL_STABILITY_RATIO = 'Label_Stability_Ratio'
LABEL_STABILITY_DIFFERENCE = 'Label_Stability_Difference'
IQR_PARITY = 'IQR_Parity'
STD_PARITY = 'Std_Parity'
IQR_DIFFERENCE = 'IQR_Difference'
STD_DIFFERENCE = 'Std_Difference'
STD_RATIO = 'Std_Ratio'
JITTER_PARITY = 'Jitter_Parity'
JITTER_DIFFERENCE = 'Jitter_Difference'

# Uncertainty disparity metrics
OVERALL_UNCERTAINTY_PARITY = 'Overall_Uncertainty_Parity'
OVERALL_UNCERTAINTY_DIFFERENCE = 'Overall_Uncertainty_Difference'
OVERALL_UNCERTAINTY_RATIO = 'Overall_Uncertainty_Ratio'
EPISTEMIC_UNCERTAINTY_PARITY = 'Epistemic_Uncertainty_Parity'
EPISTEMIC_UNCERTAINTY_DIFFERENCE = 'Epistemic_Uncertainty_Difference'
EPISTEMIC_UNCERTAINTY_RATIO = 'Epistemic_Uncertainty_Ratio'
ALEATORIC_UNCERTAINTY_PARITY = 'Aleatoric_Uncertainty_Parity'
ALEATORIC_UNCERTAINTY_DIFFERENCE = 'Aleatoric_Uncertainty_Difference'
ALEATORIC_UNCERTAINTY_RATIO = 'Aleatoric_Uncertainty_Ratio'
16 changes: 8 additions & 8 deletions virny/custom_classes/metrics_composer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
class MetricsComposer:
"""
Metric Composer class that combines different subgroup metrics to create disparity metrics
such as 'Disparate_Impact' or 'Accuracy_Parity'.
such as 'Disparate_Impact' or 'Accuracy_Difference'.
Definitions of the disparity metrics could be observed in the __init__ method of the Metric Composer:
https://github.com/DataResponsibly/Virny/blob/main/virny/custom_classes/metrics_composer.py
Expand All @@ -33,22 +33,22 @@ def __init__(self, models_metrics_dct: dict, sensitive_attributes_dct: dict):
TNR: [(EQUALIZED_ODDS_TNR, self._difference_operation)],
FPR: [(EQUALIZED_ODDS_FPR, self._difference_operation)],
FNR: [(EQUALIZED_ODDS_FNR, self._difference_operation)],
ACCURACY: [(ACCURACY_PARITY, self._difference_operation)],
ACCURACY: [(ACCURACY_DIFFERENCE, self._difference_operation)],
POSITIVE_RATE: [(DISPARATE_IMPACT, self._ratio_operation)],
SELECTION_RATE: [(STATISTICAL_PARITY_DIFFERENCE, self._difference_operation)],
# Stability disparity metrics
LABEL_STABILITY: [(LABEL_STABILITY_RATIO, self._ratio_operation),
(LABEL_STABILITY_DIFFERENCE, self._difference_operation)],
JITTER: [(JITTER_PARITY, self._difference_operation)],
IQR: [(IQR_PARITY, self._difference_operation)],
STD: [(STD_PARITY, self._difference_operation),
JITTER: [(JITTER_DIFFERENCE, self._difference_operation)],
IQR: [(IQR_DIFFERENCE, self._difference_operation)],
STD: [(STD_DIFFERENCE, self._difference_operation),
(STD_RATIO, self._ratio_operation)],
# Uncertainty disparity metrics
OVERALL_UNCERTAINTY: [(OVERALL_UNCERTAINTY_PARITY, self._difference_operation),
OVERALL_UNCERTAINTY: [(OVERALL_UNCERTAINTY_DIFFERENCE, self._difference_operation),
(OVERALL_UNCERTAINTY_RATIO, self._ratio_operation)],
ALEATORIC_UNCERTAINTY: [(ALEATORIC_UNCERTAINTY_PARITY, self._difference_operation),
ALEATORIC_UNCERTAINTY: [(ALEATORIC_UNCERTAINTY_DIFFERENCE, self._difference_operation),
(ALEATORIC_UNCERTAINTY_RATIO, self._ratio_operation)],
EPISTEMIC_UNCERTAINTY: [(EPISTEMIC_UNCERTAINTY_PARITY, self._difference_operation),
EPISTEMIC_UNCERTAINTY: [(EPISTEMIC_UNCERTAINTY_DIFFERENCE, self._difference_operation),
(EPISTEMIC_UNCERTAINTY_RATIO, self._ratio_operation)],
}

Expand Down
12 changes: 6 additions & 6 deletions virny/custom_classes/metrics_interactive_visualizer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,10 @@ def __init__(self, X_data: pd.DataFrame, y_data: pd.DataFrame, model_metrics, se
self.all_accuracy_metrics = [STATISTICAL_BIAS, TPR, TNR, PPV, FNR, FPR, F1, ACCURACY, POSITIVE_RATE]
self.all_stability_metrics = [STD, IQR, JITTER, LABEL_STABILITY]
self.all_uncertainty_metrics = [ALEATORIC_UNCERTAINTY, OVERALL_UNCERTAINTY, EPISTEMIC_UNCERTAINTY]
self.all_error_disparity_metrics = [EQUALIZED_ODDS_TPR, EQUALIZED_ODDS_TNR, EQUALIZED_ODDS_FPR, EQUALIZED_ODDS_FNR, DISPARATE_IMPACT, STATISTICAL_PARITY_DIFFERENCE, ACCURACY_PARITY]
self.all_stability_disparity_metrics = [LABEL_STABILITY_RATIO, LABEL_STABILITY_DIFFERENCE, IQR_PARITY, STD_PARITY, STD_RATIO, JITTER_PARITY]
self.all_uncertainty_disparity_metrics = [OVERALL_UNCERTAINTY_PARITY, OVERALL_UNCERTAINTY_RATIO, ALEATORIC_UNCERTAINTY_PARITY, ALEATORIC_UNCERTAINTY_RATIO,
EPISTEMIC_UNCERTAINTY_PARITY, EPISTEMIC_UNCERTAINTY_RATIO]
self.all_error_disparity_metrics = [EQUALIZED_ODDS_TPR, EQUALIZED_ODDS_TNR, EQUALIZED_ODDS_FPR, EQUALIZED_ODDS_FNR, DISPARATE_IMPACT, STATISTICAL_PARITY_DIFFERENCE, ACCURACY_DIFFERENCE]
self.all_stability_disparity_metrics = [LABEL_STABILITY_RATIO, LABEL_STABILITY_DIFFERENCE, IQR_DIFFERENCE, STD_DIFFERENCE, STD_RATIO, JITTER_DIFFERENCE]
self.all_uncertainty_disparity_metrics = [OVERALL_UNCERTAINTY_DIFFERENCE, OVERALL_UNCERTAINTY_RATIO, ALEATORIC_UNCERTAINTY_DIFFERENCE, ALEATORIC_UNCERTAINTY_RATIO,
EPISTEMIC_UNCERTAINTY_DIFFERENCE, EPISTEMIC_UNCERTAINTY_RATIO]

self.all_overall_metrics = self.all_accuracy_metrics + self.all_stability_metrics + self.all_uncertainty_metrics
self.all_disparity_metrics = self.all_error_disparity_metrics + self.all_stability_disparity_metrics + self.all_uncertainty_disparity_metrics
Expand Down Expand Up @@ -268,11 +268,11 @@ def create_web_app(self, start_app=True):
)
group_uncertainty_metrics_vw2 = gr.Dropdown(
sorted(self.all_uncertainty_disparity_metrics),
value=['Overall_Uncertainty_Parity'], multiselect=True, label="Uncertainty Disparity Metrics", info="Select uncertainty disparity metrics to display on the heatmap:",
value=[OVERALL_UNCERTAINTY_DIFFERENCE], multiselect=True, label="Uncertainty Disparity Metrics", info="Select uncertainty disparity metrics to display on the heatmap:",
)
group_stability_metrics_vw2 = gr.Dropdown(
sorted(self.all_stability_disparity_metrics),
value=['Label_Stability_Ratio', 'Std_Parity'], multiselect=True, label="Stability Disparity Metrics", info="Select stability disparity metrics to display on the heatmap:",
value=[LABEL_STABILITY_RATIO, STD_DIFFERENCE], multiselect=True, label="Stability Disparity Metrics", info="Select stability disparity metrics to display on the heatmap:",
)
group_btn_view2 = gr.Button("Submit")
with gr.Column(scale=2):
Expand Down
Loading

0 comments on commit ddf756d

Please sign in to comment.