[ENH] Replace prts metrics

[aryanpola]

Reference Issues/PRs

Addresses #2066

What does this implement/fix? Explain your changes.

Implementation of Precision, Recall and F1-score metrics.

PR checklist

For all contributions

• I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you.
• The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

[aeon-actions-bot]

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ $\color{#FEF1BE}{\textsf{enhancement}}$ ].
I have added the following labels to this PR based on the changes made: [ $\color{#264F59}{\textsf{benchmarking}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

• Run pre-commit checks for all files
• Run mypy typecheck tests
• Run all pytest tests and configurations
• Run all notebook example tests
• Run numba-disabled codecov tests
• Stop automatic pre-commit fixes (always disabled for drafts)
• Disable numba cache loading
• Push an empty commit to re-run CI checks

[MatthewMiddlehurst]

Some of these functions look like they should be private/protected. In the tests please run both the new and current functions to ensure the output is the same (using pytest)

[aryanpola]

Changes made from the previous commit:

1. Added a function for flattening lists of lists into a list.
2. Removes overlapping functions.
3. Changed the metric calculations from averaging(w.r.t cardinality) to global(w.r.t overlapping positions) calculation of real and pred ranges.

[aryanpola]

@MatthewMiddlehurst Please let me know if the test cases are fine, I can separate them into different functions if necessary.

[TonyBagnall]

thanks for this, its really helpful. We will look again next week

[aryanpola]

Write tests comparing the old and new versions to ensure they have the same output. previous comment

This is still missing, right?

The updated test compares the results of the previous version (_binary.py) and the new version (range_metrics).

Also, let me know if you want me to remove some test cases, these cover all the edge cases.

[MatthewMiddlehurst]

Do you know why the new functions have different parameters from the old ones? Don't know if this was explained previously.

[aryanpola]

Do you know why the new functions have different parameters from the old ones? Don't know if this was explained previously.

1. I've added the beta parameter in fscore function, which follows the same logic from prts package.
2. I also added the udf_gamma function which allows the user to alter the input for gamma parameter.
3. Since alpha has no role in precision calculation, took the liberty to not include it in the list of parameters.

I've just changed the names of parameters.
cardinality -> gamma
bias -> bias_type
y_true -> y_real
y_pred -> y_pred

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl Do you want me to convert all the test cases in a format that compares the output from prts and current implementation?
example in test_metrics.py (line 298)

[SebastianSchmidl]

If I understand you correctly, yes. The new implementation should produce the same results as the old implementation for all cases.

Please excuse that I am slow to respond and cannot spend much time here. I will be busy with other obligations until the middle of summer this year.

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl , I’ve updated the test cases so that they now compare the current implementation with the prts package for all parameter configurations.

[SebastianSchmidl]

Please add a check for the soft-dependency "prts" to those test where you call into the existing range-based metrics to make the CI happy; similar to the existing tests:

aeon/aeon/benchmarking/metrics/anomaly_detection/tests/test_ad_metrics.py

Lines 228 to 232 in 8770d2f

	@pytest.mark.skipif(
	not _check_soft_dependencies("prts", severity="none"),
	reason="required soft dependency prts not available",
	)
	def test_range_based_p_range_based_r_auc_perfect_hit():

[aryanpola]

Please add a check for the soft-dependency "prts" to those test where you call into the existing range-based metrics to make the CI happy; similar to the existing tests:

aeon/aeon/benchmarking/metrics/anomaly_detection/tests/test_ad_metrics.py

Lines 228 to 232 in 8770d2f

	@pytest.mark.skipif(
	not _check_soft_dependencies("prts", severity="none"),
	reason="required soft dependency prts not available",
	)
	def test_range_based_p_range_based_r_auc_perfect_hit():

Done. Let me know if there's anything else to be changed.

[MatthewMiddlehurst]

Thanks a lot for addressing my concerns. I'm happy for this to go in the with testing done.

I would not close the issue yet, the next step is to remove the old functions and prts as a dependency. Unfortunately, this would involve changing a lot of the testing you have done here again, but the main purpose of the output comparison is to smooth the transition anyway.

[SebastianSchmidl]

Just some small documentation changes

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl
Further changes to be made:

1. Remove prts from all_extras
2. Update tests to compare the implementation output with hard-coded values (already done in previous commits).
3. Remove the older function, which includes aeon\benchmarking\metrics\anomaly_detection\_binary.py. aeon\benchmarking\metrics\anomaly_detection\_continuous.py still uses prts as a dependency. What should be done with that?

This should be everything?

[SebastianSchmidl]

@MatthewMiddlehurst @SebastianSchmidl Further changes to be made:

1. Remove prts from all_extras
2. Update tests to compare the implementation output with hard-coded values (already done in previous commits).
3. Remove the older function, which includes aeon\benchmarking\metrics\anomaly_detection\_binary.py. aeon\benchmarking\metrics\anomaly_detection\_continuous.py still uses prts as a dependency. What should be done with that?

The functions in _continuous.py can use the new implementation within aeon.

This should be everything?

Could you add this information to the corresponding issue? This should be addressed in separate PRs.

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl Further changes to be made:

1. Remove prts from all_extras
2. Update tests to compare the implementation output with hard-coded values (already done in previous commits).
3. Remove the older function, which includes aeon\benchmarking\metrics\anomaly_detection\_binary.py. aeon\benchmarking\metrics\anomaly_detection\_continuous.py still uses prts as a dependency. What should be done with that?

The functions in _continuous.py can use the new implementation within aeon.

This should be everything?

Could you add this information to the corresponding issue? This should be addressed in separate PRs.

3 different PR's for 3 mentioned changes?

[aryanpola]

@MatthewMiddlehurst Please merge this PR so I can pull the changes and update the test cases to hard-coded ones.

[aryanpola]

@MatthewMiddlehurst Please merge this PR so I can pull the changes and update the test cases to hard-coded ones.

@MatthewMiddlehurst A gentle reminder.