[WIP] Implement current MIREX key scoring method and warn if using old one

[stefan-baumann]

This PR aims to solve the issue discussed in #337 in a backwards-compatible manner.
An additional argument allow_descending_fifths was added to the key.weighted_score and key.evaluate methods, which is set to False by default (and thus being backwards-consistent), but raises a warning stating that this method is no longer used for MIREX. If set to True, descending fifth errors are given a score of 0.5, matching current MIREX scoring.

I'd love to get feedback on whether the maintainers approve of the way I implemented this change and whether they might have ideas how to improve keyword naming or the added documentation.

If the feedback is positive, I'll extend the tests to cover this behaviour as well.

[craffel]

I think this is a reasonable approach, but we should plan to switch the default to True eventually if it matches the new convention.

[craffel]

@bmcfee @jpauwels @iansimon PTAL

[stefan-baumann]

but we should plan to switch the default to True eventually if it matches the new convention

In that case, we should probably change the warning message to reflect that. Maybe something like this?

The selected key scoring method does not match that currently used by MIREX. To use the same method, specify allow_descending_fifths=True. The default behaviour will change to allow_descending_fifths=True in the future.

Potentially substituting in the future with a specific version number.

[craffel]

but we should plan to switch the default to True eventually if it matches the new convention

In that case, we should probably change the warning message to reflect that. Maybe something like this?

The selected key scoring method does not match that currently used by MIREX. To use the same method, specify allow_descending_fifths=True. The default behaviour will change to allow_descending_fifths=True in the future.

Potentially substituting in the future with a specific version number.

SGTM

[stefan-baumann]

Okay, I added the note to the warning and the docstring, specifying that the default behaviour will change in the future.

[instr3]

Hi @stefan-baumann , since both metrics have been widely used in the community, I suggest the fix to retain both evaluation metrics like:

scores["Weighted Score"] = util.filter_kwargs(
    weighted_score, reference_key, estimated_key, allow_descending_fifths=False
)

scores["Weighted Score MIREX"] = util.filter_kwargs(
    weighted_score_mirex, reference_key, estimated_key, allow_descending_fifths=True
)

I can directly modify this PR if possible. Can you remove the draft/WIP designation as mentioned by #337 (comment) ? Thanks.

[stefan-baumann]

Hi @instr3, sure, I removed the draft designation :)

Please check whether the code I wrote back then actually makes sense, and then feel free to modify it further. It's literally been years since I touched this code, and I have long since left the AI for MIR community.

Best,
Stefan

[instr3]

@stefan-baumann @craffel Thanks for the messages!

Just came back from ISMIR. I think the fix provided by stefan is clear and efficient. Since I am unable to edit this PR, I reviewed it and resolved the conflicts in another PR: #427

You may directly merge it, or tell me if there are other things need to care about.

Thanks,
Junyan