How to add custom_metric

[hanhan (hanhanwu)]

Hello FLAML team,

I'm trying to do HPO with customized metrics.
My customized metric does have the same signature ask this documentation requested: https://microsoft.github.io/FLAML/docs/reference/automl/

Here's my code:

But tried multiple settings, not sure where to add this customized metric in automl... Got this type of error:

Do you know how to solve this problem? It's kinda urgent for me... How to hear from you soon... Thank you!

[Chi Wang (sonichi)]

Hello FLAML team,

I'm trying to do HPO with customized metrics. My customized metric does have the same signature ask this documentation requested: https://microsoft.github.io/FLAML/docs/reference/automl/

Here's my code:

But tried multiple settings, not sure where to add this customized metric in automl... Got this type of error:

Do you know how to solve this problem? It's kinda urgent for me... How to hear from you soon... Thank you!

Please change the signature into:

def custom_metric(
    X_val, y_val, estimator, labels,
    X_train, y_train, weight_val=None, weight_train=None,
    *args,
):

We'll update the documentation.

[hanhan (hanhanwu)]

Thank you Chi Wang (@sonichi) ! Now the code can finish HPO work. But when I'm trying to check the output, got this error:

Looks like the result is the same as an empty AutoML instance... Do you know how to resolve this? Thank you!

[Chi Wang (sonichi)]

Has at least one trial finished within the budget? Could you check the output of the AutoML run?

[hanhan (hanhanwu)]

Ohh! It's because my divisor in the customized metric was 0.... Now I got the output. Thank you very much Chi Wang (@sonichi) !