What is the corresponding metrics with respect to sklearn?

[cnut1648]

Hi, I would like to use the torch-metrics with my project.
So far I have a lightning module that looks like this

# in init
mc = MetricCollection({
    "accuracy": Accuracy(threshold=0.0),
    "recall": Recall(threshold=0.0, num_classes=7, average='macro'),
    "precision": Precision(threshold=0.0, num_classes=7, average='macro'),
    "f1": F1(threshold=0.0, num_classes=7, average='macro'),
    "macro_auc": AUROC(num_classes=7, average='macro'),
})
self.metrics: ModuleDict[str, MetricCollection] = ModuleDict({
    f"{phase}_metric": mc.clone()
    for phase in ["train", "valid", "test"]
})

# in each *_step_end
# phase is train or valid or test
metrics = self.metrics[f"{phase}_metric"]
# update metrics
metrics(output['prob'], output['label'])

# in each *_epoch_end
# phase is train or valid or test
# (bsz, num_classes)
probs = torch.cat([o['prob'] for o in outputs])
# (bsz, )
labels = torch.cat([o['label'] for o in outputs])
metrics = self.metrics[f"{phase}_metric"]

for metric_name, metric in metrics.items():
    self.log(f"{phase}/epoch/{metric_name}", metric.compute(), logger=True, prog_bar=True,
             # on_step if train; on_epoch if not train
             on_step=False, on_epoch=True)

I would like to match the results of sklearn, namely, in the *_epoch_end I would like metrics to match with the following

 y_true = labels.detach().cpu().numpy()
y_prob = probs.detach().cpu().numpy()
y_pred = probs.argmax(-1).detach().cpu().numpy()
macro_auc = roc_auc_score(y_true, y_prob, average="macro", multi_class="ovo")
weight_auc = roc_auc_score(y_true, y_prob, average="weighted", multi_class="ovo")
precision, recall, f1, _ = precision_recall_fscore_support(y_true, y_pred, average='macro')
acc = accuracy_score(y_true, y_pred)

But for now for some reason although the others matched, macro auc does not. I am wondering if this is because there are something I missed. Thank you!

[SkafteNicki]

It should be due to the implementation of AUROC only supporting one-vs-rest and you are trying to compare to the one-vs-one implementation in sklearn.