Skip to content

Add NDCG metrics #3346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Add NDCG metrics #3346

wants to merge 3 commits into from

Conversation

exitflynn
Copy link

Adds NDCG metrics as a part of adding RecSys metrics in #2631

This PR builds on the work done in #3054 and #2632 (much thanks to ili0820 and kamalojasv181!)

The primary changes left at this stage as per my understanding would be around vectorising parts of the DCG computation as previously discussed here and here, and then docs.

I've considered different ways to address that, namely using torch.vmap (but I think that will be better to pursue once pytorch/pytorch#124423 is resolved); a batched vectorisation and a full one. Right now I think the batch-based vectorisation could be a good way to go since a full vectorisation approach could be a bit too complex (/overkill?) here, the main cause of complexity being different numbers of unique values across samples. Let me know what you think! I'm still in the early stages of working at this level and so open to feedback is all I am 🤠

@github-actions github-actions bot added the module: metrics Metrics module label Mar 19, 2025
@vfdev-5
Copy link
Collaborator

vfdev-5 commented Mar 19, 2025

Thanks for the PR @exitflynn
Concerning the vectorization, can you please check how others implement this, for example here:

The part to try to vectorize is _tie_averaged_dcg_batched. I do think vmap will ever support getitem operator.
What we can do first is to run some benchmarks on the same data between ignore_ties=True vs ignore_ties=False and see how slow _tie_averaged_dcg_batched implementation.
Next, we can try to improve it, maybe padding with zeros can help?
I haven't yet checked the details.

@vfdev-5
Copy link
Collaborator

vfdev-5 commented Mar 25, 2025

@exitflynn if you have any questions, blockers, feel free to ask here or on our discord: https://pytorch-ignite.ai/chat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: metrics Metrics module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants