problems about poverty worst r

[ypwang61]

Thanks for your implementation of fish, but when I test poverty with algorithm erm, the worst r is just about 0.2, while the performance noted in wilds is about 0.5 (https://wilds.stanford.edu/leaderboard/), is there some problem?

[YugeTen]

Hi -- thanks for bringing this to my attention! I am working on a fix for this. If you are interested, I think the difference between performance can be explained as follows:

Our work was developed on WILDS v1.0, where the metric of interest for Poverty is average r instead of worst r. You can compare the current (v2.0) hyperparameter configuration of wilds vs. ours and notice some differences (for e.g. they use a step lr scheduler whereas we don't, and they use 8 groups per batch whereas we use random batches). I believe that once we implement the same hyperparameters as the current version of WILDS benchmark for Poverty we will observe similar performance with ERM using our implementation.

Stay tuned!!