Repository navigation
Minor Revisions #93
Description
@abuszydlik I've started gathering some thoughts for the minor revisions:
Notation
Indeed some things that could be improved:
- Equation (1): reviewer bsoG takes issue with the regularization term, but current notation is consistent with what we’ve used in the past. Could still double-check if it’s also consistent with related literature (e.g. Wachter) and adjust if necessary.
- Equation (2): AI review asks for explicit notation for ridge penalty. Not sure it’s really needed by it would be: $\frac{1}{n_{\text{CE}}} \sum_{i=1}^{n_{\text{CE}}} \left( \mathcal{E}{\theta}(\mathbf{x^+},y^+)^2 + \mathcal{E}\theta(\mathbf{x}_{\text{CE}}^\prime,y^+)^2 \right)$.
- Equation (3): reviewer bsoG takes issue with
$y$ vs$y^+$ . In my mind, the energy function is a function of the variable y, where we follow the functional form in Grathwohl/ECCCo which specifies it with respect to$y^+$ (where$y^+$ is a value that$y$ can take). Perhaps it’s worth being more explicit about that. Note: the AI review also points this out, so to avoid confusion we let’s indeed consistently use$y^+$ , e.g.$E(x,y^+)$ . - Tau=0.5: this is indeed a bit misleading for the multi-class case … under the hood, CE.jl transforms 0.5 -> 1/3 (three classes), 0.5 -> 1/4 (four classes), etc … For the special case of 0.5, this corresponds simply to whether or not the logit for the target class is higher than all other logits. Perhaps we could add an explanation in the appendix (not sure we will be able to re-render though). #doublecheck
- Definition 3.1: AI review claims that integral cannot reach 1, but the definition merely states that it approaches 1. In any case, I was never fully confident about this notation, so instead we could just say:
- (Faithfulness) $P(\mathbf{x}^\prime \in \mathcal{X}\theta|\mathbf{y}^+) = 1 - \delta$, where $\delta$ is a small value (e.g. $\delta=0.01$) and $\mathcal{X}\theta|\mathbf{y}^+$ denotes the model’s conditional posterior distribution over inputs (as in ECCCo paper Def. 4.1: https://arxiv.org/pdf/2312.10648)
- (Plausibility)
$P(\mathbf{x}^\prime \in \mathcal{X}|\mathbf{y}^+) = 1 - \delta$ , where$\mathcal{X}|\mathbf{y}^+$ is the conditional distribution of inputs in the target class (Def. 2.1 in ECCCo paper).
- Equation (4): AI review points out that “Adversarial loss notation and linkage: The text defines but does not clearly link to in the loss, and (presumably ) is not defined. It remains ambiguous whether and what happens if no iterate satisfies the -constraint; is also omitted from the main objective’s call to .” — that’s an easy fix and would indeed make things clearer.
- Actionability: AI review says the Bayesian perspective is misleading. I personally have found it helpful to think about it like this and believe that it’s technically correct, but happy to rephrase in terms of masking.
- All other things pointed out by the AI reviews are either fixable typos (e.g. should not use ECCo anywhere anymore), minor additions (Algorithms 1) or provided in the appendix (e.g. Gaussian kernel with length-scale 0.5 for MMD). For distance-based IP we use the
$\ell_1$ -norm, consistent with the conventional choice for the cost metric. (We tested different metrics in the ECCCo paper but couldn’t establish any meaningful patterns.)
Weakness
- Reviewer Nb3w mistakenly states that (1) we are proposing a new CE method; (2) we are not checking if our approach also works for other CE methods than ECCCo. They are the first reviewer to misunderstand this, but we should pre-empt further confusion:
- For (1), make it explicit that we are not proposing a new CE method, but in fact addressing a gap in the literature: we use existing CE methods during training to improve models.
- For (2), make it more explicit that while we test multiple CE methods (including surrogate-based REVISE), we find ECCCo to yield the best results (point to appendix). This is intuitive, because maximally faithful explanations should be most useful for feedback: we cannot expect the model to learn as much from counterfactual explanations that largely depend surrogate models.
- Reviewer Nb3w therefore claims our evaluation is insufficient. To pre-empt this:
- Again, be even more explicit about the fact that we have indeed investigated other CE generators, but that we do not aim to be agnostic with respect to the generator, for reasons mentioned above. I think we just need to be clearer and more explicit about our argument and motivation here and point to the appendix for reference.
- The reviewer’s suggestion about adjusting the narrative in the introduction could actually be helpful, although I wouldn’t go over board with adjustments since other reviewers seem to have understood the point.
- Weak baseline: here we should perhaps tone done “greatly improved” robustness (or explicitly reference the “compared to weak baselines” in that context). With the SaTML audience in mind, we should be careful to state our goals and claims here:
- Principal goal is explainability and actionability, robustness is a byproduct.
- Since CT involvEs multiple objectives, we do not aim to outperform state-of-the-art approaches that target any specific objective, nor do we claim that CT achieves this. Specifically, we do not (cl)aim to beat JEMs with respect to their generative capacity, SOTA robust neural networks with respect to robustness or (quasi-)Bayesian neural networks with respect to uncertainty quantification. Literature has shown that all of these tend to correlate (explaining some of our positive findings), but we frame our objective very specifically in the context of explainability and algorithmic recourse.
Ablation
Working on experiments whenever I have stable connection to DelftBlue. I believe we should have some additional space so hopefully this could then be added as a final subsection of Section 4 (possibly could combine it with the current section on hyperparameters).
Activity
abuszydlik commentedon Sep 21, 2025
For now, the two weaknesses are addressed in the new manuscript; will get to notation later/tomorrow.
abuszydlik commentedon Sep 22, 2025
@pat-alt regarding Equation (1): I think it would be fine to leave as is, but we could rename reg -> cost to make it a bit clearer that we care about the cost of moving towards x'. This is how it was in the Macrodynamics and ECCCo papers, and it is a bit more consistent with other works we're citing [e.g., for Wachter et al. it is d(x_i, x'), for REVISE it is c(x, Gθ(z)) ]
abuszydlik commentedon Sep 22, 2025
regarding Equation (2): added the formula since we have space and being explicit may be good. Not sure if other points from the AI review about explaining how Ridge penalty works are needed... it feels a bit disrespectful towards a technically-minded audience to introduce very basic ML concepts
abuszydlik commentedon Sep 22, 2025
regarding Equation (3): all mentions of the energy term now include y+ instead of y
abuszydlik commentedon Sep 22, 2025
Clarified tau = 0.5 directly in text; I think it fits there well
abuszydlik commentedon Sep 22, 2025
Good point about Definition 3.1; I adjusted it according to your suggestion and made some changes to the surrounding text.
abuszydlik commentedon Sep 22, 2025
@pat-alt regarding Equation (4): I adjusted the description but would appreciate if you take a second look
abuszydlik commentedon Sep 22, 2025
Regarding actionability: I also think the Bayesian perspective is correct and helpful; I assume the AI remark was included because we do not explicitly set a Dirac prior on the feature. I slightly elaborated on this topic.
abuszydlik commentedon Sep 22, 2025
I went through the other AI comments and adjusted where I could find the problems. May be good if you skim through to double-check if notation is consistent now @pat-alt. Cannot find where it identified "ECCo"...
abuszydlik commentedon Sep 22, 2025
pat-alt commentedon Sep 23, 2025
Very much agree
pat-alt commentedon Sep 23, 2025
Then reason I've used reg instead of cost here is that regularization does not necessarily need to address only costs (e.g. ECCCo penalizes energy and conformal set size, since it's targeting different objectives). But if that's not clear enough, then I'm also happy to just change it to cost
pat-alt commentedon Sep 23, 2025
Regarding ECCCo vs. others I see you’ve already added quite a bit in the experimental setup section. I’ve added a little more detail in the results section, but I’m not sure there’s any specific finding in the appendix we can easily highlight (Figures 3-6, for example, all demonstrate the finding, but they’re all too large too include, I would say). Technically, we do have the space of course, to include Figure 3 (see snapshot below), for example - but I’m not sure how useful it is to use a single page to show grid results for a single dataset? At the same time, maybe it can serve as a reference for the remainder of the hyperparm discussion and to highlight the extent of our grid searches. What’s your view @abuszydlik
abuszydlik commentedon Sep 23, 2025
I think it is fine as is!
abuszydlik commentedon Sep 24, 2025
I'm very happy with the manuscript! I summarised a few more details that existed only in the appendix, following Cynthia's suggestions and additionally briefly outlining computing resources.
Would be great if you can still give it a full read, Patrick (maybe after skimming through the IEEE formatting guidelines to triple-check it after me) :-)
pat-alt commentedon Sep 25, 2025
Regarding format guidelines:
\paragraphin the appendix to format the reviews which involve deeply nested paragraphs. Guidelines say that this command is not allowed incompsocmode, but I don't think that affects us.algorithmis not an approved float environment, so I've turned it into a figure (don't think this was an issue with our macro dynamics paper, but let's play it safe)