JSMA implementation is incorrect

[carlini]

The JSMA implementation in this repository is significantly less effective than as reported in prior work. In Table XIV JSMA appears to succeed 76% of the time. When I run the code in the repository, I get a very similar result: 72.3%.

This should be somewhat surprising given that prior work reports JSMA succeeds above 90% of the time with the same distortion bound of 10% of pixels changed. Unfortunately, Papernot et al. (2016) uses a bound of 14.5% so is not directly comparable, but, in Carlini & Wagner (2016) we re-implemented JSMA and found a 90% attack success rate at 78. Indeed, when I run the JSMA attack from CleverHans on this exact same network (using the approach discussed in #3) I reach 95% attack success rate.

When investigating this just a bit more, I observe that when attacking a solid-black image targeting each possible target label 0 through 9 the code in this repository returns a substantially different adversarial example than the code from CleverHans returns, which this code is based on.