Infinite loop on bitcount not handled by tester.py

[sophiakolak]

So the bug in bitcount.py is in that n^=n-1 creates an infinite loop in the program.
When tester.py executes this program with the line return getattr(fx,algo)(*args), the tester program itself does not return.

In my own use of the Quixbugs project, I am just setting a time out that determines the cause of failure to be an infinite loop after some reasonable period, but I thought it was worth mentioning here for others using the project.

[jkoppel]

@drrckln Should we do anything?

[drrckln]

@sophiakolak Thanks for the comment and interest! Are you using tester.py in a non-manual way?

@jkoppel As a benchmark the focus of the repo is the programs and testcases, and tester.py's mostly for convenient and manual visual inspection. The py_try() function could be modified to set a timer as suggested, in the try block. Main caution is that the most the tester could report is "did not return in x arbitrary amount of time" -- it'd be inaccurate to simply judge long-running processes as an infinite loop, especially if QuixBugs is expanded on in the future.

One could allow the user to set the amount of time to wait.. but I'm not inclined to do that. It's more complex and users can simply halt the program.

Perhaps a note on bitcount might be appropriate.

[jkoppel]

@drrckln If had written it from scratch with Sophia's comment in mind, would you have done it differently? If so, how? Do you think doing it differently / addressing Sophia's comment would help QuixBugs get more users? Suppose you already had an IDE open with Quixbugs; how long would this take to address?

[drrckln]

I've changed my mind; this behavior is a silent failure, so some change should be made. Good questions! I'll need to think more about this.

[drrckln]

For infinite loops and other long-running processes, I think the tester should default to waiting a reasonable amount of time (say, 5 minutes) and terminating thereafter. It can let the user know this upfront and return an error string if too much time is taken. Of course, it'd also allow the user to specify the wait time directly. Should make it easier to expand on QuixBugs.

Probably a separate issue, but tester.py could certainly be refactored. It'd be good if tester.py was designed to be used as an importable module in addition to manual operation.

[sophiakolak]

Hi sorry, I didn't see this whole thread happening. But yes I am using this data for a research project where many patches are generated and then tested. I've made a lot of changes to it locally (to simplify my own work, and speed up the test time), and I think some of these changes would be useful for others who are using the benchmark for a similar purpose. I have a pretty busy week ahead of me, but I can fork and make a pull request next week if there is interest.

[drrckln]

Hi Sophia -- yes, please do! It's very much appreciated. Feel free to PR what you have as is; I plan to set aside some time to work on this issue (and possible refactoring) next month. Again, thanks.

…

On Sun, Feb 20, 2022 at 10:06 AM Sophia Kolak ***@***.***> wrote: Hi sorry, I didn't see this whole thread happening. But yes I am using this data for a research project where many patches are generated and then tested. I've made a lot of changes to it locally (to simplify my own work, and speed up the test time), and I think some of these changes would be useful for others who are using the benchmark for a similar purpose. I have a pretty busy week ahead of me, but I can fork and make a pull request next week if there is interest. — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALWPWIO4KT4CQGAIWM4LD3U4EUTTANCNFSM5KEXBMOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were assigned.Message ID: ***@***.***>