Create a feature for specifying non-responsive answers to questions

[rbyh]

For example, a user may want to allow a non-responsive answer such as "I don't know" to a numerical question.

Currently, you can use an optional question parameter answering_instructions (or the question text directly) to instruct the agent to return a specific response that will be validated for the question type (e.g., "Return '9999' if unknown."), but this is not ideal because the format is constrained.

This feature is probably unnecessary for multiple choice, checkbox or linear scale (labeled option) questions. But it could also be useful for free text questions to avoid any extraction steps.

[iwr-redmond]

This feature is probably unnecessary for multiple choice, checkbox or linear scale (labeled option) questions

Quoting the meta-analysis from Sischka et al (2020), page 14:

Using an [Forced Answer] design in such a situation [i.e. self-administered/online surveys] might not be the best option as this increases state reactance and, subsequently, dropout rates and faking behavior. This is especially worrisome because there are gender and ethnicity differences regarding the perception of sensitivity of certain questions [...] as well as age differences in trait reactance proneness [...] that increase the risk of selective survey dropout.

Kmetty & Stefkovics (2022) report that there is a difference between an explicit non-responsive answer option and a skip option (p667):

Explicitly offering DK and NA options caused a modest increase in item-nonresponse, thus it is not advisable to use them unless researchers find it necessary due to theoretical or other reasons. In our experiment, letting respondents to skip, but not offering them an explicit DK/NA option proved to be the best choice.

This suggests that there is a need to consider the design of available experiments with regards to the replication of human refusal and dropout behavior. If it is desirable to do build such a feature, then I reckon that a good place to start would be:

1. Add a new "allow_skip" parameter to QuestionBase
2. Add a new "explain_skip" parameter to QuestionBase (requires allow_skip = True)
3. Create a new internal question type that is a variant of QuestionYesNo requiring the agent to decide whether to answer the survey question

It would then be necessary to test whether real-world refusal rates are more closely mirrored by having include_comment made mandatory or optional, across a variety of models and agent demographic types, and whether inserting an instruction that the mandatory/optional comment will not be shared with the researchers makes aggregate responses rates more or less human-aligned. This would be similar to the <thinking> tag that was used to such extraordinary effect in Meinke et al, 2024 (preprint).

[rbyh]

Thank you @iwr-redmond ! This is really interesting and helpful. I'd love to include some guidance in docs around any options we add, and call out these areas for investigation.