Question and Questionnaire Design Essay

Introduction. Cognitive Process and Question Types

• Four steps: interpretation, information, judgment, response
• Optimizing behavior: self-expression, self-understanding, altruism, challenge
• Weak and strong satisficing: arbitral, irrelevant
• Satisficing factors: task difficulty, respondent ability, motivation
• Open questions: rich in meaning, time-consuming
• Closed questions: comprehensive, less reliable

Rating Scale and Its Length

• Scale length affects validity, reliability, clarity, satisficing
• Dichotomous scale: clear in meaning and inconsistent
• Trichotomous scale: clear and not accurate
• Long scales (101-points): incomprehensible, prone to errors
• Midpoints lower motivation, improve reliability
• “7-point scales are optimal in many instances” Krosnick & Presser (2009, p. 283)

Labeling and Order Response

• Verbal labeling increases respondent’s satisfaction and reliability
• “Yes/no” questions: practical, promote “acquiescence” Krosnick & Presser (2009, p. 21)
• Acquiescence reasons: fatigue, politeness, personal characteristics
• Possible approaches: adjust/exclude answers, “item reversal”
• Primacy effects: attributable to “weak satisficing”, verbal
• Recency effects: attributable to short-memory, oral

No-Opinion Answers

• +“Don’t know” answers help not to appear uninformed
• + the lack of necessary information and experience
• +/-Help to escape socially undesirable images
• -DK options prevent generation of meaningful answers
• -Do not increase reliability and validity of the survey
• -Encourage satisficing by providing shortcuts

Response Bias

Reasons: favorable image, reduce punishment, increase rewards

Possible spheres: racial prejudice, political views

Ways of counteraction:

• “Bogus pipeline technique”
• “Randomized response”
• “Item count method” Krosnick and Presser (2009, p. 301)
• Anonymity: self-administered questionnaire, web surveys

Error Sources

Main sources: “comprehension” and “frailties of memory” Krosnick and Presser (2009, p. 305)

Possible ways to avoid errors:

• Interviewer’s instructions to the respondent
• Long, redundant phrases and sentences
• Explicit instructions about the importance of survey
• Simplified tasks and wording of items, decomposition

Question Order

• The main aim is to minimize errors
• Question order major types: serial, semantic
• Serial order affects: motivation, fatigue, learning
• Semantic order affects: coherency, logic
• “Funnel” order (general and specific items) Krosnick and Presser (2009, p. 313)
• Reasons for specific order: context, survey focus

Questionnaires Evaluation

Methods without data collection:

• Expert review (critiques of the questionnaire)
• The Questionnaire Appraisal System
• The Question Understanding Aid

Methods with data collection:

• Behavior coding
• Response latency

Reference

Krosnick J.A., & Presser S. (2009). Question and questionnaire design. In J.D. Wright & P.V. Marsden (Eds.), Handbook of survey research (pp. 263-314). Bingley, UK: Emerald Group Publishing.