Learning patterns/Framing survey questions

From Meta, a Wikimedia project coordination wiki
A learning pattern forsurvey
Framing survey questions
problemIf people misunderstand your survey questions, they may not answer them the way you intended.
solutionWhen asking multiple-choice questions, make sure you provide choices for all the major types of answers you expect to receive, at the right level of specificity. Ask open-ended questions in a way that encourages people to elaborate, rather than give one word answers.
creatorJtmorgan
endorse
created on27 August, 2013


What problem does this solve?[edit]

When people misunderstand survey questions, they often don't answer them the way the survey creator intended. By the time the survey creator realizes this, it's usually too late.

Most people fill out surveys quickly, and don't always read every question carefully. The survey creator usually has just one chance to ask these questions—most people are not willing to fill out the same survey twice! So it's important to ask the right questions, and to ask them in the right way.

The "right questions" and the "right way" may be different for each survey, but there are some good general principles that you can follow. Above all, try to make the survey as short as possible: that way, people are more likely to concentrate on giving reliable responses.

What's the solution?[edit]

When asking multiple-choice questions, provide choices for all of the major types of answers you expect to receive, at the right level of specificity. Ask open-ended questions in a way that encourages people to elaborate rather than give one-word answers.

Examples[edit]

Asking multiple-choice questions[edit]

Take a look at an example of a multiple-choice question, and think about how it could be better (don't type!). Then click on the hint and think again. Then click on the possible solution, and so on.

Example 1


How active are you on Wikipedia?
  • 1–10 edits per month
  • 11–20 edits per month
  • 21–30 edits per month
  • 31–40 edits per month
  • 41–50 edits per month
  • 51–60 edits per month
  • More than 60 edits per month


  • Perhaps the wording of "How active are you on Wikipedia?" could relate more clearly to the choices you give them.
  • Will people have to think too hard about whether they make 30–40 or 41–50 edits per month? And will it really be useful for you to know this fine distinction?
  • Can you identify where repeated words could be removed to make it simpler to read?



Now have a rest and come back for the next set of examples. We need you to be fresh!

Asking open-ended questions[edit]

Open-ended questions can draw out more detailed, personal responses from people in their own words. But people may skip over these because they don't have anything to add or they don't feel like writing a lot.

But you can increase both the number and the quality of their open-ended responses by phrasing the questions in ways that encourage storytelling.

Again, let's take an example and think how it could be improved:

Example 2


Has the kind of editing you do changed since you first joined Wikipedia?



Example 3


What's the most difficult thing about editing Wikipedia?


This is asking people to come up with something quite abstract; but people generally find it easier to remember concrete things than abstract things, and specific examples can produce especially useful data for you.

And does "difficult" mean a negative experience, or one that was challenging but ultimately positive? Asking someone how they were affected by an experience (how they felt) will usually lead to more accurate and useful feedback.

So, think how you might change the question to make it easy for them to recall something specific in their experience? We've come up with three ways, and our possible solution is only three words longer, too!

Endorsements[edit]

See also[edit]

Related patterns[edit]

Other resources[edit]

other resources (links to surveys, articles about survey design, handy tools) related to this pattern