Questionnaire design

It never ceases to amaze me how badly questionnaires are designed. And it's particularly irritating when I consider the number of times I hear from people in business that 'quant' studies are the only ones they consider valid.

Today's example: I've received a request by email from a market research company contracted by one of the universities I've attended, UCL. The request is for former students to fill out an online questionnaire. So I click on the link, and go to the first question. And here I hit the first problem: 'What do you think are the key issues facing UK higher education?' asks Question 1, with four and only four options being given (top-up fees, choice of courses, competition, government targets). There is no option for an 'other' response.

Question 1 turns out to be compulsory: I cannot move onto the next page without selecting at least one of the limited choices given, none of which I agree with. The choices, however, do not allow for any indication of what I consider to be the main issue(s): standards and staff pay. So I either lie, conforming to the questionnaire designer's expectations, in order to be able to move on to further questions, or don't complete the questionnaire. Either way, the accumulated bias created by bad design will, I suspect, not be apparent to anyone in either the market research company or the university, unless they receive an onslaught of critical emails.

And this on behalf of a university, and a good one, that ought to know better. I almost feel inclined to answer questions 'incorrectly' just to find out how many other howlers the document contains. But I don't have the time.

Postscript: A response to this post received from a director of the market research organisation concerned:

"I fully understand your anger at being unable to answer ‘Other’ in the alumni questionnaire. This was a warm up question (also used in the qualitative research) and the reason ‘Other’ was not put in was one of cost – i.e. the cost of coding up to 6,000 responses (some of which could be several pages long as we have found in similar alumni studies in the past). That said I have instructed for ‘Other’ to be put in, as well as a code that says ‘Too many issues to list in this way’."

Nice to see that a blog post about questionnaire design can have an impact on its designers and the questionnaire itself, but then he continues..."Overall while I recognize your irritation I do not think the questionnaire is quite as bad as you say on your website."

My response is that if I cannot fill in a questionnaire that I want to fill in, aimed at my specific community (UCL alumni), because of a rogue compulsory question one - and all, it appears, over a warm-up question - then that is as bad as it gets. That's 100% failure.

As Caroline Jarrett - who runs a company specialising in forms design, and has written a book on the subject - said to me, "no question in a questionnaire should be compulsory", adding "you can always throw away the results for 'other'". Caroline is adamant that 'other' is a much neglected category in the forms world, and I have to agree with her.