Welcome to Orcmid's Lair, the playground for family connections, pastimes, and scholarly vocation -- the collected professional and recreational work of Dennis E. Hamilton

Recent Items
 
Stop Lying About Surveys!
 
... Staying Afloat ...
 
Slipping Into the Water
 
Orcmid's Lair Disruption
 
Steve Wozniak on Designing and Building Apples 1 and 2.
 
It Was the Best of Webs, the Worst of Webs
 
Ignaz Semmelweis and the State of Software Reliability
 
A Turn at Live Blogging
 
There Are No Ordinary Moments
 
Shared Identity Surprises - How Not To Do Single Sign-On

Stop Lying About Surveys!

I must remember to have that “Lies to Phone Surveyors” T-shirt made up.  This isn’t about that though.

{tags: orcmid system incoherence confirmable experience trustworthiness online surveys}

Today’s lament is about on-line surveys and questionnaires.  I’ve been receiving invitations to several of those quirky things lately.

Here’s the lie I’m bitching about:

• Any survey that says it will only take 5 minutes, will really take 10 or more. 
    
• Any invitation that simply says “quick survey” like the one I answered today will likely take 20 minutes or longer, assuming that I finish it.
     
• And in general, any survey invitation for which the surveyors haven’t actually bothered to sample how long it really takes is simply a lie.

I understand the goal is to have responses to the survey, but making up a statement about how short and quick it is and having that be a lie is a demonstration of incompetence if not untrustworthiness.  It will also lead me to stop at some point, with a comment that “you are giving me too much work and I stopped/abandoned/cancelled here.”  Usually there is no place for such a comment and all the surveyor will receive is a shockingly low response rate.  That might be good: fewer responses to evaluate. 

Here are some clues for the unclued:

Any survey with 9 web pages and an earned-value progress bar (1/9, 2/9, ..) is not a quick survey.

Any survey with essay questions is not a 5–10 minute survey and will never be a quick survey unless you’re seriously seeking thoughtless responses.

Asking questions because you can and for which the responses have no conceivable use, and also demonstrate the surveyor's laziness, is particularly offensive.

Having no way for respondents to question the questions or to report difficulties or even why the response is incomplete may be denying you the most important information you need about the peculiarity (including high-absence/low-quality) of the responses you are getting.

By the way, I stop phone survey responses and other conversations when the short time I am promised is exceeded and there is no clue how much overtime it will take.  This is not like writing some complex program for the first time.  It is possible to provide grounded estimates on how long a questionnaire will take and to do the right thing when the answer is not one you are willing to give to the respondent up front.

---

I first noticed this situation while I was working on my since-abandoned M.Sc in IT.  It was amazing how many dissertation projects involved performing surveys.  There were often these anxious on-line posts to fellow students to respond to the survey in their IT-professional capacities.  I never once received an accurate time estimate, and the most suspicious ones were the ones that were unbelievably optimistic (two minutes and  “only a few minutes” being the winners here until I ran into “quick” yesterday).  Since this is a good opportunity to re-learn that IT is an empirical activity, I was always helpful in pointing out to the surveyors that they should have timed the questionnaire a few times with family and friends or fellow students.

Another feature of that setting was the desire to use on-line survey systems of one kind or another.  Bugs, browser and plug-in incompatibilities, and numerous other exciting adventures were to be found in those cases.  The biggest take-away is that the use of radio buttons that require one and only one choice are almost always going to be a problem, especially if some selection is mandatory.

The questionnaire that I just completed was notable for asking essay questions.  When I ran into that the first time late last night, I stopped and went to bed.  Refreshed this morning, I had my thoughts in order and I went back to the questionnaire.  I had to start over, of course, and I did that because I wanted to respond.  It was an application to participate in an user council and I want to do that.  There was also no mechanism for feedback, I haven’t blogged in some time, and, well, here we are.

[update 2006–12–17T23:09Z Hmm.  Something was nagging me about the quick survey that wasn’t quick at all.  I know what it was.  I received an e-mail invitation. It said that all I need to do was complete a quick survey.   At the very end of that statement it mentioned that I would hear if I’d been selected (that should have been the moment of truth right there).  Then, when we get to the essay questions, it turns out I am making an application.  So, apart from being annoyed that I also just wasted a considerable time not ordering on-line IMAX tickets for Happy Feet, a completely-separate but attitude challenging experience, I realize that the whole non-invitation invitation thing and the disguised solicitation of applications for screening is really sucky.  Hey, I wonder if I just dooced myself as a volunteer user-council member?  So I’ll stop while I’m so pissed off at how my day is going.]

posted by orcmid at 11/17/2006 02:06:48 PM