> This survey about how people use their phones while on the toilet only had ten questions,
The problem with surveys asking about behaviors is that it gathers unreliable data.
That's why you get contradictions where men/women say they want features x,y,z in a partner but they end up being attracted to something else.
Or you survey people about 2 food menu choices and the respondents answer with the "healthier" choice but the real data from point-of-sale system shows they ordered the unhealthier option.
Or they say they look at X 1st and then Y 2nd on websites but eye-tracking software doesn't agree.
It's not that people are dishonest; it's that people are terrible at self-reporting how they actually behave. The counter-intuitive results need to come from hidden cameras, point-of-sale data, functional MRI scans, etc instead of questionnaires.
If the survey is asking about nouns (what they have) instead of verbs (what they think they do), it's more reliable. For example, a survey asking "what's your shoe size?" or "how many children do you have?" doesn't cause people to give contradictory answers.
I'm also skeptical of websites using "focus groups" asking people "would you pay for X if it had Y?" type of questions. You may get better answers by just trying different things and parsing server logs to measure actual user engagement. People don't know what they want, or they'll tell you what you want to hear. When it comes time to actually make a decision about spending money, their actions contradict the survey answers.
> The problem with surveys asking about behaviors is that it gathers unreliable data.
But the point of doing the surveys is not to gather reliable data -- it's to generate clickbait headlines that will go viral. Even if the sample size is 10 and p is hacked to bits, people are going to share your "Android Users 3 Times More Likely to Drop Their Phone in Toilet" blog post.
Can't good survey question design, mitigate this? Haven't psychologists and researchers been doing this for decades and solved the problem of asking verb questions properly?
Because at the end of the day you have to ask what people want, right?
[+] [-] jasode|10 years ago|reply
The problem with surveys asking about behaviors is that it gathers unreliable data.
That's why you get contradictions where men/women say they want features x,y,z in a partner but they end up being attracted to something else.
Or you survey people about 2 food menu choices and the respondents answer with the "healthier" choice but the real data from point-of-sale system shows they ordered the unhealthier option.
Or they say they look at X 1st and then Y 2nd on websites but eye-tracking software doesn't agree.
It's not that people are dishonest; it's that people are terrible at self-reporting how they actually behave. The counter-intuitive results need to come from hidden cameras, point-of-sale data, functional MRI scans, etc instead of questionnaires.
If the survey is asking about nouns (what they have) instead of verbs (what they think they do), it's more reliable. For example, a survey asking "what's your shoe size?" or "how many children do you have?" doesn't cause people to give contradictory answers.
I'm also skeptical of websites using "focus groups" asking people "would you pay for X if it had Y?" type of questions. You may get better answers by just trying different things and parsing server logs to measure actual user engagement. People don't know what they want, or they'll tell you what you want to hear. When it comes time to actually make a decision about spending money, their actions contradict the survey answers.
[+] [-] gee_totes|10 years ago|reply
But the point of doing the surveys is not to gather reliable data -- it's to generate clickbait headlines that will go viral. Even if the sample size is 10 and p is hacked to bits, people are going to share your "Android Users 3 Times More Likely to Drop Their Phone in Toilet" blog post.
[+] [-] doragcoder|10 years ago|reply
Because at the end of the day you have to ask what people want, right?
[+] [-] unknown|10 years ago|reply
[deleted]