I love Mechanical Turk. We used it at my agency where we were having a big problem with time clock fraud (a.k.a. people were asking their friends to punch them out and then leaving hours early)
So I built a very simple add on to our timeclock (which is essentially an ELO touch screen and a PC) and had it use a web cam to take a picture of each person punching. Then we created a program that showed turk users a person's badge picture and then the picture taken at the punch and asked "Is this the same person?" (with answers of Yes/No/Maybe) Doing that virtually eliminated the problem overnight.
I agree with the author that M.T. won't completely change the market. But what it will do is allow people to monitize their spare seconds and in doing so make a lot of the "no-brainer" grunt work that companies have to do a lot cheaper
You just inspired me for a CAPTCHA replacement. Simply asking questions can't work because we either have to have a preformulated DB of questions and answers (which could be cracked easily) or have to generate the question from a simple enough formula that a computer could derive the answer. How about, instead, we generate CAPTCHA questions on-demand from humans? Or, even better, don't bother with an automated Turing Test at all: just pay people to have a short conversation with the possible bot, and then say whether they think they're human or not.
The Calorie Tracking example (in the blog posting) illustrates that Mechanical Turk relies heavily on tremendous inequities in wealth.
A rich, fat, lazy person snaps a picture of his lavish meal with a fancy iPhone, and rather than take a brief moment to ponder the calories, … uploads it for, not just one, but three other people to ponder and estimate the calories and report back. The most dissimilar answer is rejected; that person is not compensated.
This only works if there is a very large pool of very poor people who are willing to do these mundane tasks for (fractions of) pennies. These “workers” certainly can’t afford an iPhone; they probably can’t even afford the meal they are looking at.
You're right that "relies heavily on tremendous inequities in wealth," but not in the way you argue.
People don't spend time on MT for the money. Ok, some people do, but I believe the overwhelming motivation is the same motivation behind casual gaming.
It's just a fun thing you do because you're bored and it's like a little internet scavenger hunt.
Most Turkers are from North America, IIRC. It's more about the cognitive surplus of western laborers versus their third-world counterparts, which is related to their earning power.
Edit: Not sure why your comment was getting downvoted. I voted it back up -- it's an insightful and appropriate comment!
I would say its actually a case of efficiencies using batching rather than the pay discrepancy.
Taking the time out from what you're doing to write down what you ate and how many calories it was incurs a lot of "transaction" costs. Its distracting.
However, if you're on your computer and are sitting there doing many of these one right after another its a lot faster and doesn't require changing your focus.
This is the exact same reason all time management books recommend answering emails in batches.
Even if it were true (and unfortunately it isn't), don't say this like it's a bad thing. Every business which relies on inequities of wealth works to eliminate them. If it succeeds, it will either adapt to a world without inequities or die.
On a similar vain I'd like to point out that Mechanical Turk jobs can only be submitted by people from the US. I can't see any reason for this. It does tend to reinforce the image of inequality.
As a developer of a calorie tracking service, also with an iPhone app in the App Store, I would point out that the example with calorie tracking does not work in RL.
1. There are hundreds of food chains in the US, with widely varying recipes, serving sizes, and nutritional contents. Even being familiar with the subject area, I generally won't be able to recognize a food. Even if I see a burger photo - how would I know the brand, flavor, whether it has cheese in it, etc. The error rate will be huge, rendering the service useless, and even worse - misleading.
2. You cannot estimate portion size on a photo, not having anything to compare with. This also results in significant errors.
I am not arguing against other uses of MT, just pointing out that this is a not very well thought through example.
MT is applicable only for very rudimentary tasks, requiring absolutely zero qualifications and training. There are fewer such tasks around than it looks at the first glance, as this example demonstrates.
Mechanical Turk is generally used for things that machines still can't do with nearly the accuracy that humans can. Image recognition, facial recognition, translation, dictation, etc.
No matter how much hardware you throw at those problems, you can't (yet) match what a human brain can do in a split second. I don't think that's depressing.
Cheer up. People are not cheaper than machines, they're just better at some tasks. You'd have to complete tens of thousands of those stupid little tasks to pay your rent for a month...if they could be done by machine reliably, they would be.
[+] [-] TomOfTTB|17 years ago|reply
So I built a very simple add on to our timeclock (which is essentially an ELO touch screen and a PC) and had it use a web cam to take a picture of each person punching. Then we created a program that showed turk users a person's badge picture and then the picture taken at the punch and asked "Is this the same person?" (with answers of Yes/No/Maybe) Doing that virtually eliminated the problem overnight.
I agree with the author that M.T. won't completely change the market. But what it will do is allow people to monitize their spare seconds and in doing so make a lot of the "no-brainer" grunt work that companies have to do a lot cheaper
[+] [-] derefr|17 years ago|reply
[+] [-] unknown|17 years ago|reply
[deleted]
[+] [-] unknown|17 years ago|reply
[deleted]
[+] [-] dxjones|17 years ago|reply
A rich, fat, lazy person snaps a picture of his lavish meal with a fancy iPhone, and rather than take a brief moment to ponder the calories, … uploads it for, not just one, but three other people to ponder and estimate the calories and report back. The most dissimilar answer is rejected; that person is not compensated.
This only works if there is a very large pool of very poor people who are willing to do these mundane tasks for (fractions of) pennies. These “workers” certainly can’t afford an iPhone; they probably can’t even afford the meal they are looking at.
[+] [-] shalmanese|17 years ago|reply
From: http://behind-the-enemy-lines.blogspot.com/2008/03/mechanica... and http://asc-parc.blogspot.com/2008/07/mechanical-turk-demogra... 82% of turkers are from the US, Canada or the UK and over 75% have a Bachelor’s degree or higher.
[+] [-] jfarmer|17 years ago|reply
People don't spend time on MT for the money. Ok, some people do, but I believe the overwhelming motivation is the same motivation behind casual gaming.
It's just a fun thing you do because you're bored and it's like a little internet scavenger hunt.
Most Turkers are from North America, IIRC. It's more about the cognitive surplus of western laborers versus their third-world counterparts, which is related to their earning power.
Edit: Not sure why your comment was getting downvoted. I voted it back up -- it's an insightful and appropriate comment!
[+] [-] lionheart|17 years ago|reply
Taking the time out from what you're doing to write down what you ate and how many calories it was incurs a lot of "transaction" costs. Its distracting.
However, if you're on your computer and are sitting there doing many of these one right after another its a lot faster and doesn't require changing your focus.
This is the exact same reason all time management books recommend answering emails in batches.
[+] [-] radu_floricica|17 years ago|reply
[+] [-] dejb|17 years ago|reply
Any thoughts and any ways to get around this?
[+] [-] asciilifeform|17 years ago|reply
Actually, almost all forms of employment do so as well.
[+] [-] sergeo|17 years ago|reply
1. There are hundreds of food chains in the US, with widely varying recipes, serving sizes, and nutritional contents. Even being familiar with the subject area, I generally won't be able to recognize a food. Even if I see a burger photo - how would I know the brand, flavor, whether it has cheese in it, etc. The error rate will be huge, rendering the service useless, and even worse - misleading.
2. You cannot estimate portion size on a photo, not having anything to compare with. This also results in significant errors.
I am not arguing against other uses of MT, just pointing out that this is a not very well thought through example.
MT is applicable only for very rudimentary tasks, requiring absolutely zero qualifications and training. There are fewer such tasks around than it looks at the first glance, as this example demonstrates.
[+] [-] unknown|17 years ago|reply
[deleted]
[+] [-] tectonic|17 years ago|reply
[+] [-] ars|17 years ago|reply
People are now cheaper than machines :(
This is a reversal of 2000 years of progress.
[+] [-] paulgb|17 years ago|reply
No matter how much hardware you throw at those problems, you can't (yet) match what a human brain can do in a split second. I don't think that's depressing.
[+] [-] ryanwaggoner|17 years ago|reply