Ok, so I found it amusing that they don't include Blekko, I mean we did make this into a fun game with our three card monte game with our /monte tag in search results :-) And I suppose it doesn't really serve their interests to 'lose' to a little guy either.
That said, its an interesting experiment. And at blekko.com we've been running for over a year [1].
What we've found is that there is a huge brand bias, which is to say that if you use some search engine as your primary search engine, you tend to think that is the best one regardless of whether its Blekko, Bing, or Google (or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index).
But 'quality' is also a very subjective thing as well. So if you search for highly SEO'd categories you will find the Blekko and Bing do better than Google mostly because there is a curated input (in Blekko's case it was from day one with it's from slashtags and in Bing's they started doing outsourced curation (putatively after seeing how effective it is in Blekko :-) in some topics with their 'editors' program [2]) At some point Google will realize what Bing and Blekko have which are that the 'indexing the web' problem became the 'filtering the web' problem when the signal to noise ratio started decreasing in about 2005, and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.
If the Bing guys are reading (and I know you are) why not open up your challenge to the new kid on the block, we don't just do Blekko vs Google in our monte results :-)
"so I found it amusing that they don't include Blekko"
Just took a look at Blekko. It looks good, but you can't honestly believe the layman cares as much about it as he does about search engines coming out of MS or Google. Does the layman (MS' target audience here) even know about Blekko?
>and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.
You are aware that Google hires thousands of search engine quality raters, all over the world right? But rather than directly curating, i suspect they just use these humans to train their algorithms. There is likely some inductive step going on, in selecting which algorithm (or 'sort' of the web) is selected, and then filtered by keywords.
Secondly, most search engines, actually do also use end-user curation. I have funky enough search queries, that I often get google redirects as search results: they want to know (and record) which link i pick. Also, because of all the google-adds-cookies, google-analytics-cookies, etc, they likely have 'session-bags' full of destination links and search queries that happen to correlate.
For me personally, neither search engine did very good in this test. But that's likely because neither result set is personalized in this test. And i don't know, if Bing does that now, but even if it does, for me to switch to another search engine, means to train another search engine in my contextual world. Hell, i even notice difference in result sets in Google based on my operating system. I get different results in Linux, than i do on my iPad.
People that get a lot of spam in search results, really ought to login, and stop clicking those links.
"or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index."
Looks like you might be wrong about that.
DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, and Blekko.
As a Google shareholder, I really want to know whether Bing search results are better than Google. And I really, really want to know what the trend is over time.
However I don't think Microsoft's study is any good. They say, "In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice."
So it seems that the study participant had to think of 10 search queries, one after the other. But this is not how search works. In the real world users need to perform one query every so often, and their choice of search terms is based on an immediate need rather than what comes to mind.
A good Bing vs Google study would ask 1000 people to use a split-test search engine for their browser search box, then monitor the results over time. I've asked HN about this before and got nowhere:
http://news.ycombinator.com/item?id=3966121
So is there business opportunity here? I imagine that there must be thousands of Google shareholders willing to pay £100 per year for the results of an unbiased, ongoing study into who gives the best search results. In fact, why doesn't Google set up the study and provide such information to shareholders?
Now, all of my searches were technical; I plan to try again later with more general, "normal people" searches. Bing may do better there.
Also, good results from even google searches have become harder to come by in the last year or so. I have to work harder at my search terms. I don't know if that's Sturgeon's Law at work, or if there's room for a search engine to improve enough to become better than google.
Perhaps targeted search engines are the answer (i.e. engines's who have both their search and crawler algorithms tuned for a specific target audience) are the answer?
Of course it's going to be a draw for most searches (especially the suggested ones: "math games", "coupons", "time zones"...), they're generic and easy to find.
You can use any search engine you want to find those things. Shoot, even AltaVista was good at those kinds of searches, and that was 10 years ago.
The ones I care about—even if I'm not a programmer or technical user—are the difficult searches. What if I need to find a specific car part that only ever existed in the 1972 Ford Pinto? What if I want to find a plumber who specializes in pre-20th-century homes? What if, what if, what if.
I used searches like that and more in this comparison, and it was 100% Google. I could even tell easily why it was Google, and it was pretty clear which one it was. I still chose Google (despite being basically unblinded) because it was fundamentally better at these specific searches.
That's what bing needs to compare. But will they show it? Of course not.
Edit: I did 5 more searches all non-technical but specific and picked only based on result quality. Again 5/5 google. (for those curious, the searches were: "lego mindstorms", "Mars curiosity hi-res", "home developments in myrtle beach", "used bookstores in portland, or", "top baby names in 1982"—the last one was very cool because Google also provided links to the 5 surrounding years right below the 1982 result. It's the little things you know...)
I got 3 for Google. The results are surprisingly similar to pick a winner. Probably because I chose some easy searches (Mr. President, person name, place).
I got Bing with 2-0 with 4 draws. I used 1 technical keyword and 1 "normal people" keyword in spanish. It was really tied though, most results were different but pretty relevant, very hard for me to pick a winner.
Maybe I was imagining it, but four out of the five I did I felt I knew which was Google and which was Bing - not from doing searches that I would recognise, but just from the extra non-search information / lay-out.
I did my best to still pick without bias to my current search engine (Google) but ended up picking them 5/5 and Bing 0 times.
I have actually found myself using bing lately only because it doesn't try to confuse me into landing on google+.
Seriously, guys; this is a problem. I'm searching for a coffee shop or restaurant or something, and I want to link my friends to the map.
I --DO NOT-- want to go to google+. In fact, I don't really want to google+ ever. For anything. Ever.
Seriously I feel like I'm navigating a maze of accidental google+ links every time I use google anymore. It's really frustrating. Bing at least seems to be just...a search engine.
In fact, google hiccuped all over my 8 years old gmail account, deleting my inbox (yay! Thanks for that guys, oh, and it's just lost, too bad for me!) two days ago.
The upside to this? It somehow also unsubbed me from google+!
I find the Google+ spam annoying too, but my pet peeve is the redirects Google put on search results so they can tell which ones you click on. I hate this, first because it feels invasive and second because it noticeably slows things down. Most of the time it's not that bad, but every now and then it adds a second, or 2 seconds, or 5. I got so mad I astonished myself by switching to Bing for a while. And indeed the Bing experience was much faster. I've switched back to Google for now, mostly out of habit. But I've gradually gone from being a big fan of theirs to a grudging captive. Something could jolt me into a different orbit.
Other than the logo, and the bar at the top that says "Join Google+" (which of course, may be a slight annoyance to some people), what difference do you find between the old Maps/Places page for a restaurant and the new "Google+ Local" restaurant page that you're afraid of being confused into landing on? I just checked a local pizza joint while not signed in to Google and those two things are the only things that aren't reviews and information about the place (i.e. the only differences of substance between the new pages and the old Place pages).
I added plus.google.com to the list of sites blocked from search results shortly after they started spamming them. Too many results were polluted with google+ above ones I was actually looking for.
Why? Because I tested it with fairly specific queries. A while ago I tried to switch to Bing because after testing it with a few general searches, it seemed just as good. However, when I switched and started searching for compiler errors, etc - I soon switched back because the results were just nowhere near as good.
Here's what I searched for:
bash: !": event not found error
gtk calendar tutorial
python import gtk error
seed funding in uk
Something to remember by the way is that Google has spent a lot of time and effort on their "bubble", that is, search results personalisation (http://dontbubble.us) and it is a big part of what makes their results so great. That influence won't be present in this test.
Ended up picking google 4-1. I'm somewhat of a bing fan in the sense that I find them good enough for most searches. So I was a bit surprised that the results ended up skewing in google's favor ... guess that kind of backfired for them :-P
5-0 in favor of Google for me. 3 of my queries had 5 words, one had 4, and one had 2. Bing is almost as good at solving the easy short ones, while Google can solve even the long queries.
Bing seems to be a bit better at generic, broad searches (games, file extension $something, etc) and Google seems to be better at exact, specific queries (stop error 7B, RPGs announced at PAX 2012).
Bing ended up winning, which shocked the hell out of me, but after trying a couple more times, I noticed that pattern.
I did a comparison by actually searching the same phrases on google.com/ncr with Incognito. The actual Google results are much better than the Google results on this page.
Search for "Hunger Games" on Google returns the IMDB page as the first result, which is exactly what I want. But neither panes on the Bing's test has the IMDB page as the first result.
5-0 for Google. I tried mostly programming related queries as I find the differences most noticeable there. Especially when searching for specific APIs or objects, Bing tends to give you just the project homepage and some unrelated pages, while Google gives you the actual deep link.
In my case Bing won 1 out 5 of the rounds. I used searches I had done recently that hadn't given satisfactory results on Google.
In any event Microsoft now at least has searches it can work with and improve. I can't imagine any other way they could get Google users to provide search data.
The study results are not very accurate. They made it with only 1000 US random users, and ~60% chose Bing more often, ~30% chose Google more often (on 10 queries).
See the following fictive example:
1000 people, 600 have the following score: 4 Bing, 3 Google, 3 Draw; 400 have the following score: 10 Google.
So, in my example, 60% of the people chose Bing more often and 40% chose Google more often. But, if you look at queries numbers, Google was chosen for 3600+10400=5800 queries, and Bing for 4*600=2400 queries, that's a 2:1 ratio, but in favor of Google.
This is not a real blind test though. (Pardon me for being a web developer, but) I can tell which is which by just looking at the green colors used in the links. The green in Google results is a little bit more saturated than the one in Bing.
Google results also seems to show share counts from Google+ as in: 73,352 people +1'd this
1. survey is organized by Microsoft, 2. I think the survey is skewed because typically when you go there you don't have anything specific in mind to search so you type one-word query (they even propose them!) and one-word queries are hard for google
I had two different runs that both ended 3-2 for Google. Even if Bing came out ahead, they seem so similar that I couldn't be bothered to retrain myself to bing stuff, instead of googling it, especially when I have gmail accounts.
[+] [-] ChuckMcM|13 years ago|reply
That said, its an interesting experiment. And at blekko.com we've been running for over a year [1].
What we've found is that there is a huge brand bias, which is to say that if you use some search engine as your primary search engine, you tend to think that is the best one regardless of whether its Blekko, Bing, or Google (or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index).
But 'quality' is also a very subjective thing as well. So if you search for highly SEO'd categories you will find the Blekko and Bing do better than Google mostly because there is a curated input (in Blekko's case it was from day one with it's from slashtags and in Bing's they started doing outsourced curation (putatively after seeing how effective it is in Blekko :-) in some topics with their 'editors' program [2]) At some point Google will realize what Bing and Blekko have which are that the 'indexing the web' problem became the 'filtering the web' problem when the signal to noise ratio started decreasing in about 2005, and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.
If the Bing guys are reading (and I know you are) why not open up your challenge to the new kid on the block, we don't just do Blekko vs Google in our monte results :-)
[1] http://techcrunch.com/2011/06/21/be-the-mark-in-blekkos-3-en...
[2] http://www.bing.com/community/site_blogs/b/search/archive/20...
[+] [-] pilgrim689|13 years ago|reply
Just took a look at Blekko. It looks good, but you can't honestly believe the layman cares as much about it as he does about search engines coming out of MS or Google. Does the layman (MS' target audience here) even know about Blekko?
[+] [-] ralfn|13 years ago|reply
You are aware that Google hires thousands of search engine quality raters, all over the world right? But rather than directly curating, i suspect they just use these humans to train their algorithms. There is likely some inductive step going on, in selecting which algorithm (or 'sort' of the web) is selected, and then filtered by keywords.
Secondly, most search engines, actually do also use end-user curation. I have funky enough search queries, that I often get google redirects as search results: they want to know (and record) which link i pick. Also, because of all the google-adds-cookies, google-analytics-cookies, etc, they likely have 'session-bags' full of destination links and search queries that happen to correlate.
For me personally, neither search engine did very good in this test. But that's likely because neither result set is personalized in this test. And i don't know, if Bing does that now, but even if it does, for me to switch to another search engine, means to train another search engine in my contextual world. Hell, i even notice difference in result sets in Google based on my operating system. I get different results in Linux, than i do on my iPad.
People that get a lot of spam in search results, really ought to login, and stop clicking those links.
[+] [-] nugget|13 years ago|reply
[+] [-] autophil|13 years ago|reply
To Everyone: Try it out. Blekko's results are a breath of fresh air.
[+] [-] epi0Bauqu|13 years ago|reply
[+] [-] natex|13 years ago|reply
Looks like you might be wrong about that.
DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, and Blekko.
http://help.duckduckgo.com/customer/portal/articles/216399-s...
[+] [-] MarkMc|13 years ago|reply
However I don't think Microsoft's study is any good. They say, "In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice."
So it seems that the study participant had to think of 10 search queries, one after the other. But this is not how search works. In the real world users need to perform one query every so often, and their choice of search terms is based on an immediate need rather than what comes to mind.
A good Bing vs Google study would ask 1000 people to use a split-test search engine for their browser search box, then monitor the results over time. I've asked HN about this before and got nowhere: http://news.ycombinator.com/item?id=3966121
So is there business opportunity here? I imagine that there must be thousands of Google shareholders willing to pay £100 per year for the results of an unbiased, ongoing study into who gives the best search results. In fact, why doesn't Google set up the study and provide such information to shareholders?
[+] [-] Hermel|13 years ago|reply
[+] [-] mynameishere|13 years ago|reply
[+] [-] Sindisil|13 years ago|reply
Now, all of my searches were technical; I plan to try again later with more general, "normal people" searches. Bing may do better there.
Also, good results from even google searches have become harder to come by in the last year or so. I have to work harder at my search terms. I don't know if that's Sturgeon's Law at work, or if there's room for a search engine to improve enough to become better than google.
Perhaps targeted search engines are the answer (i.e. engines's who have both their search and crawler algorithms tuned for a specific target audience) are the answer?
[+] [-] calinet6|13 years ago|reply
Of course it's going to be a draw for most searches (especially the suggested ones: "math games", "coupons", "time zones"...), they're generic and easy to find.
You can use any search engine you want to find those things. Shoot, even AltaVista was good at those kinds of searches, and that was 10 years ago.
The ones I care about—even if I'm not a programmer or technical user—are the difficult searches. What if I need to find a specific car part that only ever existed in the 1972 Ford Pinto? What if I want to find a plumber who specializes in pre-20th-century homes? What if, what if, what if.
I used searches like that and more in this comparison, and it was 100% Google. I could even tell easily why it was Google, and it was pretty clear which one it was. I still chose Google (despite being basically unblinded) because it was fundamentally better at these specific searches.
That's what bing needs to compare. But will they show it? Of course not.
Edit: I did 5 more searches all non-technical but specific and picked only based on result quality. Again 5/5 google. (for those curious, the searches were: "lego mindstorms", "Mars curiosity hi-res", "home developments in myrtle beach", "used bookstores in portland, or", "top baby names in 1982"—the last one was very cool because Google also provided links to the 5 surrounding years right below the 1982 result. It's the little things you know...)
[+] [-] FigBug|13 years ago|reply
[+] [-] sondh|13 years ago|reply
[+] [-] JepZ|13 years ago|reply
All searches were related to pages I know and I searched for some keywords or parts of the domain.
[+] [-] mindprince|13 years ago|reply
Here's what I searched for -
-my name (Google, I chose Google because it gave more of my results, which may not be fair.)
-extract articles from websites (Google, in this search google's result were much better.)
-python ide for mac (Draw)
-new york time (Draw)
-glusterfs (Draw)
[+] [-] rfergie|13 years ago|reply
[+] [-] pmelendez|13 years ago|reply
[+] [-] kodisha|13 years ago|reply
[+] [-] bwanab|13 years ago|reply
[+] [-] ximi|13 years ago|reply
[+] [-] corin_|13 years ago|reply
I did my best to still pick without bias to my current search engine (Google) but ended up picking them 5/5 and Bing 0 times.
[+] [-] quandrum|13 years ago|reply
[+] [-] dspig|13 years ago|reply
[+] [-] joelrunyon|13 years ago|reply
[+] [-] blhack|13 years ago|reply
Seriously, guys; this is a problem. I'm searching for a coffee shop or restaurant or something, and I want to link my friends to the map.
I --DO NOT-- want to go to google+. In fact, I don't really want to google+ ever. For anything. Ever.
Seriously I feel like I'm navigating a maze of accidental google+ links every time I use google anymore. It's really frustrating. Bing at least seems to be just...a search engine.
In fact, google hiccuped all over my 8 years old gmail account, deleting my inbox (yay! Thanks for that guys, oh, and it's just lost, too bad for me!) two days ago.
The upside to this? It somehow also unsubbed me from google+!
[+] [-] gruseom|13 years ago|reply
[+] [-] wutbrodo|13 years ago|reply
[+] [-] raldi|13 years ago|reply
[+] [-] GraemeL|13 years ago|reply
[+] [-] kaolinite|13 years ago|reply
Why? Because I tested it with fairly specific queries. A while ago I tried to switch to Bing because after testing it with a few general searches, it seemed just as good. However, when I switched and started searching for compiler errors, etc - I soon switched back because the results were just nowhere near as good.
Here's what I searched for:
Something to remember by the way is that Google has spent a lot of time and effort on their "bubble", that is, search results personalisation (http://dontbubble.us) and it is a big part of what makes their results so great. That influence won't be present in this test.[+] [-] CodeCube|13 years ago|reply
[+] [-] mtgx|13 years ago|reply
[+] [-] Karunamon|13 years ago|reply
Bing ended up winning, which shocked the hell out of me, but after trying a couple more times, I noticed that pattern.
[+] [-] esolyt|13 years ago|reply
Search for "Hunger Games" on Google returns the IMDB page as the first result, which is exactly what I want. But neither panes on the Bing's test has the IMDB page as the first result.
[+] [-] ch0wn|13 years ago|reply
[+] [-] timothya|13 years ago|reply
So I wonder if this test is really "Google vs. Bing + Google".
EDIT: Source - http://news.ycombinator.com/item?id=2165469
[+] [-] michaelcampbell|13 years ago|reply
[+] [-] rogerbinns|13 years ago|reply
In my case Bing won 1 out 5 of the rounds. I used searches I had done recently that hadn't given satisfactory results on Google.
In any event Microsoft now at least has searches it can work with and improve. I can't imagine any other way they could get Google users to provide search data.
[+] [-] hk__2|13 years ago|reply
See the following fictive example:
1000 people, 600 have the following score: 4 Bing, 3 Google, 3 Draw; 400 have the following score: 10 Google. So, in my example, 60% of the people chose Bing more often and 40% chose Google more often. But, if you look at queries numbers, Google was chosen for 3600+10400=5800 queries, and Bing for 4*600=2400 queries, that's a 2:1 ratio, but in favor of Google.
[+] [-] hk__2|13 years ago|reply
[+] [-] parfe|13 years ago|reply
After 5 searches the screen greyed out and was unresponsive.
Guess Google wins by disqualification?
[+] [-] sgricci|13 years ago|reply
[+] [-] wickedchicken|13 years ago|reply
http://blekko.com/ws/hunger+games+/monte
[+] [-] tree_of_item|13 years ago|reply
[+] [-] upinsmoke|13 years ago|reply
[+] [-] niyazpk|13 years ago|reply
Google results also seems to show share counts from Google+ as in: 73,352 people +1'd this
[+] [-] tucson|13 years ago|reply
[+] [-] crazypyro|13 years ago|reply
[+] [-] islon|13 years ago|reply