Great article. One note: Google's dynamic abstracts were not only very useful, they also improved perceived relevance because they let users see why the pages were selected.
When I was at Altavista, we were also blocked from doing dynamic abstracts by cost.
Google's main advantages were:
- managed by the founders with a total focus on search and measurable results
- google's hiring process produced a very strong team early on
- strong focus on controlling costs from the beginning (Altavista's use of the DEC Alpha was a huge handicap.
Close. Lots of other companies were also hiring pretty high tier talent as well, and had intense focus. Google's success came down to effectively executing across typically disparate disciplines. You have hard core research level CS eggheads, you have top tier software engineers, and you have state of the art data center operations. In a typical organization these groups have competing interests, they fight amongst themselves and in the end some sort of compromise is reached that allows everyone to grudgingly get along.
At google these three groups worked hand in hand and complemented each other's work. The eggheads came up with page rank, the coders figured out how to make pagerank scale through massive paralellism via sharding and mapreduce, and the data center folks figured out how to make sharding cheap and fast through commodity pc based servers and massive amounts of automation for management. In the end everyone was working at the top of their game to help everyone else. The result was that google was able to deliver better results (pagerank) faster (mapreduce) and cheaper (automated commodity hardware datacenters) than the competition.
There were lots of other fine details that led to google's success, but in the end those core factors are what allowed them to deliver a better search experience to users (better/faster) and to be more competitive in the marketplace (lower cost per search means more profit even with lower per search ad revenue).
No one else in search was pushing on all the right pressure points the way google was, and the rest is history.
When Altavista launched, it was an impressive showcase of the DEC Alpha's power. Intel only became usable for serious servers (with the exception of exotic stuff like Sequents), as did Linux, years later. Google had the good fortune to be in the right place at the right time, when Lintel became a commodity in the datacentre. 5 years earlier, they'd have been on Sun probably.
The better abstracts is the reason I use DuckDuckGo at home.
If I just want to know when the next episode of Big Bang Theory is out or what the weather is today I rarely need to even click on a result.
For more obscure technical searches at work, Google still finds more answers.
But remember - the barrier to change for a search engine's customers is very very low
Inktomi killed Inktomi long before Google helped put the nail in the coffin.
What the article doesn't say is Inktomi had a dual sided business. One side was in Caching Proxies the other was licensing a search API.
Inktomi decided to focus on the caching proxy business and de-emphasized their search product, only to watch the proxy business evaporate as internet bandwidth became cheaper/better.
The focus on a shrinking market (proxies) and the lack of focus on growing market (search) killed them. Had search been a priority from the beginning things may have ended very differently with Inktomi creating their own front end.
Indeed. Inktomi also tried to position themselves as an arms supplier to the CDN business. It didn't help that the CDN business basically disappeared from 2001-2004, and that CDNs, to this day, rarely buy software.
I was going to mention this. It seemed like the management at Inktomi let it fall once the engineers started using Google search engines. Their response is a likely bellweather of the attitude of the time.
It was clear that Yahoo.com was the definitive result for the query "yahoo" so it would score a 10. Other Yahoo pages would be ok (perhaps a 5 or 6). Irrelevant pages stuffed with Yahoo-related keywords would be spam.
As someone who worked on search quality at Google for some time, this bit jumped out at me as a terrible mistake. The correct way to judge results for the query [yahoo] is:
(a) Where is yahoo.com? At the top?
(b) There is no (b).
It seems like a slight difference, but it leads to the wrong priorities. For the query [yahoo], it does not matter if spam or non-spam is in spot #5. The only thing that matters is where you put yahoo.com.
I just realized that Google won on search the way Apple has won on smartphones. They control the full stack -- frontend, relevance, indexing, advertising -- and tightly couple these pieces. Inktomi couldn't control the user interface the way Google can't really control the interface on Android.
You are right, but IMO you miss the real point. Both Google's search and Apple's iPhone were about delivering a wholly satisfying product/service, and controlling the whole stack was needed to do that for those cases. This is not always true (though it often is).
Most interesting to me is that Inktomi had all the power to beat, acquire or replicate Google but didn't have the right mindset. They were operating under a few bad assumptions:
- search is a commodity for licensing (making them resistant to launching a "cleaner" engine that would alienate their clients)
- what worked for a smaller internet (100 million pages) could scale appropriately with the growing internet (100 billion pages) without rethinking everything
- "Page rank" only helped relevance (it was also about spam)
I think Google is stuck in a rut of their own right now. Here's some faulty assumptions I think Google is making:
- users always want faster, more direct answers (rather than controlling the filtering/categorization of their searches)
- users want Google to predict what they mean rather than clarify what they mean
- users always want faster, more direct answers (rather than controlling the filtering/categorization of their searches)
That's a very power-user centric attitude, don't you think? As a power user I preferred to type long, complicated Sabre queries to find exactly which airplane flight I wanted. It was much faster, and I had memorized all of the complicated mnemonics. But that's not what a casual user would want to use.
Asking users to specify categories for what they want means requiring a certain orientation in their thinking which is shared by computer scientists and trained librarians. But to an average user, that's extra work. And think about how this might work if you're talking to an actual human librarian: if you start asking about TV shows, and then mention "The Big Bang Theory", do you think the librarian will ask you, "Did you mean the scientific theory, or the TV show?" That's only something a stupid computer would do. A smart librarian would take the context of the previous queries that you've made of him or her, and provide the right answer quickly and efficiently. Wouldn't you want the same thing from a search engine?
To be fair, faster answers + the ability to undo a do-what-I-mean guess lets me correct Google's assumptions pretty fast. The tools at the left allow for some quick refining as well; that's pretty useful when I need particularly fresh results, or a time window from when some news was breaking. And the fast completion is useful to refine a query before it returns results (though occasionally annoying when it erases quotes and the like).
After I had switched to Google, I never understood why all of the competition just disappeared over night. You would think they would have given it a fight, but that never seemed to happen. At least this article gives a little insight to that. I still wonder what happened to Altavista.
AltaVista tried to jump on the 'portal' bandwagon. I remember at the time how stupid I thought they were, trying to beat Excite and Yahoo at something that was already old hat, a concept that had the Internet had outgrown. Then they screwed up pretty much the same way Inktomi did.
AltaVista still exists. It's awful, and it's powered by the Yahoo search engine. Which is pretty much the same thing, I suppose.
I had no idea that that Diego worked on Inktomi. Although it makes sense why IndexTank worked fantastically.
If I had to pick one reason why Google triumphed (and you can only pick one), I think it would be their Page Rank algorithm. It added that extra bit of awesome-sauce to and already tasty stew.
According to a friend of mine who worked on the search team, Inktomi shifted its (management and CapEx) focus away from search and onto other projects. He thought at the time that even with the constraints of not competing with their own customers, there were things they could have done to better compete if management had chosen to do so.
Yes, that's true. They didn't think search would be a huge business. Back then the model was to sell search to portals charging by query volume, and it was a race to the bottom. Our Solaris servers were more expensive than Google's Linux boxes.
After reading the "Tale" it appears that Inktomi killed itself. It is a good example of what happens if top dog companies fail to innovate in face of sudden superior competition. RIM is another example, but would be wrong to say that Apple is killing them. Apple is just making and selling superior products.
True. In this case, I forgot to mention that not having a world-facing UI deprived us of vital signals that Google used to improved their experience. We had query logs, but we had no idea of what our customers' users were doing within a session, no user history, etc.
"Are there any lessons to be learned from this? For one, if you work at a company where everyone wants to use a competitor's product instead of its own, be very worried."
This is because companies sometimes (maybe often?) ban the use of competitor products to their detriment.
That was a good read. I remember I used Yahoo for searching the web. Due to relevancy factor I moved to Altavista (but it didn't improve anything until the day I found out about Google and still use Google). I didn't know that Inktomi was powering search at that time. If Yahoo was so dependent on Inktomi or Google for its Search, I wonder why didn't they work on Search by themselves. After all they were information organization tool. Why did they ignore such a huge market. VC's were going crazy for funding search engines and number of search engines companies were either getting funded or going public. Based on these signals and the traffic they had during dot com era, they could have easily built substantially good search engine; yet they ignored it. Can anyone shed light on it?
Yahoo owned Overture so their results were pure pay-for-placement. If anyone wanted to pay $100 per click to have the #1 spot on "beef" go to a site about chicken, that was a-OK with Yahoo management. When Yahoo realized their auction system was stupid, they had "project Panama" which was also a joke and by that time Google had the market to themselves anyway.
Inktomi engineers using Google reminds me of TI engineers using HP calculators. TI banned them, which apparently just moved the calculators under the desk
When I started at Yahoo Search (in 2005; it was Inktomi), I quickly learned that "just Google it" was frowned upon, and modified my vocabulary to say "just use web search". To this day, I still use this phrase.
Most engineers that I knew in YST did not use Google at all. We preferred to eat our own dogfood, and filed query triages against bad results (and only used Google to compare).
Ok, so the timing on this is really amazing. Techcrunch, reporting on Facebook's S1, mentioned that Yahoo! has suggested another 12 patents they may try to throw against Facebook and their Open Compute project. The Yahoo! letter is here: http://www.scribd.com/doc/92280387/TechCrunch-Letter-From-Ya... and the source of many of these patents? Inktomi!
In the next year and a half the stock went down by 99.9%. In the end, Inktomi was acquired by Yahoo for 250M.
This is unrelated to the main point, but does anyone know if Yahoo re-used a significant amount of Inktomi tech acquired for that $250M? Or was it spoiled?
I've been told (by people from Inktomi who then worked at Yahoo), that yes, the Inktomi technology became the main Yahoo search engine. (How much it changed or not in the process, I don't know)
[+] [-] paulsutter|14 years ago|reply
When I was at Altavista, we were also blocked from doing dynamic abstracts by cost.
Google's main advantages were:
- managed by the founders with a total focus on search and measurable results
- google's hiring process produced a very strong team early on
- strong focus on controlling costs from the beginning (Altavista's use of the DEC Alpha was a huge handicap.
[+] [-] InclinedPlane|14 years ago|reply
At google these three groups worked hand in hand and complemented each other's work. The eggheads came up with page rank, the coders figured out how to make pagerank scale through massive paralellism via sharding and mapreduce, and the data center folks figured out how to make sharding cheap and fast through commodity pc based servers and massive amounts of automation for management. In the end everyone was working at the top of their game to help everyone else. The result was that google was able to deliver better results (pagerank) faster (mapreduce) and cheaper (automated commodity hardware datacenters) than the competition.
There were lots of other fine details that led to google's success, but in the end those core factors are what allowed them to deliver a better search experience to users (better/faster) and to be more competitive in the marketplace (lower cost per search means more profit even with lower per search ad revenue).
No one else in search was pushing on all the right pressure points the way google was, and the rest is history.
[+] [-] gaius|14 years ago|reply
[+] [-] excuse-me|14 years ago|reply
If I just want to know when the next episode of Big Bang Theory is out or what the weather is today I rarely need to even click on a result. For more obscure technical searches at work, Google still finds more answers.
But remember - the barrier to change for a search engine's customers is very very low
[+] [-] wavesplash|14 years ago|reply
What the article doesn't say is Inktomi had a dual sided business. One side was in Caching Proxies the other was licensing a search API.
Inktomi decided to focus on the caching proxy business and de-emphasized their search product, only to watch the proxy business evaporate as internet bandwidth became cheaper/better.
The focus on a shrinking market (proxies) and the lack of focus on growing market (search) killed them. Had search been a priority from the beginning things may have ended very differently with Inktomi creating their own front end.
[+] [-] jusben1369|14 years ago|reply
[+] [-] squeed|14 years ago|reply
[+] [-] tomrod|14 years ago|reply
[+] [-] darrellsilver|14 years ago|reply
[+] [-] lacker|14 years ago|reply
As someone who worked on search quality at Google for some time, this bit jumped out at me as a terrible mistake. The correct way to judge results for the query [yahoo] is:
(a) Where is yahoo.com? At the top?
(b) There is no (b).
It seems like a slight difference, but it leads to the wrong priorities. For the query [yahoo], it does not matter if spam or non-spam is in spot #5. The only thing that matters is where you put yahoo.com.
[+] [-] bitops|14 years ago|reply
[+] [-] avichal|14 years ago|reply
[+] [-] dwc|14 years ago|reply
[+] [-] zmmz|14 years ago|reply
It's a recording of a very very good talk by Inktomi co-founder Eric Brewer called "Inktomi's Wild Ride - A Personal View of the Internet Bubble"
[+] [-] diego|14 years ago|reply
[+] [-] kitsune_|14 years ago|reply
[+] [-] lubujackson|14 years ago|reply
- search is a commodity for licensing (making them resistant to launching a "cleaner" engine that would alienate their clients)
- what worked for a smaller internet (100 million pages) could scale appropriately with the growing internet (100 billion pages) without rethinking everything
- "Page rank" only helped relevance (it was also about spam)
I think Google is stuck in a rut of their own right now. Here's some faulty assumptions I think Google is making:
- users always want faster, more direct answers (rather than controlling the filtering/categorization of their searches)
- users want Google to predict what they mean rather than clarify what they mean
- algorithms > human decisions
[+] [-] tytso|14 years ago|reply
That's a very power-user centric attitude, don't you think? As a power user I preferred to type long, complicated Sabre queries to find exactly which airplane flight I wanted. It was much faster, and I had memorized all of the complicated mnemonics. But that's not what a casual user would want to use.
Asking users to specify categories for what they want means requiring a certain orientation in their thinking which is shared by computer scientists and trained librarians. But to an average user, that's extra work. And think about how this might work if you're talking to an actual human librarian: if you start asking about TV shows, and then mention "The Big Bang Theory", do you think the librarian will ask you, "Did you mean the scientific theory, or the TV show?" That's only something a stupid computer would do. A smart librarian would take the context of the previous queries that you've made of him or her, and provide the right answer quickly and efficiently. Wouldn't you want the same thing from a search engine?
[+] [-] obtu|14 years ago|reply
[+] [-] bstar77|14 years ago|reply
[+] [-] lobster_johnson|14 years ago|reply
AltaVista still exists. It's awful, and it's powered by the Yahoo search engine. Which is pretty much the same thing, I suppose.
[+] [-] nl|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] chrisacky|14 years ago|reply
If I had to pick one reason why Google triumphed (and you can only pick one), I think it would be their Page Rank algorithm. It added that extra bit of awesome-sauce to and already tasty stew.
[+] [-] silvestrov|14 years ago|reply
The Inktomi managers had the point of view of a generic MBA.
Everything followed from that difference of view.
[+] [-] FilterJoe|14 years ago|reply
Edit: fixed typo
[+] [-] diego|14 years ago|reply
[+] [-] cpeterso|14 years ago|reply
[+] [-] kloc|14 years ago|reply
[+] [-] gee_totes|14 years ago|reply
Agreed. If I had a nickel for every time my Blackberry crashed since I upgraded the OS a year ago, I'd buy some Apple stock.
[+] [-] gaius|14 years ago|reply
[+] [-] diego|14 years ago|reply
[+] [-] drucken|14 years ago|reply
"Are there any lessons to be learned from this? For one, if you work at a company where everyone wants to use a competitor's product instead of its own, be very worried."
This is because companies sometimes (maybe often?) ban the use of competitor products to their detriment.
[+] [-] dm8|14 years ago|reply
[+] [-] thrownaway2424|14 years ago|reply
[+] [-] aidenn0|14 years ago|reply
[+] [-] ajays|14 years ago|reply
Most engineers that I knew in YST did not use Google at all. We preferred to eat our own dogfood, and filed query triages against bad results (and only used Google to compare).
[+] [-] ChuckMcM|14 years ago|reply
Is that crazy or what?
[+] [-] kenrikm|14 years ago|reply
A) it was fast, it loaded fast. B) it was not filled with ads and pop ups.
Only one of those is still true.
[+] [-] rkudeshi|14 years ago|reply
Easy test: are you using Bing now? (with their new clean results pages)
[+] [-] brown9-2|14 years ago|reply
This is unrelated to the main point, but does anyone know if Yahoo re-used a significant amount of Inktomi tech acquired for that $250M? Or was it spoiled?
[+] [-] brendano|14 years ago|reply
[+] [-] aristus|14 years ago|reply
[+] [-] rometest|14 years ago|reply
[+] [-] The1Mirage|14 years ago|reply
x's so it won't become a link!
hxxps://wxw.google.com/search?ix=aca&sourceid=chrome&ie=UTF-8&q=domino+pizza+phone+number#hl=en&gs_nf=1&tok=DBTJEp2_3oW1F2ietDMecQ&pq=ipad%20screen%20resolution&cp=1&gs_id=5w&xhr=t&q=new+ipad+screen+resolution&pf=p&safe=off&sclient=psy-ab&oq=nipad+screen+resolution&aq=0&aqi=g2g-b2&aql=&gs_l=&pbx=1&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.,cf.osb&fp=8316e992ae23057e&ix=aca&biw=1363&bih=647
[+] [-] cabalamat|14 years ago|reply
[+] [-] diego|14 years ago|reply