In PG's Frighteningly Ambitious Startup Ideas essay he talks at length about building a new kind of search engine. I was curious, who is working on this angle and makes your search engine innovative?
I have an idea( prototype? ) for a search engine that could be disruptive, with a bit of luck. It has nothing to do with bangs and slashtags.
I've hired a freelancing coder to do some "wrapping up" so I could show the prototype but it is over a month now and still I haven't heard from him.
I do not want to engage too much time and resources into this (hiring stationary coders) but this really is the search engine that is closest to disrupting Google and Bing.
I believe that slashtags and bangs and non-spam results are not the way to go, my search engine tries a different route altogether which those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big.
I can produce results semi-manually and all I need is automation for that method but the coder I've hired, stopped responding over a month ago.
I really see big future in it but I do not want to invest too much effort and money into something so risky.
I wish I could show you what it is all about but it would reveal too much. And without the automated prototype, some good coder could simply copy it in like 2 weeks.
I believe that for $10000 I could present a really basic prototype of something that is disruptive to Google and Bing and if it isn't I think nothing will ever be.
"those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big." and "some good coder could simply copy it in like 2 weeks." seem very contradictory.
Sounds like an interesting idea from a UX perspective. If you're looking to get this developed quickly, go ahead and contact me here: http://prontotype.us/ (I would have contacted you if I could find your email)
I would assume they're heading in that direction given that they decided not to renew their contract with Google which used to let Google display real-time search results from Twitter.
I didn't write a search engine of any kind, I wrote a search front-end of sorts which redirects you to actual search engines (or Wikipedia if your query is an exact article title match). It doesn't handle not-ASCII text very well and there are some other bugs I should iron out and sometimes Heroku is down (hosted for free so I shouldn't complain) but it works well enough for me 99% of the time so I'm not actively working on it.
I would love to see if Facebook could create a better search engine by using an individual's social network to weight links instead of backlinks.
It might be too much work just to see if it is useful but I'm thinking they have a ton of data about where everyone goes on the web because the Like button is so pervasive. Why not use that data to make search extremely personalized?
Reasons why I like this:
1) It's not like anyone currently uses facebook for search, they literally have nothing to lose.
2) It would be interesting to see if the aphorism, "Show me your friends and I'll show you your future" holds up.
3) It's data that google doesn't have access to, unless G+ takes off.
4) It would be cheaper to crawl the web, essentially your users are your web crawlers.
Take the same graph theory algorithms used for backlinks and apply it to views and instead of page rank use time that their friends spend on the site.
Then you can do some interesting stuff like weight the page rank more heavily for individuals you interact with more.
Or super creepy (but statistically interesting) stuff could be done like if you want someone to reconnect with a friend they haven't spoken to in a while, weight the "long lost friend's" pages more heavily. Essentially, influence people to talk to each other by converging their data about the world.
EDIT: Another thought - if websites know that they're only admission into the search engine is putting a Like button on their page it gives them an incentive to do so.
I am working on a very specialized search engine for software libraries. You can find it here: http://www.versioneye.com.
I don't think that we need better search algorithms for search engines. I think we just need more filters in the internet. With filters I mean more specialized search engines who are not crawling everything.
If I am typing in "Spring" into the search field, I am not interested in "Spring", I am interested in the spring framework. But because most of the people on this planet are not software developers and Google don't know that I am a developer, the spring framework will not be the first result on google.
Of course we need general search engines like google. But I would love to use more specialized search engines who are just showing results to a special topic.
I consider writing Digest for a 3rd time. The first time was a proof of concept in Perl, the 2nd time was product in Java for [name of customer deleted]. The 3rd version could be in PHP for the masses.
Digest is not a normal search engine, but an XML description language to tell a spider where to find what on a web layout. Its aimed on the dark side of the internet, able to login on website, use cookies, and even delegate captcha solving.
The main problem of Digest is that its maintenance intensive because websites are changing layouts regularly. The idea of a Digest for the masses would be an open source distributed search engine, that runs on normal peoples web servers. So every (power) user has access to the search engine itself.
My problem, and question to this audience is: How to make money with such an project? The only idea I have currently is kickstarter.
That's not a new kind of search engine, it is a frontend for bing that can be used without the shame of having to say you use bing.
As far as I know, nobody is building a new kind of search engines, although I'd guess that a new kind of search engine won't really look like a search engine at first.
Shameless plug: we're working on a product focused solely on opinions called Cheerboo (http://www.cheerboo.com/).
It's part search/discovery part social, and is trying to solve the problem that all the opinions shared in places like Twitter/Facebook aren't preserved, aggregated or made useful outside of the social graph.
We organize opinions around topics, make people weigh in on topics with a binary rating system (cheer or boo) and use the cheers/boos to give each topic a "RottenTomatoes-style" score. Let us know your thoughts - we love feedback.
I love the concept and the design, but I have a few suggestions:
1. Give immediate visual after clicking the cheer/boo buttons. It can take a few seconds for the loading icon and comments to appear, so having the button lighten or change at all would let me know that it is in fact working.
2. Show the number of comments on each subject so that users have an idea of how popular a subject is at a glance without loading the comments.
3. Add a setting to not automatically load comments after cheer/boo -ing.
Shameless plug: My startup, Semantics3 (http://www.semantics3.com/), is building an vertical based search engine for products data - track product pricing changes, get detailed metadata etc. Our current strategy is to monetize on selling the indexed data as part of a pay-as-you-go api.
Later on we plan to build a consumer facing website/search engine, essentially a front-end for our api, once we have moved into building large indexes around people and places (the two other verticals we are targeting)
Does there exist any non-commercial search engine? When I say "non-commercial", I don't just mean a search engine that doesn't make money, I mean a search engine that excludes all commercial websites. You accept credit cards? Sorry, you're out. You have an ad on your page? Out. Simple as that. I suspect this view of the web could bring back some of the old magic feeling. It would include Terry Tao's blog, and all of Wikipedia, but exclude the SEO'd crap that Google has incentivized.
Speaking of which, there are 'open' indexes of the web that anyone can download and use to build their own search product. But I can't remember the name of them. Can anyone help? Googling it results in dismal failure.
I think there is still huge room for innovation in search. Google has taken a different track, and they're focusing on Machine Intelligence. (in my opinion) search is just a nice side effect, but I really don't think it's their focus any longer.
Maybe it's just me, but search engines suffer from a vicious circle. Noone uses your search engine unless you have enough indexed pages, and you can't index the entire web (or even a small portion of it) unless you have enough generated $$$ for high-performance servers.
That probably explains why search engine space is dominated by a few power players, with newer players like DuckDuckGo being used by only a dedicated bunch of fans.
I think it strips out the top however many results you get from google. It can possibly (depending on the search), get rid of SEO heavy results that don't have much useful content.
[+] [-] fear91|13 years ago|reply
I've hired a freelancing coder to do some "wrapping up" so I could show the prototype but it is over a month now and still I haven't heard from him.
I do not want to engage too much time and resources into this (hiring stationary coders) but this really is the search engine that is closest to disrupting Google and Bing.
I believe that slashtags and bangs and non-spam results are not the way to go, my search engine tries a different route altogether which those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big.
I can produce results semi-manually and all I need is automation for that method but the coder I've hired, stopped responding over a month ago.
I really see big future in it but I do not want to invest too much effort and money into something so risky.
I wish I could show you what it is all about but it would reveal too much. And without the automated prototype, some good coder could simply copy it in like 2 weeks.
I believe that for $10000 I could present a really basic prototype of something that is disruptive to Google and Bing and if it isn't I think nothing will ever be.
[+] [-] Liongadev|13 years ago|reply
[+] [-] sprobertson|13 years ago|reply
[+] [-] iheartmemcache|13 years ago|reply
[+] [-] ianterrell|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] cojourneo|13 years ago|reply
Unlikely I know, but their access to true real-time data presents interesting possibilities I think.
[+] [-] Scrockett|13 years ago|reply
[+] [-] cojourneo|13 years ago|reply
[+] [-] arpit|13 years ago|reply
http://blekko.com/
[+] [-] reiz|13 years ago|reply
[+] [-] reiz|13 years ago|reply
[+] [-] mun2mun|13 years ago|reply
[+] [-] uiri|13 years ago|reply
http://s.xqz.ca/
[+] [-] milesokeefe|13 years ago|reply
[+] [-] ljd|13 years ago|reply
It might be too much work just to see if it is useful but I'm thinking they have a ton of data about where everyone goes on the web because the Like button is so pervasive. Why not use that data to make search extremely personalized?
Reasons why I like this:
1) It's not like anyone currently uses facebook for search, they literally have nothing to lose.
2) It would be interesting to see if the aphorism, "Show me your friends and I'll show you your future" holds up.
3) It's data that google doesn't have access to, unless G+ takes off.
4) It would be cheaper to crawl the web, essentially your users are your web crawlers.
Take the same graph theory algorithms used for backlinks and apply it to views and instead of page rank use time that their friends spend on the site.
Then you can do some interesting stuff like weight the page rank more heavily for individuals you interact with more.
Or super creepy (but statistically interesting) stuff could be done like if you want someone to reconnect with a friend they haven't spoken to in a while, weight the "long lost friend's" pages more heavily. Essentially, influence people to talk to each other by converging their data about the world.
EDIT: Another thought - if websites know that they're only admission into the search engine is putting a Like button on their page it gives them an incentive to do so.
[+] [-] reiz|13 years ago|reply
I don't think that we need better search algorithms for search engines. I think we just need more filters in the internet. With filters I mean more specialized search engines who are not crawling everything. If I am typing in "Spring" into the search field, I am not interested in "Spring", I am interested in the spring framework. But because most of the people on this planet are not software developers and Google don't know that I am a developer, the spring framework will not be the first result on google.
Of course we need general search engines like google. But I would love to use more specialized search engines who are just showing results to a special topic.
I would like to hear your thoughts about this.
[+] [-] kephra|13 years ago|reply
Digest is not a normal search engine, but an XML description language to tell a spider where to find what on a web layout. Its aimed on the dark side of the internet, able to login on website, use cookies, and even delegate captcha solving.
The main problem of Digest is that its maintenance intensive because websites are changing layouts regularly. The idea of a Digest for the masses would be an open source distributed search engine, that runs on normal peoples web servers. So every (power) user has access to the search engine itself.
My problem, and question to this audience is: How to make money with such an project? The only idea I have currently is kickstarter.
[+] [-] Jacqued|13 years ago|reply
[+] [-] notatoad|13 years ago|reply
As far as I know, nobody is building a new kind of search engines, although I'd guess that a new kind of search engine won't really look like a search engine at first.
[+] [-] Scrockett|13 years ago|reply
It's part search/discovery part social, and is trying to solve the problem that all the opinions shared in places like Twitter/Facebook aren't preserved, aggregated or made useful outside of the social graph.
We organize opinions around topics, make people weigh in on topics with a binary rating system (cheer or boo) and use the cheers/boos to give each topic a "RottenTomatoes-style" score. Let us know your thoughts - we love feedback.
[+] [-] milesokeefe|13 years ago|reply
1. Give immediate visual after clicking the cheer/boo buttons. It can take a few seconds for the loading icon and comments to appear, so having the button lighten or change at all would let me know that it is in fact working.
2. Show the number of comments on each subject so that users have an idea of how popular a subject is at a glance without loading the comments.
3. Add a setting to not automatically load comments after cheer/boo -ing.
[+] [-] netvarun|13 years ago|reply
Later on we plan to build a consumer facing website/search engine, essentially a front-end for our api, once we have moved into building large indexes around people and places (the two other verticals we are targeting)
[+] [-] cousin_it|13 years ago|reply
[+] [-] chris_dcosta|13 years ago|reply
[+] [-] rainysunday|13 years ago|reply
[+] [-] dholowiski|13 years ago|reply
I think there is still huge room for innovation in search. Google has taken a different track, and they're focusing on Machine Intelligence. (in my opinion) search is just a nice side effect, but I really don't think it's their focus any longer.
[+] [-] rainysunday|13 years ago|reply
[+] [-] madmax108|13 years ago|reply
That probably explains why search engine space is dominated by a few power players, with newer players like DuckDuckGo being used by only a dedicated bunch of fans.
[+] [-] jrussbowman|13 years ago|reply
www.unscatter.com
I use it to keep up to date with topics on a regular basis and also when searching for recent info on topics
I haven't heard about common crawl mentioned in another comment, going to go check that out now.
[+] [-] rudiger|13 years ago|reply
http://drift.io
[+] [-] ARavara|13 years ago|reply
[+] [-] j21|13 years ago|reply
I think it strips out the top however many results you get from google. It can possibly (depending on the search), get rid of SEO heavy results that don't have much useful content.
[+] [-] amitamb|13 years ago|reply
Specifically http://www.verticalset.com/extensions
And build your own here
http://www.verticalset.com/extensions/new
[+] [-] nmb|13 years ago|reply