Holy shit, so you are uh building a search engine from scratch.
Do you crawl yourself? What is your infrastructure? What is your goal for search.marginalia ?
> Holy shit, so you are uh building a search engine from scratch.
Yup
> Do you crawl yourself?
Yup
> What is your infrastructure?
All custom built in Java, sitting on a rack server in a basement in Sweden.
> What is your goal for search.marginalia ?
I'm basically building what I feel is lacking in internet search and discovery, which is tools for finding stuff based on something other than a popularity metric, as those tend to feed into themselves to make the web seem so small.
marginalia_nu|9 months ago
Yup
> Do you crawl yourself?
Yup
> What is your infrastructure?
All custom built in Java, sitting on a rack server in a basement in Sweden.
> What is your goal for search.marginalia ?
I'm basically building what I feel is lacking in internet search and discovery, which is tools for finding stuff based on something other than a popularity metric, as those tend to feed into themselves to make the web seem so small.
nichol4s|9 months ago
Can you give some rough indications of how many pages you index in total? How many page you crawl each day? Size of the machine(s) in RAM and HDD?
Sorry, many questions, just genuinely intrigued!