Yeah that's one of the best things about them for me. And then I go to the website and often it's some janky UI with content buried super deep. Or it's like Reddit and I immediately get slammed with login walls and a million annoying pop ups. So I'm quite grateful to have an ability to cut through the noise and non-consistency of the wild west web. I agree the idea that we're somewhat killing traffic to the organic web is kind of sad. But at the same time I still go to the source material a lot, and it enables me to bounce more easily when the website is a bit hostile.I wonder if it would be slightly less sad if we all had our own decentralized crawlers that simply functioned as extensions of ourselves.
Zambyte|1 year ago
This is something I'm (slowly) working on myself. I have a local language model server and 30 tb usable storage ready to go, just working on the software :)