top | item 37420281

Kagi Small Web

668 points| u2077 | 2 years ago |blog.kagi.com

185 comments

order

freediver|2 years ago

Vlad from Kagi here, thanks for posting. The RSS feed unexpectedly broke (edit: the feed is back up! [3]) just as we published the blog post, in the true spirt of "small web" :) Should be up in 30 minutes which will enable the site to function too (it uses the same feed).

This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].

Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.

You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).

[1] https://github.com/kagisearch/smallweb

[2] https://github.com/kagisearch/smallweb/blob/main/smallyt.txt

[3] https://kagi.com/smallweb

mhitza|2 years ago

I really enjoy that this feature got released. Adding more visibility to the indie/small web is a great thing. I've complained in the past on HN that any useful personal blog is buried under spammy content in search engines.

It would be nice to also be able to just search within the small web, maybe using a lens in the future?

Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.

palmer_fox|2 years ago

The more I uses Kagi (paid) the more I am feeling sad for the possibility that it might go away if it doesn't get more traction.

For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.

Thank you.

mrcgnc|2 years ago

I'm a paying customer and this is yet another nice addition. It's something only you can do. Please keep doing this kind of stuff!

sergiosgc|2 years ago

Congratulations on the initiative. The results seem cool, and integration in the main search results may give a glimpse outside the big web.

I'm a Kagi customer, and a very happy one. The search engine is amazingly good. This only makes me happier with my search engine choice.

aftergibson|2 years ago

No other search engine would do this and it's awesome! Always feel like my subscription is well worth it. Thanks Vlad!

raybb|2 years ago

Why do you let YouTube channels have advertisements/sponsors but require sites to be ad-free?

I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.

mbb70|2 years ago

FYI your "Why pay for search" and "FAQ" links on https://kagi.com/pricing are broken, perhaps another refactoring victim?

epilys|2 years ago

Discord really? why not IRC? I don't think having to use proprietary communication software that require accounts did with small web and open source.

manuelmoreale|2 years ago

This is so neat!

So happy so see the small, independent and more humane web being highlighted.

I’m trying to do my part[0] but I have no doubt that a search engine—even if still a niche one— can have a much bigger impact.

Really well done Vlad!

[0] https://peopleandblogs.com

BoorishBears|2 years ago

Semi-unrelated, but is there any reason you don't make your search API as turn key as the other APIs?

It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.

matheusmoreira|2 years ago

I have been developing a "small web" static site, forked and customized a templating engine into a static site generator just for my site. Even wrote a new post recently, last week or so. How can I add it to the list? The GitHub says to make a pull request, is that really all there is to it?

Edit: just saw this:

> Do not submit your own website.

I see. I'm okay with that. Maybe it will show up there one day.

VapidLinus|2 years ago

Are English "Small Web" results included even if I use the "Sweden" search region?

I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.

coding123|2 years ago

Honestly the reality is a good coder is one that keeps chugging and doesn't give up.

digitalsin|2 years ago

This is absolutely fantastic, thank you!

arboles|2 years ago

> These notes will vanish in about a week as we cycle in new content - emphasizing the fleeting, imperfect nature of the small web.

Kagi could just admit they don't want to moderate notes or store them permanently. No need to push down the small web, because a lot of small sites preserve their content.

I get that Kagi probably has data indicating the reality of how often sites down, but it seems from my experience that content in big platforms disappears often as well, even in the cases where the creator hadn't forgotten about it. The "Small web" websites made by a creator that cares have the room to be much more permanent.

slushh|2 years ago

Where do those notes appear?

It would be nice if Kagi Small Web would have an ActivityPub interface so that the most appreciated sites of a day could be added to a timeline on mastodon or lemmy.

wink|2 years ago

That's a funny sentence indeed, most personal sites take painstaking care to not break their 10 or 20 year old links :P

kickdaddy|2 years ago

Kagi is worth every penny. Been using it for 6 months as my primary search engine. I think I have used the Google !g 10 or so times in that period maybe?

packetlost|2 years ago

The ability to block SEO garbage sites like GeeksForGeeks and not filling the first 3-5 results with ads is worth it alone. Not to mention the ability to boost certain sites results over others on a personalized level. Unfortunately for me, I end up regularly going over the "Pro" tier of searches a month (partially because I accidentally search all the time, but also I'm heavy search-engine user) is a bit of a letdown. At least they have an option to purchase additional searches instead of going up to the rather expensive unmetered tier.

deepspace|2 years ago

It is such a weird experience to do a Google search on someone else's computer after being used to Kagi. I recently requested a small usability enhancement, and it was implemented within a few weeks. Zero chance of that happening with any other search engine.

The search results are consistently better than anyone else's, including DuckDuckGo, so I am, and will remain, a happy paying customer.

gnyman|2 years ago

Yes, second that. Happy paying user for bit over a year now (and was in the beta before that).

It feels a bit like how it felt discovering Google back when AltaVista was still a thing.

scroot|2 years ago

Another very satisfied Kagi user here. It's totally worth the money, and a great example of the kinds of services that are possible if you're just willing to pay a little for 'em

imiric|2 years ago

To give a slightly less positive perspective: I trialed Kagi for a few weeks, and while I liked the features and their business model, its search results are no better than from my own Searx instance. Kagi essentially does the same thing as Searx: anonymized API calls to 3rd party search engines. It's just packaged in a friendlier UI, but the experience is not far off.

Plus, Searx supports many more search engines, and I can customize it exactly to my liking.

I wish them well, as they clearly have good intentions and a good product, but I prefer using an equivalent OSS and self-hosted solution over a proprietary SaaS I have to create an account for, even if it's not as polished or featureful.

EDIT: Actually, I'm wrong. Kagi apparently also has their own crawlers and indexes[1]. Still, I'm not finding Searx results to be deficient, so I'm not missing out on much.

One thing that does concern me with Searx, and partially with Kagi, is that those 3rd parties could decide to block these API requests at any point, leaving Searx unusable, and Kagi's results less relevant. I'm not sure this is a sustainable way to build a search engine, but I do appreciate both Kagi's and Searx's stance on ads. Using any mainstream search engine via their own frontends is a frustrating experience at best.

[1]: https://help.kagi.com/kagi/search-details/search-sources.htm...

waveBidder|2 years ago

one nice feature I found recently: you can actually search special characters: R "%<>%" actually explains operator, rather than linking to random websites about R.

rpastuszak|2 years ago

I’m mostly happy with ddg but I wish they had a feature allowing me to disable certain sites by default like kagi, even if that information was stored in local storage only and I’d have to add the exclude list manually.

slipperlobster|2 years ago

My only worry is random searches, where I default to a "free" bang (!g, !b, etc.) to not burn through one of my paid searches. I know it's a mental thing, but it's still a thing.

mberning|2 years ago

I like it but a month after paying for it the subscription model was changed significantly to be way more expensive. I just can’t let myself become depending on something with such capricious business practices. Maybe it is a positive indication that they are better at product than they are business and customer relations.

dustincoates|2 years ago

I love it, though my only complaint is that I usually want sites in English from the US, but if I want to switch to finding sites in French in France, I have to switch regions. Google is much better at localizing me, and so I'll use !g for those.

Yiin|2 years ago

same, and realized in those cases that google results were still worse compared to whatever Kagi found.

unshavedyak|2 years ago

This sort of stuff makes me really happy to be a Kagi subscriber. Not only do i get value out of Kagi, but this shows me that the money is being used to develop Kagi in a way i agree with. By comparison, Spotify (just picking one of my subs) feels hostile to me. I pay them, but would cancel in a heart beat if i felt i had options.

I really appreciate Kagi's development matching what i feel like i'm buying. Thanks Kagi Team <3

Subsector6616|2 years ago

Music streaming is such a mess. I've tried a ton of services in an attempt to find one that's decent, and have found embarrassing problems for all of them.

Spotify: Really hostile, manipulative, and terrible to artists. On my end I hate how commercial the homescreen is, and the CarPlay interface is just a user-hostile to a degree that is frankly unsafe disaster.

TIDAL: Pretty good in a lot of ways. They pay their artists well. The recommendations are decent. The apps have some really stupid bugs that have persisted for years though, the most annoying of which is that if you shuffle a playlist it only shuffles the dozen or so tracks that the interface had pre-cached from the top. So if you try to shuffle your full library you wind up just hearing the same dozen songs only, over and over again.

Deezer: Wanted to like it, but the apps aren't great, and I ran into more missing tracks than I'd like.

So I finally settled on Apple Music. I've got an iPhone, so it's a natural fit there, and the CarPlay interface is great. They also pay artists almost as well as TIDAL. The recommendations are super good enough, and I don't really feel like the home page is constantly trying to push me to whatever the huge labels are paying them to promote (damn you Spotify). The Windows apps are terrible (like flat out embarrassing), and the Linux apps are non-existent, but luckily there's a pretty great open source app called Cider that solves that.

jefftk|2 years ago

It looks like you're stripping MathML out of the RSS feed -- is that intentional, or are you using an older sanitizer that doesn't recognize it? For example, my RSS feed [1] for my most recent post [2] has:

    &lt;math display=&quot;block&quot;&gt;
    &lt;msup&gt;&lt;mi&gt;e&lt;/mi&gt;
          &lt;mrow&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/mrow&gt;&lt;/msup&gt;
    &lt;/math&gt;
Which shows up in your RSS feed [3] as:

    e
          kt
[EDIT: filed an issue: https://github.com/kagisearch/smallweb/issues/10]

[1] https://www.jefftk.com/news.rss

[2] https://www.jefftk.com/p/weekly-incidence-vs-cumulative-infe...

[3] https://kagi.com/api/v1/smallweb/feed

marginalia_nu|2 years ago

Some Q:s

#1 What's the rationale behind favoring recent blog updates? In my experience, recent updates make the weakest search results, more prone to updates and link breakage, and overall tend to be of lower quality. I also wonder if promoting recent content might incentivize pumping out low-quality entries to increase the odds of being listed.

#2 In dabbling in the domain I've always ended up with an almost absurd skew toward technical programmer:y blogs. While there is a strong overlap between the cohort with a blog, and the cohort with programmer interests, I feel it would be more inviting to other groups if other interests were better represented. Is this something you've thought about, and if so, what do you think might be done?

freediver|2 years ago

Hey Viktor!

#1 There are a couple of factors.

- More recent content tends to be more relevant given the same search query, or at least its freshnes will contribute to it not being completely irrelevant (which is the worse thing you want in a search engine). The quality of it is already guaranteed to some extent by this being a curated list to begin with.

- It was relatively easy to assemble and maintain the list because it relies on RSS feed tech, and there were a lot of sources to seed it.

- Focus on recent writing can encourage some people to write (more) as we had an example highlighted in the blog post. In general, the web needs more high quality, non-commercial content, and this is ultimately what we want to contribute towards with this. By providing a platform (even if very small) to encourage this behavior we get a step closer to the web we like.

#2 In general I agree. Although I should say we did spend effort to create a diverse pool of websites for this initiative (for example I am seeing a lot of economy or photography). Again, we can only encourage the creation of more content in various areas through platforms like Kagi (and Marginalia) and hope that it will work out at the end.

mindwork|2 years ago

This is amazing! Over the last 2 months I've been on a personal journey to browse through small web links. It all started from the "Ask HN: Could you share your personal blog here?" post on HN, and then I found out that somebody put a website with the links from this post organized(https://dm.hn/). I exported all 1651 blog link in to XLS and now every now and then I open up 5-7 just to read random articles and mark it as "viewed". So far I've been through 250 out of 1651!

I wish kagi has something similar, one place where I can see all the links to the personal websites collected via all it's sources

benhoyt|2 years ago

Thanks for the prominent link to my article! ("The Small Web is Beautiful") I consider that my personal software manifesto, so I'm glad to see it being promoted more widely.

I really like what you're doing with Kagi Small Web -- love that you've taken the initiative to start surfacing all this excellent content. Keep up the good work. I think I'll try out Kagi search...

freediver|2 years ago

Thanks for writing that Ben, it is truly an inspirational piece. I hope you like your Kagi experience.

vladstudio|2 years ago

Vlad, you're awesome! It's hard to tell to what extent it is actually beneficial for users or your company, but I feel warm and fuzzy inside knowing these kinds of things still happen in modern Internet.

freediver|2 years ago

Hah, thanks. It was one of those things that I just wanted to do and I actually wrote 100% of code for this end to end which took me a good part of a month. Not the best use of CEOs time you might say, but it was fun and felt right.

sotix|2 years ago

I would love to have my website added to the list because I block all crawlers in my robots.txt, but it looks like I’m not allowed to submit my own website. Is there a way to go about having it added? It would be great to be part of kagi search results but don’t want to be scraped by Google, ChatGPT, or the next group, so I found it easiest to block everything.

thrtythreeforty|2 years ago

This is quite interesting. I feel you should include content from blogs that publish less often than once a week - some of the best blogs post once in a blue moon and are still quite interesting.

landgenoot|2 years ago

> We gather new content, published within the last week, from a handpicked list of blogs and surface it in multiple ways.

I think the only condition is that the article is less than one week old.

freediver|2 years ago

We do as a part of our own search index, just not as a part of this particular initiative where we wanted a focus on the fresh stuff from the small web "oven".

rebeccaskinner|2 years ago

I’ve been using Kagi for over a year and it just keeps getting better. I really appreciate Kagis user and privacy-centric view of the web, and I love how the quality of the product proves that sticking to those ideals is really a better way to build. I hope they continue to gain enough momentum with non-tech audiences to continue to exist for a long time.

blitz_skull|2 years ago

Things like this are why I keep rooting for Kagi. I haven’t purchased a subscription yet because I’m still on my free 100 queries, but I will as soon as they run out. I really love this project!

jabroni_salad|2 years ago

the smallwebsite is delightful. It reminds me of Stumbleupon, which I miss dearly.

And of course it showed me Questionable Content, which I first got to via stumbleupon.

Einenlum|2 years ago

I also thought right away about stumbled upon. It was an amazing idea and great implementation too.

ot1138|2 years ago

Nice, but I can't figure out how to search it. The "Small Web" site linked to in the article just shows a number of links to Kagi articles.

freeAgent|2 years ago

It sounds like it’s integrated into Kagi’s main results and you can use the API to reference only the small web results, but I also don’t see a way for a Kagi search user to toggle “small web” results as a lens or whatever. I assume and hope that may be added in the future.

ilrwbwrkhv|2 years ago

This is really good. With more and more of the large web filled with SEO crap and gargantuan articles written because that's what google wants, it is great to read from the people who actually write the interesting things the AIs train on.

jci|2 years ago

Very happy paying Kagi search and Orion browser user here. Just want to send some general good vibes your way. I really love what you guys are doing, and this just adds to that.

vk7eu|2 years ago

Great idea!

https://kagi.com/smallweb displays the homepage for the Kagi Blog at the moment, though.

freediver|2 years ago

We had a 'glitch', it should be fixed now.

pavlov|2 years ago

Great initiative, but it feels like they're not quite eating their own dogfood. This post contains a link titled "highlighting blog posts from HN users" which goes to Twitter — a massive centralized social media app that's pretty much the opposite of "small web".

mbwgh|2 years ago

Kagi is the ten bucks per month that just keep giving.

ecshafer|2 years ago

This is cool. The web has become too centralized.

goplayoutside|2 years ago

Another satisfied Kagi customer here ('early adopter's badge). Just wanted to chime in and say I think KSW is awesome. I'll happily renew my annual subscription.

uconnectlol|2 years ago

getting tired of hearing about the old web. and still not seeing anything that resembles it. everything still needs js. everything still needs cloudflare bypasses if you view it on tor. this includes whatever cool hacker site you think you have because i almost never see these. i have had a site with just plain html for 10 years and it has no nonsense just content, you can do this too, it doesnt need a movement that just talks about doing it.

not that it matters since the old web wasn't good. it was as terrible as now. the UI of absolutely every website ever made has been terrible quirky garbage compared to something like windows 98. Even back then there was a massive difference going from windows 98 (sane GUI) to web (garbage hackjob GUI + ads (YES REMEMBER 40 POPUPS? ADS? TOOLBARS? THE OLD WEB WAS NOT GOOD IT WAS A HELL JUST LIKE NOW)).

the content was never good either. every topic discussed on the web is little cliques who believe some easy to digest nonsense and then if you go skim some books on the subject the meta is completely different. except programming since that just centers around the web [1]. think of anything else like cooking or engineering

the web is a terrible protocol that should have died 20 years ago and been replaced with something that was modern at the time like freenet (and they should have made an alternative to html etc).

1. and this is ironic too since programming is the one field that is steered by the web's body of pseudoknowledge and as a result you have people who think C, PHP, and OOP are legitimate programming practices.

fLaMEd|2 years ago

I think you’re misunderstanding and confusing “old web” with “small web”.

endorphine|2 years ago

As a SWE/SRE, how does Kagi compare in quality of results (i.e. finding you the answer easily) vs. DuckDuckGo and Google?

bigpapikite|2 years ago

I haven't really had issues. They have a "programming" toggle that filters results to what looks like mainly forum type content like on github or stackoverflow, but I don't use it often, I'm sure some people do though

w0mbat|2 years ago

I find this kind of domain name reuse unsettling: for years, Kagi.com was the address of a payment service for shareware authors. Man those were the days. Lots of people were perfectly willing to pay me $20 for a useful little utility app and now they won't pay $2 in an App Store.

eitland|2 years ago

> Lots of people were perfectly willing to pay me $20 for a useful little utility app and now they won't pay $2 in an App Store.

what kind of utilities?

I am happy to pay $2 (or more) for tools I use if it is a one time payment or a payment for tokens (e.g. $n for $m ocr scans, generated images or whatever).

I loathe apps that demand monthly payments unless it is really understandable why they have to have it that way (service that require permanent storage comes to mind, although I think I only use iCloud for storage now).

k0k0r0|2 years ago

Kagi appearently had a project "expertGPT" (contrast to FastGPT). Does anybody know what happened to that one?

On a side note, there is now a - to my best knowledge - completely unrelated product "ExpertGPT" from some totally different company. I am not talking about that one.

freediver|2 years ago

Launching Oct 2, as a part of Kagi Assistant. Already in beta with a number of members.

caskstrength|2 years ago

And yet in the article they reference their Discord channel and Twitter sigh

freediver|2 years ago

Rooting for small web does not mean one needs to be against other technology.

For example Kagi uses Google and even hosts on GCP - I think Google's technology and people are great, it is just the business model that is rotten and contributes to the deterioration of the web.

And interestingly enough, at least Discord (and to some extent Twitter) are trying to have a business model that does not put more ads down your throat (although admittedly I did cancel my Twitter subscription as unexplainably they still showed ads even when subscribing - you can't sit on two chairs).

toastal|2 years ago

They also mentioned their Microsoft GitHub forge to file issues. They really want you to create proprietary accounts to interact with them. You’d think a small web initiative would be led with small, decentralized/federated, libre software choices.

vinceguidry|2 years ago

Can't reinvent everything all at once lol.

tailspin2019|2 years ago

I could already tell I was going to like this from the title alone!

fLaMEd|2 years ago

This is great to see, what a fantastic addition to Kag,i i.

Yeah I see similarities to marginalia, but it’s great to have multiple services for the small web.

I need to get my website on the lists asap!

replwoacause|2 years ago

This is great. Just wanting to chime in that I am also a happy paying customer, and love seeing this kind of work from Kagi. Keep it up!

Gud|2 years ago

Thanks a lot for this. I really appreciate kagi and the work you put in to it.

Now more than ever do we need a user friendly search engine.

58ok5|2 years ago

Okay, it's great, and now I'm existed >> Now how can I raise those results in my personalized results?

bennyp101|2 years ago

Pretty cool - just wondering why one of the requirements is "The website can appear in an iframe"?

freediver|2 years ago

That is a requirement for it to be featured in the KSW website (as it uses iframes to embed). We will still crawl it and surface in search results regardless..

freeAgent|2 years ago

It looks like that’s because that’s how the pages are displayed through this feature.

MatthiasPortzel|2 years ago

This sort of centralized, hand-curated list makes me nervous. The beauty of the web is that it’s open, you don’t have to “apply” or meet “criteria” for your website to viewable by other people. Obviously the status-quo (of Google Search) isn’t great, and I’m glad Kagi is trying to fix this problem. But this isn’t going to scale.

scarmig|2 years ago

I can't see how it will scale, but in the immediate present, it seems useful and good to me. I might even prefer if this small web remained small (though that's likely... unappealing to Kagi's investors).

58ok5|2 years ago

Great initiative, now how can I raise those results above all my already raised domains?

gcoguiec|2 years ago

Very excited about this!

nsonha|2 years ago

so a search engine "handpicked" internet content for you? We already have social network for that.

sdm|2 years ago

> Content must be in English (hard to curate non-English).

Well this is disappointing. It's no harder to curate other languages. You're just say you don't care.

kyleee|2 years ago

Unfortunately people don’t have unlimited time/energy/money. It may not be that they don’t care, but rather they are expending their full effort on other tasks