Show HN: oLLM – LLM Inference for large-context tasks on consumer GPUs

[Haeuserschlucht]

20 minutes is a huge turnoff, unless you have it run over night.... Just to get the hint that you should exercise self care in the morning when presenting a legal paper and have the ai check it for flaws.

[anuarsh]

We are talking about 100k context here. 20k would be much faster, but you won't need KVCache offloading for it

[Haeuserschlucht]

It's better to have software erase all private details from text and have it checked by cloud ai to then have all placeholders replaced back at your harddrive.

[attogram]

"~20 min for the first token" might turn off some people. But it is totally worth it to get such a large context size on puny systems!

[anuarsh]

Absolutely, there are tons of cases where interactive experience is not required, but ability to process large context to get insights.

[anuarsh]

Hi everyone, any comments or questions are appreciated