[nuz]

Seems like a non pessimistic idea of something LLMs could help us out with. Mass analysis of old texts for new finds like this. If this one exists surely there are many more just a mass analysis away

[steve_adams_86]

I accidentally got Zed to parse way more code than I intended last night and it cost close to $2 on the anthropic API. All I can think is how incredibly expensive it would be to feed an LLM text in hopes of making those connections. I don’t think you’re wrong, though. This is the territory where their ability to find patterns can feel pretty magical. It would cost many, many, many $2 though

[diggan]

> I accidentally got Zed to parse way more code than I intended last night and it cost close to $2 on the anthropic API

Is that one API call or some out of control process slinging 100s of requests?

Must have been a ton of data, as their most expensive model (Opus) seems to $15 per million input tokens. I guess if you just set it to use an entire project as the input, you'll hit 1m input tokens quickly.

[pcthrowaway]

This is a pretty good case for just using a local model. Even if it's 50% worse than Anthropic or whatever the gap is now between open models and proprietary state of the art, it's still likely 'good enough' to categorize a story in an old newspaper as missing from an author's known bibliography.

[hyperbrainer]

The problem with copyright is going to be a big hurdle though.

[diggan]

Why? Old texts would be out of copyright, and even if they weren't, as long as you're not publishing the source material or anything containing the source material (or anything that can verbatim output the source), it seems you'd be in the clear.

[ebiester]

If we go to the era of public domain, there is no worry about copyright.