[sobiolite]

The article says that LLMs don't summarize, only shorten, because...

"A true summary, the kind a human makes, requires outside context and reference points. Shortening just reworks the information already in the text."

Then later says...

"LLMs operate in a similar way, trading what we would call intelligence for a vast memory of nearly everything humans have ever written. It’s nearly impossible to grasp how much context this gives them to play with"

So, they can't summarize, because they lack context... but they also have an almost ungraspably large amount of context?

[usefulcat]

I think "context" is being used in different ways here.

> "It’s nearly impossible to grasp how much context this gives them to play with"

Here, I think the author means something more like "all the material used to train the LLM".

> "A true summary, the kind a human makes, requires outside context and reference points."

In this case I think that "context" means something more like actual comprehension.

The author's point is that an LLM could only write something like the referenced summary by shortening other summaries present in its training set.

[jjaksic]

But "shortening other summaries from its training set" is not all an LLM is capable off. It can easily shorten/summarize a text it had never seen before, in a way that makes sense. Sure, it won't always summarize it the same way a human would, but if you do a double blind test where you ask people whether a summary was written by AI, a vast majority wouldn't be able to tell the difference (again this is with a completely novel text).

[jchw]

I think the real takeaway is that LLMs are very good at tasks that closely resemble examples it has in its training. A lot of things written (code, movies/TV shows, etc.) are actually pretty repetitive and so you don't really need super intelligence to be able to summarize it and break it down, just good pattern matching. But, this can fall apart pretty wildly when you have something genuinely novel...

[strangattractor]

Is anyone here aware of LLMs demonstrating an original thought? Something truly novel.

My own impression is something more akin to a natural language search query system. If I want a snippet of code to do X it does that pretty well and keeps me from having to search through poor documentation of many OSS projects. Certainly doesn't produce anything I could not do myself - so far.

Ask it about something that is currently unknown and it list a bunch of hypotheses that people have already proposed.

Ask it to write a story and you get a story similar to one you already know but with your details inserted.

I can see how this may appear to be intelligent but likely isn't.

[gus_massa]

Humans too. If I were too creative writing the midterm, most of my students would fail and everyone would be very unhappy.

[jjaksic]

And what truly novel things are humans capable of? At least 99% of the stuff we do is just what we were taught by parents, schools, books, friends, influencers, etc.

Remember, humans needed some 100, 000 years to figure out that you can hit an animal with a rock, and that's using more or less the same brain capacity we have today. If we were born in stone age, we'd all be nothing but cavemen.

What genuinely novel thing have you figured out?

[btown]

It's an interesting philosophical question.

Imagine an oracle that could judge/decide, with human levels of intelligence, how relevant a given memory or piece of information is to any given situation, and that could verbosely describe which way it's relevant (spatially, conditionally, etc.).

Would such an oracle, sufficiently parallelized, be sufficient for AGI? If it could, then we could genuinely describe its output as "context," and phrase our problem as "there is still a gap in needed context, despite how much context there already is."

And an LLM that simply "shortens" that context could reach a level of AGI, because the context preparation is doing the heavy lifting.

The point I think the article is trying to make is that LLMs cannot add any information beyond the context they are given - they can only "shorten" that context.

If the lived experience necessary for human-level judgment could be encoded into that context, though... that would be an entirely different ball game.

[entropicdrifter]

I agree with the thrust of your argument.

IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops.

For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.

[tovej]

You can reconcile these points by considering what specific context is necessary. The author specifies "outside" context, and I would agree. The human context that's necessary for useful summaries is a model of semantic or "actual" relationships between concepts, while the LLM context is a model of a single kind of fuzzy relationship between concepts.

In other words the LLM does not contain the knowledge of what the words represent.

[neerajsi]

> In other words the LLM does not contain the knowledge of what the words represent.

This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation.

If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.

[ratelimitsteve]

I think the differentiator here might not be the context it has, but the context it has the ability to use effectively in order to derive more information about a given request.

[cainxinth]

This can be solved with prompting. You can say: “summarize this document, but don’t just recap, give me the big picture” or anything to that effect.

[kayodelycaon]

They can’t summarize something that hasn’t been summarized before.

[timmg]

About a year ago, I gave a film script to an LLM and asked for a summary. It was written by a friend and there was no chance it or its summary was in the training data.

It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.

[originalcopy]

I'd like to see some examples of when it struggles to do summaries. There were no real examples in the text, besides one hypothetical which ChatGPT made up.

I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.

[naikrovek]

they can, they just can't do it well. at no point does any LLM understand what it's doing.