[barishnamazov]

The turkey is fed by the farmer every morning at 9 AM.

Day 1: Fed. (Inductive confidence rises)

Day 100: Fed. (Inductive confidence is near 100%)

Day 250: The farmer comes at 9 AM... and cuts its throat. Happy thanksgiving.

The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.

This is why Meyer's "American/Inductive" view is dangerous for critical software. An LLM coding agent is the Inductive Turkey example. It writes perfect code for 1000 days because the tasks match the training data. On Day 1001, you ask for something slightly out of distribution, and it confidently deletes your production database because it added a piece of code that cleans your tables.

Humans are inductive machines, for the most part, too. The difference is that, fortunately, fine-tuning them is extremely easy.

[p-e-w]

> The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.

But we already know that LLMs can do much better than that. See the famous “grokking” paper[1], which demonstrates that with sufficient training, a transformer can learn a deep generalization of its training data that isn’t just a probabilistic interpolation or extrapolation from previous inputs.

Many of the supposed “fundamental limitations” of LLMs have already been disproven in research. And this is a standard transformer architecture; it doesn’t even require any theoretical innovation.

[1] https://arxiv.org/abs/2301.02679

[barishnamazov]

I'm a believer that LLMs will keep getting better. But even today (which might or might not be "sufficient" training) they can easily run `rm -rf ~`.

Not that humans can't make these mistakes (in fact, I have nuked my home directory myself before), but I don't think it's a specific problem some guardrails can solve currently. I'm looking for innovations (either model-wise or engineering-wise) that'd do better than letting an agent run code until a goal is seemingly achieved.

[encyclopedism]

LLM's have surpassed being Turing machines? Turing machines now think?

LLM's are known properties in that they are an algorithm! Humans are not. PLEASE at the very least grant that the jury is STILL out on what humans actually are in terms of their intelligence, that is after all what neuroscience is still figuring out.

[usgroup]

This issue happens at the edge of every induction. These two rules support their data equally well:

data: T T T T T T F

rule1: for all i: T

rule2: for i < 7: T else F

[p-e-w]

That’s where Bayesian reasoning comes into play, where there are prior assumptions (e.g., that engineered reality is strongly biased towards simple patterns) which make one of these hypotheses much more likely than the other.

[mirekrusin]

AGI is when turkey cuts farmer's throat on day 249, gets on farmer's internet, makes money on trading and retires on an island.

[funkyfiddler69]

> The difference is that, fortunately, fine-tuning them is extremely easy.

Because of millions of years of generational iterations, by which I mean recursive teaching, learning and observing, the outcomes of which all involved generations perceive, assimilate and adapt to in some (multi-) culture- and sub-culture driven way that is semi-objectively intertwined with local needs, struggles, personal desires and supply and demand. All that creates a marvelous self-correcting, time-travelling OODA loop. []

Machines are being finetuned by 2 1/2 generations abiding by exactly one culture.

Give it time, boy! (effort put into/in over time)

[] https://en.wikipedia.org/wiki/OODA_loop

[myth_drannon]

"fine-tuning them is extremely easy." Criminal courts, jails, mental asylums beg to disagree.

[marci]

"finetune"

Not

"Train from scratch"

[aleph_minus_one]

> The difference is that, fortunately, fine-tuning them is extremely easy.

If this was true, educating people fast for most jobs would be a really easy and solved problem. On the other hand in March 2018, Y Combinator put exactly this into its list of Requests for Startups, which gives strong evidence that this is a rather hard, unsolved problem:

> https://web.archive.org/web/20200220224549/https://www.ycomb...

[armchairhacker]

Easier than to an LLM, compared to inference.

“‘r’s in strawberry” and other LLM tricks remind me of brain teasers like “finished files” (https://sharpbrains.com/blog/2006/09/10/brain-exercise-brain...). Show an average human this brain teaser and they’ll probably fall for it the first time.

But never a second; the human learned from one instance, effectively forever, without even trying. ChatGPT had to be retrained and to not fall for the “r”’s trick, which cost much more than one prompt, and (unless OpenAI are hiding a breakthrough, or I really don’t understand modern LLMs) required much more than one iteration.

That seems to be the one thing that prevents LLMs from mimicking humans, more noticeable and harder to work around than anything else. An LLM can beat a Turing test where it only must generate a few sentences. No LLM can imitate human conversation over a few years (probably not even a few days), because it would start forgetting much more.

[graemep]

The problem with education is that existing ways of doing things are very strongly entrenched.

At the school level: teachers are trained, buildings are built, parents rely on kids being at school so they can go out to work....

At higher levels and in training it might be easier to change things, but IMO it is school level education that is the most important for most people and the one that can be improved the most (and the request for startups reflects that).

I can think of lots of ways things can be done better. I have done quite a lot of them as a home educating parent. As far as I can see my government (in the UK) is determined to do the exact opposite of the direction I think we should go in.

[naveen99]

LLM’s seem to know about farmers and turkeys though.

[glemion43]

You clearly underestimate the quality of people I have seen and worked with. And yes guard rails can be added easily.

Security is my only concern and for that we have a team doing only this but that's also just a question of time.

Whatever LLMs ca do today doesn't matter. It matters how fast it progresses and we will see if we still use LLMs in 5 years or agi or some kind of world models.

[barishnamazov]

> You clearly underestimate the quality of people I have seen and worked with.

I'm not sure what you're referring to. I didn't say anything about capabilities of people. If anything, I defend people :-)

> And yes guard rails can be added easily.

Do you mean models can be prevented to do dumb things? I'm not too sure about that, unless a strict software architecture is engineered by humans where LLMs simply write code and implement features. Not everything is web development where we can simply lock filesystems and prod database changes. Software is very complex across the industry.

[bdbdbdb]

> You clearly underestimate the quality of people I have seen and worked with

"Humans aren't perfect"

This argument always comes up. The existence of stupid / careless / illiterate people in the workplace doesn't excuse spending trillions on computer systems which use more energy than entire countries and are yet unreliable