[dartos]

you're talking about specifically using genetic programming to create new programs as opposed to gradient decend in LLMs to minimize a loss function, right?

How would you construct a genetic algorithm to produce natural language like LLMs do?

Forgive me if i'm misunderstanding, but in programming we have "tokens" which are minimal meaningful bits of code.

For natural languages it's harder. "Words" are not super meaningful on their own, i don't think. (at least not as much as a token) so how would you break down natural language for a genetic algorithm?

[bob1029]

> how would you break down natural language for a genetic algorithm?

The entire point is that you do not bother trying. From an information theory and computational perspective, raw UTF-8 bytes can work just as well as "tokens".

The program that is being evolved is expected to develop whatever strategy is best suited to providing the desired input/output transformation. Back to the bitter lesson on this one.

[dartos]

I’ll need to read up on genetic algorithms, I think.

That sounds really cool, but coming from training other statistical models, im having a hard time imagining what the training loop looks like.