[ariwilson]

Is there value in adding an overseer LLM that measures the progress between n steps and if it's too low stops and calls out to a human?

[CuriouslyC]

I don't think you need an overseer for this, you can just have the agent self-assess at each step whether it's making material progress or if it's caught in a loop, and if it's caught in a loop to pause and emit a prompt for help from a human. This would probably require a bit of tuning, and the agents need to be setup with a blocking "ask for help" function, but it's totally doable.

[solumunus]

And how does it effectively measure progress?

[NotMichaelBay]

It can behave just like a senior role would - produce the set of steps for the junior to follow, and assess if the junior appears stuck at any particular step.

[p_v_doom]

Bruh, we're inventing robot PMs for our robot developers now? We're so fucked

[suninsight]

Yes it works really well. We do something like that at NonBioS.ai - longer post below. The agent self reflects if it is stuck or confused and calls out the human for help.