Interesting. LLMs are already shining at PR reviews even if they struggle with massive codebases right now. And they are evolving fast enough that those scaling limits might not stay limits much longer.
I'd love to see your evidence that "LLMs are already shining at PR reviews". We've used a handful of them here where I work for months now and they are rarely correct, and thus, rarely useful. Instead they tend to just summarize nonsense that wasn't even introduced in that PR, make shit up entirely, or recommend bad fixes to things that would be better solved by being removed entirely.
Fair point. I think the bottom line is that it depends a lot on the context and how the prompt is framed. For PRs with small enough scope, I have seen LLMs provide decent value, mostly in surfacing potential issues or offering quick summaries. That said, the Semgrep analysis highlights that accuracy and coverage still fall short even in these narrow cases, so clearly there is still a lot of work to be done before this becomes broadly reliable.
kogatlas|5 months ago
aafanah|5 months ago