[emeraldd]

My experience with AI code reviews has been very mixed and more on the negative side than the positive one. In particular, I've had to disable the AI reviewer on some projects my team manages because it was so chatty that it caused meaningful notifications from team members to be missed.

In most of the repos I work with, it tends to make a large number of false positive or inappropriate suggestions that are just plain wrong for the code base in question. Sometimes these might be ok in some settings, but are generally just wrong. About 1 in every 10~20 comments is actually useful or something novel that hasn't been caught elsewhere etc. The net effect is that the AI reviewer we're effectively forced to use is just noise that get's ignored because it's so wrong so often.

[fusslo]

one person proved the uselessness of ai reviews for our entire company.

He'd make giant, 100+ file changes, 1000+ worded PRs. Impossible to review. eventually he just modified the permissions to require a single approval, approves his changes and merges. This is still going on, but he's isolated to repos he made himself

He'd copy/paste the output from AI on other people's reviews. Often they were false positives or open ended questions. So he automated his side, but doubled or tripled the work of the person requesting the review. not to mention the ai's comments were 100-300 words with formatting and emojis.

The contractors refused to address any comments made by him. Some felt it was massively disrespectful as they put tons of time and effort into their changes and he can't even bother to read it himself.

It got to the CTO. And AI reviews have been banned.

But it HAS helped the one Jr guy on the team prepare for reviews and understand review comments better. It's also helped us write better comments, since I and some others can be really bad at explaining something

[Smaug123]

This seems like quite a dramatic organisational response to a single incompetent employee!

[syntheticcdo]

Sometimes the only review a PR needs is "LGTM" - something today's LLMs are structurally incapable of.

[Smaug123]

When you say "structurally incapable", what do you mean? They can certainly output the words (e.g. https://github.com/Smaug123/WoofWare.Zoomies/pull/165#issuec... ); do you mean something more along the lines of "testing can prove the presence of bugs, but not their absence", like "you can't know whether you've got a false or true negative"? Because that's also a problem with human reviewers.

[CuriouslyC]

The solution to this is to get agents to output an importance scale (0-1) then just filter anything below whatever threshold you prefer before creating comments/change reqs.

[Smaug123]

What model were you using? This is true for e.g. the best Qwen model, which I believe to be too dumb to be useful, but not GPT-5.0 or above in my experience (and I consider myself to be a pretty demanding customer).

[insin]

I love having to hit Resolve Conversation umpteen times before I can merge because somebody added Copilot and it added that many dumb questions/suggestions