[SJMG]

> there is a smaller % that benefits and learns more and faster

That's not what the study says nor it is capable of credibly making that claim. You are reasoning about individuals in an RCT where subjects did not serve as their own control. The high performers in the treatment group may have done even better had they been in the control and AI is in fact is slowing them down.

You don't know which is true because you can't know because of the study design. This is why we have statistics.

[epolanski]

So you don't doubt their conclusion that most sucked by using AI, but you doubt that they found that some learned more?

[SJMG]

The conclusion of the paper doesn't say that "most sucked using AI". It's says the mean quiz score was both significantly and sizably lower in the intervention group vs the control. No significant difference detected on speed.

The qualitative breakdown says how you use AI matters for understanding. It doesn't say some learned more than the control group and even if it did, it's not powered to show a statistical difference which is one of the only things keeping a study from not being another anecdote on the internet.

For the sake of argument let's say there is an individual in the treatment arm who scored higher than the highest control participant. What some want that to mean is, "Some engineers perform better using AI". It does not say that. That could be an objective fact(!), it doesn't matter. This study will not support it; it's an RCT. What if that programmer is just naturally gifted or lucky(!). This is the point of statistics.

The best you can do with outliers is say "AI usage didn't hinder some from attaining a high score" (again maybe it would have been higher w/o you just can't reason about individuals in a study like this).

I hope this helps.