[syrak]

> If the underlying premise is flawed, who cares if the methodology is correct?

How do we know the premise is flawed?

> The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice.

How do you concretely do that? The initial paper, even if flawed, did a lot of work to try to control for confounding factors. To do what you said, you would have to put as much effort, if not more, to convince people that the results can be falsified to claim anything, with the same level of control in place.

It's not even clear that counting commits is entirely meaningless. Getting some signal out of it does seem tremendously difficult, but not impossible. So questioning the methodology is really the only way to rebut the experiment.