[SEMW]

> If the intervals overlap, the difference is not statistically significant.

Demonstrably false. Obvious counterexample: the study in the OP, which has overlapping confidence intervals and a statistically significant difference.

Proof: just calculate the 95% confidence interval for the difference between the two means. You can figure out what the stddev was from half the confidence interval divided by the z-score for a 95% confidence interval, 1.96, and you get 1.02 and 1.30 for the two groups. Then the confidence interval is: (10.4 - 6.3) +/- 1.96*sqrt(1.02^2 + 1.30^2) gives [0.86, 7.34]. This does not include 0, therefore the difference is significant.

> The probability that a sample mean for a large sample is above the 90th percentile is massively lower than 10%, and depends on n.

I was trying to give a basic intuition about normal distributions with a simple example, the distribution of one sample is a simpler example of a different normal distribution. Yes obviously the distribution of an estimate of X given lots of samples is not the same as the distribution of a single sample, I never claimed it was.

[spywaregorilla]

> You can figure out what the stddev was from half the confidence interval divided by the z-score for a 95% confidence interval, 1.96, and you get 1.02 and 1.30 for the two groups.

I'm not really interested in double checking your math, but you cannot derive the standard deviation of a sample mean confidence interval without considering the sample size. You seem to be making the same mistake again, confusing the Z score of a single value vs. the Z score of a sample mean. The standard deviation is of course going to be much larger. Why? Because you're actually looking at a difference of proportions where the values are either 1 or 0. The standard deviation is of course going to be much larger than 1%.

Ignoring that and assuming you meant to say standard error, where your math appears to work at a glance; in general, sure, overlapping confidence intervals don't mean that statistical tests of mean difference won't be significant. But... if you don't have that your effect size is probably pretty small. I would not put a lot of faith on these particular results as strong evidence of anything.

I would advocate for people to just look for overlapping curves.

> Yes obviously the distribution of an estimate of X given lots of samples is not the same as the distribution of a single sample, I never claimed it was.

Not number of samples. The sample size.