In the comments, the authors give the sample size as about 20 in each group. Here's a more sensible graph (error bars to create a 95% CI on the estimated group means):
Obviously, there are some statistical power issues. It's unfortunate that they didn't get someone to help them with a power analysis up front---given how many very tech savvy people are interested in Kahn Academy and want to see it succeed, I think they could have gotten this sort of assistance pretty easily.
Thanks for doing that, I found myself wishing they had put error bars on when I saw that graph but was too lazy to do it myself.
With respect to the statistical power issue, it may be they just didn't have enough students. Unlike your typical clinical trial or scientific study where you're recruiting people, I got the impression this was from an existing pool of students for a specific school. Once you go beyond this, the design gets tricky since there are practical issues to how big a class can be. Then you need more teachers, and you need to normalize for per-teacher variability. Not saying a study like that wouldn't be great, but it would be complex to get right.
I try to give educationalists the benefit of the doubt, because they probably mean well, but at the same time, this stuff is important so I want them to be held to a higher standard.
The only "result" that the article could claim is the unfortunate There was no significant difference between the two teaching methods.
However, we could probably regain hope for Khan Academy by pointing out that the (five-week) summer school was almost entirely worthless for 2/3rds of the students anyway:
In terms of distribution, in each class approximately one third of the students saw some significant gains (ten percent or higher gains in percentage of questions answered correctly), whereas two thirds of the students’ scores were essentially flat (less than four percent increase or decrease).
If anything, it is this that should make us eager to explore alternative methods of teaching. I am hopeful that Khan Academy or a similar online learning provider will revolutionise education, and this article is the first step along the way to discovering how to use it.
I really hate it when people (mathematicians?) misuse the percentage notation...
> increased their percentage of correct answers by 5.2% over the five-week period
> ...
> For example, a student who started the summer knowing 60% of the correct answers in the traditional class ended the five weeks knowing 65.2% of the correct answers.
This means that the percentage of correct answers increased 5.2 percentage points, which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 = 65.2%).
I thought the Khan Academy was about 'leave no learner behind'. That is, if you miss a concept in traditional school, you flail around the rest of the term. But in Khan Academy, you can review lessons, stop and start them, ask limitless questions in the middle because they're all taped.
SO, I would expect to see extensive data on the distribution of the scores - how long is the lower tail of the curve? Did they pick up those that would have lagged behind?
As is, they give a single percentage - improvement on an algebra test. So no meaningful data at all. Hm.
In my experience, I see Khan academy being most valuable in a blended setting when the teacher isn't very good (because Salman explains things so well), but also over a longer period of time. Khan academy encourages retaining old material in at least two important ways: it makes the old material easily accessible, and Salman's teaching method strongly encourages you to have a better understanding of the concepts behind the math (for me, that's been in proving trig identities instead of memorizing and explaining how the process of multiplying matrices is arbitrary, but it's clearly a fundamental aspect of how Salman teaches).
[+] [-] john_horton|14 years ago|reply
https://skitch.com/johnjosephhorton/fih62/quartz-2
Obviously, there are some statistical power issues. It's unfortunate that they didn't get someone to help them with a power analysis up front---given how many very tech savvy people are interested in Kahn Academy and want to see it succeed, I think they could have gotten this sort of assistance pretty easily.
[+] [-] JunkDNA|14 years ago|reply
With respect to the statistical power issue, it may be they just didn't have enough students. Unlike your typical clinical trial or scientific study where you're recruiting people, I got the impression this was from an existing pool of students for a specific school. Once you go beyond this, the design gets tricky since there are practical issues to how big a class can be. Then you need more teachers, and you need to normalize for per-teacher variability. Not saying a study like that wouldn't be great, but it would be complex to get right.
[+] [-] ZeroGravitas|14 years ago|reply
I can't help but feel that this image they used sums up the whole project. Two (basically meaningless) numbers, presented in a 3-D bar chart. http://blendmylearning.files.wordpress.com/2011/08/resultsgr...
[+] [-] Jabbles|14 years ago|reply
However, we could probably regain hope for Khan Academy by pointing out that the (five-week) summer school was almost entirely worthless for 2/3rds of the students anyway:
In terms of distribution, in each class approximately one third of the students saw some significant gains (ten percent or higher gains in percentage of questions answered correctly), whereas two thirds of the students’ scores were essentially flat (less than four percent increase or decrease).
If anything, it is this that should make us eager to explore alternative methods of teaching. I am hopeful that Khan Academy or a similar online learning provider will revolutionise education, and this article is the first step along the way to discovering how to use it.
[+] [-] tomp|14 years ago|reply
> increased their percentage of correct answers by 5.2% over the five-week period > ... > For example, a student who started the summer knowing 60% of the correct answers in the traditional class ended the five weeks knowing 65.2% of the correct answers.
This means that the percentage of correct answers increased 5.2 percentage points, which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 = 65.2%).
[+] [-] irahul|14 years ago|reply
Yes, it is 8.67%.
Illustrating the calculation is better served by:
Initial: 60
After: 65.2
Increase: 5.2
% Increase: 5.2 / 60 * 100 = 8.76
[+] [-] JoeAltmaier|14 years ago|reply
SO, I would expect to see extensive data on the distribution of the scores - how long is the lower tail of the curve? Did they pick up those that would have lagged behind?
As is, they give a single percentage - improvement on an algebra test. So no meaningful data at all. Hm.
[+] [-] MaxGabriel|14 years ago|reply
[+] [-] tzs|14 years ago|reply