Having recently (last year) enrolled in a Computer Programming program, I can say this is likely the case. The class has a number of people who are simply enrolled to get a piece of paper that states they know how to code and then there are those that are breaking new ground in learning.
I have the logic down quite well but my biggest issue is the syntax - it would appear that Stanford thinks I'm doomed to fail.
It seems to me that there are way too many variables that aren't really measurable for this to be completely accurate -- such as this example, some people probably had programmed before.
From the actual article (and not the news article): "The results show that students’ change in programming patterns is only weakly predictive of course performance"
Not as "juicy" as the website - but then again, it's the actual results :)
Hey! I worked on the preliminary efforts of this project in 2012, and was also a grader for the course. (I'm "Gibbons, A." in the citation list)
Many people are making great points - there's a wealth of uncontrolled variables (ie some students already know how to program), but it was a fascinating dataset and I could even see the paradigms play out over the quarter during discussions with my students, both experienced programmers and students getting their first exposure to programming.
I would offer as a larger takeaway that work like this has the potential to vastly increase our understanding of learning and development with the rise of MOOCs, ipads-in-every-classroom, and of course IDEs that save every incremental compilation :-)
Oh and as another mention - working with Marcelo and Paulo was awesome, both are super passionate, intelligent dudes.
Fantastic! Good steps towards "artificial wisdom". Define wisdom-- skill in a craft (here: knowing how to think). Using ML to identify "wise" programming mindsets. Then (presumably) developing educational programs which teach these.
“Educational data mining should not be used to reinforce the existing ineffective forms of assessment, but to reimagine it completely,” Blikstein said. “Pre- and post-assessment is a black-box. We do not know much about what is happening in between.
“Ultimately, this work may help us better understand some forms of human cognition because you can see what people are doing and how they are thinking in real time.”
>> the authors did not find any correlation between the amount of “tinkering” — which is often thought to signify lack of programming expertise — and course performance. However, the authors found that students who changed their programming style — going from being a “planner” to a “tinkerer,” or vice versa — were the ones who performed best, suggesting that behavior change (rather than learning one single behavior) was determinant for course performance.
Not being flippant: The conclusion here seems to be evidence of an ability to learn something new, regardless of what, is strongly indicative of performance.
> "The discovery of these “sink states,” and how students
> got into them, offers opportunities beyond predicting
> grades: They open the door for developing systems to
> encourage students to go down more fruitful paths before
> they become lost in the programming weeds."
In the real world, my most valuable learning experiences have been those dead ends. Because I don't repeat those mistakes - better to make them and learn from them than to be steered away from them before they happen.
Relatedly, I have seen plenty of very smart people that were able to bowl over these "sink states" completely unaware of just how unmaintainable their ultimate solution was.
>> Alpha students, explained co-author Piech, moved efficiently from one point to another as they wrote code, creating a streamlined program. Beta and gamma students, on the other hand, wrote themselves into so-called “sink states” in which their programs slammed into dead ends from which they had to back out before they could resume progress.
Very interesting, and intuitively right to me. The difference is likely in the ability to create and maintain a consistent mental model of the program's structure and behavior during the implementation.
I think data mining editor/IDE use is a pretty useful idea overall and the "paradigm shift" of constant monitoring which allows immediate teacher feedback is much needed (not just for programming).
The other interesting approach I read about a while back was using tracing quiz data to identify "blind spots". A typical example was identifying students that had problems understanding loop constructs (they would always assume one iteration for example).
My gut instinct and thinking back to my university days make me think that a blind spot for recursion may very well be a thing. I'd be pretty interested in identifying students that struggle with loops and/or recursion and investigating that further. For example...is there a correlation? If not what happens if you give a loop-struggler group only recursive tasks and a recursion-struggler group only loop tasks? Etc.
It's not necessarily surprising that they might be able to detect students who will likely perform poorly in the course overall by monitoring their behavior early on. But the real challenge is not in determining which students are struggling, but rather in when and how you should intervene to improve these students' chances of success.
Begs the question: "Is there a 'right' way to think about programming problems?"
My knee-jerk reaction would be "No! Everyone learns and thinks differently"-- but being able to put programming styles into buckets and correlate final grades from it suggests otherwise, doesn't it?
It might suggest something, but it's just a hint. In order to take that approach as conclusive much more comprehensive studies in to highly diversified environments should take place.
All that leaving aside that it's a social science we're talking so... It's not exactly science as in scientific method. It's just models that (under X circumstances) might or might not apply... Go figure ;-)
I think the emphasis is more about the way people think about solving programming problems than the way they think about the problem itself. An analogy is that two people may differ in their mental models at the level of syntactic sugar rather than Turing completeness.
Completely anecdotal, but I vividly remember my own experience learning to program.
I learned in a vacuum, having nobody around with any software experience. I was given a PC and then saved up for an Amiga 500 (and later a $500 Lattice C compiler). My only link to software was through the bookstore and magazines.
Being a nerd, my main motivation at the time (of course) was to make a D&D game. I remember my first code was in Basic (no subroutines!) so I eventually learned to create subroutines based on line numbers (100-900 was main, 1000 was map, 1100 was character A, etc)
I managed to make about 50% of a single game before I ran out of memory. So I then set out to shrink (optimize) what I had made. After reading an article on dungeon map creation, I figured out that large arrays could replace the hand-coded drawing I had written. Check.
Ok, so now I start doing everything in arrays and pretty soon my code is smaller but "fixing stuff" (maintaining code) was stupidly hard. Oh okay, so I learned then some that data and code should be together rather than global (encapsulation).
I then needed to create random maps. After struggling with that, I read a paper on fractals, and implemented my first algorithm.
You can probably guess by now where I'm going with all this... I had crudely recreated many things by knocking up against hardware and brain limitations and figuring out how to work around them.
I progressed into Pascal, C, C++, assembly and eventually created my own assembler and graphical operating system at 16 years old. It was not a thing of beauty, but it did work. (Time was free, and compilers weren't cheap.)
I think that most people learn programming by failing, hitting traps, finding limits, crashing, burning and trying again. The only real identifier of outcomes, in the long term, is the willingness to continue trying and learning from mistakes.
So from my point of view, the outcomes of the paper are arbitrary in the long term. I would definitely had been a "Gamma" programmer. I hit every problem, and got terribly stuck. It took years for me to progress, learn good practices and become a proficient developer.
What was interesting is that the higher-performing group (Alpha) was stuck less often that the lower-performing group (Gamma). Is this the next programming motto after "Don't Repeat Yourself"?
Don't Get Stuck!
Now how does one avoid getting stuck while programming? I still get stuck a lot, but these really helped:
- Test what you write as quickly as you can.
- Hold it in your head! Or you won't have a clue what all your code, together, is doing.
- To get started, ask yourself, what is the simplest thing that could possibly work?
> Hold it in your head! Or you won't have a clue what all your code, together, is doing.
Do you mean to say: if you can't hold it all in your head, you're overcomplicating things? Because if so, I agree.
I just finished giving an introduction to programming course, using Processing. I tried teaching my students to write structured code from the start, splitting functionality into smaller methods as much as sensible. That way they never have to juggle too much functionality in their head at once.
On the first day I had them call built-in methods. Then I introduced the concept of variables and types, then I introduced defining your own methods, and only after that did I move on to booleans, if/else, for-loops and arrays.
It seems to have worked well for getting them to structure their code from the start, using methods more or less the same way you would use chapters, headers, sub-headers and paragraphs to structure a paper.
I know this isn't the best way to program, obviously, but this analogy is (relatively) easy for them to understand and loads better than the ball of muddy spaghetti you often see with people who just start out. It's also probably fairly easy to make the jump to other more advanced forms of code organisation.
I also completely agree with your other two points by the way, I kept repeating similar statements throughout the course:
- By splitting functionality into small methods, it's easy to test if something does what it is supposed to do (and I could easily correct them: "Hey, this method drawBall() is also doing hit detection. That's not what the method is supposed to do, restructure the code!").
- "The computer only does what it is literally instructed to do. What do you literally want to do? What are you literally instructing the computer right now?"
Overcoming getting stuck is the difference between a highly productive dev and the average dev. This is especially the case if the problem being worked on is hard. I am a very average dev in regards coding skills, but due to my background I have learned strategies to “unstick” myself after ending up in the middle of a massive mud pit. This has allowed me to solve problems better devs have not been able to accomplish.
Until you run into that one problem where you can't test between the small steps, because you need the whole thing to be up (to some degree) and working for any test to work.
Operating system kernels would be an example of that: The best test for a *nix OS kernel is, if it can run a shell. You need all the essential syscalls to do something sensible and if any of the required parts doesn't work the whole thing fails.
Another example would be refactoring a complex library into something more manageable. If you keep working in small, testable steps you can move only along the gradient, bordered by "can execute" and "doesn't execute". So you'll be able to reach only a local extremum. Now if you're in the fog and don't know where to go, that's fine. And for most development this is exactly how it happens. But sometimes you can see that summit on the other side of that rift and you know you have to take a leap to get over there.
I spent the past 3 weeks doing exactly that, refactoring a code base. I knew exactly where I wanted to go, but eventually it meant working for about a week on code without being able to compile, not even think of testing it, because everything was moving around and getting reorganized. However now I'm enjoying the fruits of that week; a much cleaner codebase, easier to work with and I even managed to eliminate some Voodoo code nobody knew why it was there, except that it made things work and things broke if you touched it.
> - Hold it in your head! Or you won't have a clue what all your code, together, is doing.
Or, sometimes it's important to get it out of your head, take a week or two off and look at it again with a fresh mind and from a different angle. Often problems seem only hard because you're approaching them from that one angle and you're so stuck with wanting to get it done, that you don't see the better way.
Instead you should write code in a way that it's easy to get back into it.
> - To get started, ask yourself, what is the simplest thing that could possibly work?
And then wonder: What would it take to make this simple thing break. Make things as simple as necessary but not simpler.
It's also likely that these guys had already programmed before coming to the class. We all tend to get stuck when programming something for the first time. Second and third times are much smoother experiences.
I'd offer two bits of marketable wisdom from this.
- think back. Students who change style do better. This implies that those students are reflecting on their own style and trying out techniques (performing experiments) to find out what works best for them.
- think ahead. Use lookahead to avoid backtracking. Some amount of top-down planning will make your life easier and avoid traps, even if your natural style is bottom-up.
My interpretation of that part of the article was "refactor early and reimplement often"
That grabs in the concept of switching between planning and experimenting modes with the end result of not getting stuck.
I may be biased in having learned this stuff a long time ago so when I get stuck its from getting hopelessly backed into a corner at which point the best solution is pretty much start over differently, and first semester programmers might get trapped a little more easily, like they don't even know common ancient anti-patterns so they don't even know where to start looking when debugging.
Reading between the lines it almost sounded like the A level programmers wrote their tests first (which is usually easy) then wrote code to pass the tests (usually not too awful) then they were done, whereas the lower grade level programmers wrote the code first, then tried to debug (and debugging is always harder than test writing or coding)
Yeah I really reckon that "what is the simplest thing that could possibly work" is the vital question. If you stray from that, getting stuck wastes what could otherwise be productive effort. I do have to wonder tho -- why is programming so complicated? Silicon and neurons so fundamentally different that we're always going to be fighting ourselves -- or one day we're going to have interfaces make programming way more natural?
> - To get started, ask yourself, what is the simplest thing that could possibly work?
I'm not sure I would go for what appear to be the simplest now ; I try to find the implementation which will have the simplest design to use/modify/delete in the future.
[+] [-] ynniv|11 years ago|reply
[+] [-] kifler|11 years ago|reply
I have the logic down quite well but my biggest issue is the syntax - it would appear that Stanford thinks I'm doomed to fail.
[+] [-] FinnDS|11 years ago|reply
[+] [-] willvarfar|11 years ago|reply
I'm curious about that too; is this course optional? Would people who can already program take this course?
[+] [-] inglor|11 years ago|reply
Not as "juicy" as the website - but then again, it's the actual results :)
[+] [-] AustinBGibbons|11 years ago|reply
Many people are making great points - there's a wealth of uncontrolled variables (ie some students already know how to program), but it was a fascinating dataset and I could even see the paradigms play out over the quarter during discussions with my students, both experienced programmers and students getting their first exposure to programming.
I would offer as a larger takeaway that work like this has the potential to vastly increase our understanding of learning and development with the rise of MOOCs, ipads-in-every-classroom, and of course IDEs that save every incremental compilation :-)
Oh and as another mention - working with Marcelo and Paulo was awesome, both are super passionate, intelligent dudes.
[+] [-] mellavora|11 years ago|reply
“Educational data mining should not be used to reinforce the existing ineffective forms of assessment, but to reimagine it completely,” Blikstein said. “Pre- and post-assessment is a black-box. We do not know much about what is happening in between.
“Ultimately, this work may help us better understand some forms of human cognition because you can see what people are doing and how they are thinking in real time.”
[+] [-] the_watcher|11 years ago|reply
Not being flippant: The conclusion here seems to be evidence of an ability to learn something new, regardless of what, is strongly indicative of performance.
[+] [-] GrinningFool|11 years ago|reply
[+] [-] taeric|11 years ago|reply
[+] [-] markbnj|11 years ago|reply
Very interesting, and intuitively right to me. The difference is likely in the ability to create and maintain a consistent mental model of the program's structure and behavior during the implementation.
[+] [-] kriro|11 years ago|reply
The other interesting approach I read about a while back was using tracing quiz data to identify "blind spots". A typical example was identifying students that had problems understanding loop constructs (they would always assume one iteration for example).
I talked to one of the authors of this paper and he had some interesting ideas fro CS education in general: http://dl.acm.org/citation.cfm?id=2526978
My gut instinct and thinking back to my university days make me think that a blind spot for recursion may very well be a thing. I'd be pretty interested in identifying students that struggle with loops and/or recursion and investigating that further. For example...is there a correlation? If not what happens if you give a loop-struggler group only recursive tasks and a recursion-struggler group only loop tasks? Etc.
[+] [-] skywhopper|11 years ago|reply
[+] [-] nerdy|11 years ago|reply
My knee-jerk reaction would be "No! Everyone learns and thinks differently"-- but being able to put programming styles into buckets and correlate final grades from it suggests otherwise, doesn't it?
[+] [-] atmosx|11 years ago|reply
All that leaving aside that it's a social science we're talking so... It's not exactly science as in scientific method. It's just models that (under X circumstances) might or might not apply... Go figure ;-)
[+] [-] brudgers|11 years ago|reply
[+] [-] convivialdingo|11 years ago|reply
I learned in a vacuum, having nobody around with any software experience. I was given a PC and then saved up for an Amiga 500 (and later a $500 Lattice C compiler). My only link to software was through the bookstore and magazines.
Being a nerd, my main motivation at the time (of course) was to make a D&D game. I remember my first code was in Basic (no subroutines!) so I eventually learned to create subroutines based on line numbers (100-900 was main, 1000 was map, 1100 was character A, etc)
I managed to make about 50% of a single game before I ran out of memory. So I then set out to shrink (optimize) what I had made. After reading an article on dungeon map creation, I figured out that large arrays could replace the hand-coded drawing I had written. Check.
Ok, so now I start doing everything in arrays and pretty soon my code is smaller but "fixing stuff" (maintaining code) was stupidly hard. Oh okay, so I learned then some that data and code should be together rather than global (encapsulation).
I then needed to create random maps. After struggling with that, I read a paper on fractals, and implemented my first algorithm.
You can probably guess by now where I'm going with all this... I had crudely recreated many things by knocking up against hardware and brain limitations and figuring out how to work around them.
I progressed into Pascal, C, C++, assembly and eventually created my own assembler and graphical operating system at 16 years old. It was not a thing of beauty, but it did work. (Time was free, and compilers weren't cheap.)
I think that most people learn programming by failing, hitting traps, finding limits, crashing, burning and trying again. The only real identifier of outcomes, in the long term, is the willingness to continue trying and learning from mistakes.
So from my point of view, the outcomes of the paper are arbitrary in the long term. I would definitely had been a "Gamma" programmer. I hit every problem, and got terribly stuck. It took years for me to progress, learn good practices and become a proficient developer.
[+] [-] partition|11 years ago|reply
Don't Get Stuck!
Now how does one avoid getting stuck while programming? I still get stuck a lot, but these really helped:
- Test what you write as quickly as you can.
- Hold it in your head! Or you won't have a clue what all your code, together, is doing.
- To get started, ask yourself, what is the simplest thing that could possibly work?
[+] [-] vanderZwan|11 years ago|reply
Do you mean to say: if you can't hold it all in your head, you're overcomplicating things? Because if so, I agree.
I just finished giving an introduction to programming course, using Processing. I tried teaching my students to write structured code from the start, splitting functionality into smaller methods as much as sensible. That way they never have to juggle too much functionality in their head at once.
On the first day I had them call built-in methods. Then I introduced the concept of variables and types, then I introduced defining your own methods, and only after that did I move on to booleans, if/else, for-loops and arrays.
It seems to have worked well for getting them to structure their code from the start, using methods more or less the same way you would use chapters, headers, sub-headers and paragraphs to structure a paper.
I know this isn't the best way to program, obviously, but this analogy is (relatively) easy for them to understand and loads better than the ball of muddy spaghetti you often see with people who just start out. It's also probably fairly easy to make the jump to other more advanced forms of code organisation.
I also completely agree with your other two points by the way, I kept repeating similar statements throughout the course:
- By splitting functionality into small methods, it's easy to test if something does what it is supposed to do (and I could easily correct them: "Hey, this method drawBall() is also doing hit detection. That's not what the method is supposed to do, restructure the code!").
- "The computer only does what it is literally instructed to do. What do you literally want to do? What are you literally instructing the computer right now?"
[+] [-] VMG|11 years ago|reply
Which may just be correlated with high performance, not necessarily the cause.
If you make "don't get stuck" a motto, you can easily imagine people skipping over difficult problems and not thinking about a good solution.
[+] [-] danieltillett|11 years ago|reply
[+] [-] datenwolf|11 years ago|reply
Until you run into that one problem where you can't test between the small steps, because you need the whole thing to be up (to some degree) and working for any test to work.
Operating system kernels would be an example of that: The best test for a *nix OS kernel is, if it can run a shell. You need all the essential syscalls to do something sensible and if any of the required parts doesn't work the whole thing fails.
Another example would be refactoring a complex library into something more manageable. If you keep working in small, testable steps you can move only along the gradient, bordered by "can execute" and "doesn't execute". So you'll be able to reach only a local extremum. Now if you're in the fog and don't know where to go, that's fine. And for most development this is exactly how it happens. But sometimes you can see that summit on the other side of that rift and you know you have to take a leap to get over there.
I spent the past 3 weeks doing exactly that, refactoring a code base. I knew exactly where I wanted to go, but eventually it meant working for about a week on code without being able to compile, not even think of testing it, because everything was moving around and getting reorganized. However now I'm enjoying the fruits of that week; a much cleaner codebase, easier to work with and I even managed to eliminate some Voodoo code nobody knew why it was there, except that it made things work and things broke if you touched it.
> - Hold it in your head! Or you won't have a clue what all your code, together, is doing.
Or, sometimes it's important to get it out of your head, take a week or two off and look at it again with a fresh mind and from a different angle. Often problems seem only hard because you're approaching them from that one angle and you're so stuck with wanting to get it done, that you don't see the better way.
Instead you should write code in a way that it's easy to get back into it.
> - To get started, ask yourself, what is the simplest thing that could possibly work?
And then wonder: What would it take to make this simple thing break. Make things as simple as necessary but not simpler.
[+] [-] random_coder|11 years ago|reply
[+] [-] pjc50|11 years ago|reply
- think back. Students who change style do better. This implies that those students are reflecting on their own style and trying out techniques (performing experiments) to find out what works best for them.
- think ahead. Use lookahead to avoid backtracking. Some amount of top-down planning will make your life easier and avoid traps, even if your natural style is bottom-up.
[+] [-] VLM|11 years ago|reply
That grabs in the concept of switching between planning and experimenting modes with the end result of not getting stuck.
I may be biased in having learned this stuff a long time ago so when I get stuck its from getting hopelessly backed into a corner at which point the best solution is pretty much start over differently, and first semester programmers might get trapped a little more easily, like they don't even know common ancient anti-patterns so they don't even know where to start looking when debugging.
Reading between the lines it almost sounded like the A level programmers wrote their tests first (which is usually easy) then wrote code to pass the tests (usually not too awful) then they were done, whereas the lower grade level programmers wrote the code first, then tried to debug (and debugging is always harder than test writing or coding)
[+] [-] diminoten|11 years ago|reply
* Test early, test often test automatically [0]
* Remember the big picture [0]
* The minimum viable product [1]
[+] [-] humanarity|11 years ago|reply
[+] [-] BenderV|11 years ago|reply
I'm not sure I would go for what appear to be the simplest now ; I try to find the implementation which will have the simplest design to use/modify/delete in the future.
[+] [-] 20edward|11 years ago|reply
[deleted]