I've been using SRS (anki, ankiweb, ankidroid) for almost four years now. I'm up to 4000 Japanese sentence cards and my amazement is constant. When I started I thought I was fooling myself, there was no way I'd be able to read real quantities of Japanese. Instead I can now read the manga for the anime I watched as a kid and enjoy the stories I thought had ended.
I've gotten into a rhythm, four new sentences per day which at my memorization rate works out to ~60 cards to review a day. Thus 20-30 minutes per day of review. It has become such a part of my day that skipping a day makes the day feel like a massive failure.
Its interesting how 4 cards can translate into 60 even with high memorization rates: for every card you add you'll see it again in 2 days, ~4 days, ~8 days, ~2 weeks, ~1 month, ~2 months, ~4 months, ~8 months, ~1 year... A review session is thus made up of multiple standing waves. The effect is noticeable you'll see a workload increase as the next standing wave of review starts. Right now some reviewed cards are getting scheduled next in ~8 years!
I have been using SRS to learn Mandarin Chinese for the last 2 years now and I can also verify it is extremely effective when used correctly, and when understanding its' advantages and limitations.
I think sentences are also a much better way to learn than learning single vocabulary words, because they show the usage of the word in context. This is very important for speaking naturally and idiomatically.
Once you get beyond a beginner level, choosing a direction of study can be a little difficult. My technique is to find sentences or phrases from conversations I have with people, books/articles I read, and from movies/television shows - and then input it into the SRS. Preferably, each sentence has 1 new novel phrase or vocabulary word. Although this can be time consuming it gives a huge benefit. Let me give an example:
So, in this case the second last two characters made up the word that I was unfamiliar with, so I put that on the answer side. The huge benefit is that I found this sentence in a book I am reading - so every time it comes up in the SRS, I not only recall the sentence, I recall the context of the story (in this case a dialogue between two characters), and this has a double reinforcing effect. In the same way, that if a TV or movie character says a sentence over and over again in a video, it will cause you to remember the scene of the sentence, and at which part of the story the movie was in at the time.
The contrast of this method would be to find "pre-made" decks of cards for sentences in the language you are learning, but I find that it is harder to remember the items when the sentences are not as meaningful to you.
A key point for using SRS effectively is to use to review material already learned, not to learn new never before seen material. The article itself even mentions this.
I've previously had good experiences with ProVoc (and now iVocabulary, a fork of ProVoc) but stopped using it and instead use Duolingo now because I lacked a good flashcard resource. I didn't have the time to think up new flashcards and learn them on a regular basis. Where do you get your sentences from for Anki? Is there a good repository online that you'd recommend (preferably with some kind of community review)?
I'm interested that you chose sentences rather than words. That seems super smart now that I think about it because you can pick up idioms and word relationships as well, and it's probably a little less boring too.
I think spaced repetition software, which typically involves flashcards, is great for memorising a large quantity of facts. E.g. for learning foreign language vocabulary, anatomy, etc... For this kind of learning it can really save you time, especially when you have 100s or 1000s of items to remember.
Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful. For these subjects the number of raw facts required is usually smaller, and the thought processes, concepts and problem solving ability are much more important.
However, spaced repetition could be beneficial to these more creative and analytical subjects if it was applied more dynamically. In this case the units to learn could be the rules, laws, concepts which can manifest themselves in different ways within different problems, instead of being static flashcards. Does anyone know of software which works in this way?
My interest in this stems from http://readlang.com, a site I'm working on at the moment for learning foreign languages. I use SRS to keep track of words, but in future it could be nice to think about applying SRS to grammatical concepts too.
Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful. For these subjects the number of raw facts required is usually smaller, and the thought processes, concepts and problem solving ability are much more important.
My programming flash card usually consists of evaluating pieces of code like:
Question -->
Evaluate the following:
puts "Hello world"
Answer -->
Hello world
I cannot memorize the solution as if I memorize the answer to multiplication problems, which is just straight rote memorization. I memorize the process for code evaluation instead. In any case, I do not think anki and similar spaced repetition flash card software are in any way a replacement for actually just writing code and solving lot of different problems. They are always complementary to learning the materials in other ways.
Hey steveridout, you've got a point there. It's a weakness I've ran into many times when I move away from languages into learning something like statistics or hard algorithms. It's possible, by simplifying the information and using a combination of text and images. The problem when you start doing this is the time it takes to create a deck of cards that make sense. It would be a great idea to be able to use a templating language for your flashcards which you can embed simple rules and plug in a source of data. SRS programs have cloze deletion but it will go much further than that. It may be a good idea for an Anki plugin, perhaps. I hadn't thought about it until just now. ......
I'm currently using SRS software (Anki) to learn more about software development. Instead of trying to memorize facts I am using cards as a cue that it is time to revise and deepen my understanding of a particular topic.
Every time I learn about a new topic/concept/technique I will add a card to my anki deck with the topic name. E.g. Today I have added JS Promises.
Then when a topic comes up for review I spend 10-15 minutes reviewing my knowledge and trying to probe my mind for parts I do not understand and trying to link it to, and see how it can be used with, other related techniques.
After reviewing I then spend 30 minutes to 1 hour investigating the technique in more depth. At this point I'll try and implement an aspect of it in code or see if I can answer a few problems people are having with the topic on stack overflow.
Then I will add a few notes to the card detailing what I have done so that I can make my next "review" of the card more challenging.
All up each review for a card takes close to 1.5 hours. I tend to have 1-2 cards come up per day to review.
I find it has given me a broad, and usable, understanding of software engineering practice and principles and has made me a better developer. The downside is I feel you will never become a master at these topics using this technique as the time investment per topic is just not enough.
I started using it because I found I had an experts understanding of some topics and a very weak understanding of other fundamental areas of computer science and software engineering. E.g. 1000+ hours of crafting complex regular expressions but lacking a proper understanding of the relational model and how a database table join works.
I would be really interested to see if there is any literature that examines using SRS in this manner (as opposed to learning simple facts).
I use it for math at the undergraduate to graduate level and have for over a year, but not super religiously. There are plenty of things in math that benefit from this treatment. It's good to just know definitions, then when you come across something that satisfies it you'll know what to look for to find out more about it. You'll know the exact requirements of theorems so that you don't have to once again look on wikipedia to see that yes this does require a compact set. You'll find a card that says something you've reviewed before but go "wait, why is that true?" and then work out the general idea again. You can memorize known identities that are just plain useful, things that can't be easily Googled but can be easily recognized. You can put warnings to yourself that certain theorems can be easily applied incorrectly. You can learn the most common names for theorems so that it is easier to communicate with others about them.
And above all, it's just another chance to look over things you already should know. I won't claim that it's more or less useful than reading a textbook for the equivalent amount of time or doing problems, but it's certainly more useful than doing nothing.
P.S. Anki has very nice LaTeX support! I don't know about the others, but this is invaluable for math.
For mathematics and physics, the 'unit' of spaced repetition should be skills, not questions. You would need a program which can generate questions that test a particular skill, such as partial differentiation or factorizing quadratics. If you get a question wrong, that indicates that you need to spend more time practicing that type of question.
Even for learning deeply creative skills, having a good scaffold of the vocabulary, concepts, and atomic pieces of knowledge is very important to construct high-level connections later on. Often, the relationship between fact-base memorization and high-level synthesis is a complex balance, but learning any skill or discipline is going to involve a fairly large amount of gruntwork in terms of just memorizing vocabulary and basic facts.
> Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful.
I thought you were wrong for a second, since there are a lot of definitions and theorems that are probably good to know on the fly. But then I thought about learning about the meaning of greek letters; okay, but in what context...
Spaced repetition is great for material which is divided into very small chunks and where you have a ton of time for comparatively little information (Vocabulary, especially in languages you are not learning for a degree).
Problems arise in practical situations when you want to pass an exam. Or a couple of them in a certain period of time.
Proponents of SR will say that it takes at least 6 times the amount of time/effort over learning "short term". So if you are learning for an exam, you better have plenty of time to spare. And you better start early, like 6 months before the exam. All that makes it pretty useless in a lot of situations I've been in.
Short term learning isn't as bad as it looks. If you really "cram" for an exam, and you think you forgot all of it a couple of days or weeks later, that's not what happened. Instead, your brain is now primed for the material, and when you come into contact with it later, a lot of information is still there and re-learning will be easier.
Proponents of SR will say that it takes at least 6 times the amount of time/effort over learning "short term". So if you are learning for an exam, you better have plenty of time to spare. And you better start early, like 6 months before the exam. All that makes it pretty useless in a lot of situations I've been in.
This is a bit of a strawman. If you need to pass an exam quickly, you should and will "cram". Afterward, you can resume your spaced repetition learning.
Assuming you're paying to be educated about a certain thing, and you actually want the knowledge and not just the diploma, walking away merely being "primed" to learn it later isn't the best outcome. As an analogy, cramming is better than failing the exam as driving into the ditch is better than a collision. However, best to avoid either scenario in the first place.
This is a bug with exams (and if you're in a situation where you need to cram, your study habits), not with spaced repetition. The fact that it's harder to pass exams while using learning strategies that enable long term retention is one of the main reasons for the disparity between academic achievement and actual competence.
Gwern, how long does it take to put together a comprehensive document like this? Also, what motivates you the most in compiling the information? Is it a rabbit hole of fascinating discovery which you joyfully explore until your knowledge is satiated or does something else compel you?
> Gwern, how long does it take to put together a comprehensive document like this?
This particular page is one of the most citation-stuffed pages on my site, and I read everything I link if the fulltext is available (which is currently almost all of them after my last batch of requests)... I know I spent at least 40 hours compiling the core of the literature review for the LW contest, and I feel like that was less than half the time I spent writing the rest of it, updating it with new papers, that sort of thing, so I would guess this took somewhere upwards of 80 hours.
I always like sticking to the very best, so SuperMemo was the choice for me. I remember spending a couple of weeks just reading the whole website (http://www.supermemo.com/) because it contained so much interesting information on learning and memorization. It also took a while to understand how to use the program properly because it probably has one of the worst UIs one can imagine ;-).
Anyways, I ended up dumping SuperMemo because once my item database grew I became too lazy to spend 30 minutes (or more!) each day going through this drill. I might get back to it at some point, though.
Edit: By the way, SuperMemo offers a few more interesting tools.
One is "Incremental Reading", where you can import an article or text file into SM and SM will let you incrementally process it.
For example, it might pop up as part of your repetition routine, and you can read a few paragraphs (you read as much as you want), and pick out the important parts of what you've read (parts you want to remember). Once you're done reading, you continue and the whole text will come up again and again, until you finish reading it and then you can dismiss it.
The parts you cut out from the text become separate learning items and they too will come up in your daily routines, which let you further work on them until you ultimately reduce them into a Question & Answer form.
Another tool is sleeping monitoring. SM will let you track your sleep data, and it will tell you how your sleep affects your memory performance.
> Anyways, I ended up dumping SuperMemo because once my item database
> grew I became too lazy to spend 30 minutes (or more!) each
> day going through this drill.
Yeah, same happened to me. But I didn't catch on so soon, and I found myself spending 2 hours/day on it before I started to scratch my head.
When I started, it wasn't obvious, but clearly there is an upper limit on the rate of new entries that the system can support. Plus, you have to factor in time spent doing data entry. I think these factors combine to make for a life-time limit on the number of entries. Suppose you use SuperMemo for 20 years (7300 days) at 2 hours per day (14600 hours) and 100 new entries per day. That's only 1.4 million unique entries.. (this analysis is severely lacking and deserves more thought).
Insert here some humbling story about the benefits of improving meta-aspects of how and what you pick to spend time on. Obviously, inserting everything you ever come across into SuperMemo is a bad idea.
Anyway, then I realized this was all stupid because I remembered everything anyway. So now I'm off SuperMemo. Hooray memory!
I review all the due items in less than 30 minutes, always. Then I fill it up to 30 minutes with new materials, sometime going a few minutes beyond 30 minutes, because the materials sometime requires some time to learn.
Spaced repetition is not new, nor fancy, and has direct biological evidence. It works. That said having a good tool and motivation are required. I can recommend ankiweb.net / anki desktop for the first, but can't help with the second. The motivation to follow through on doing cards is what I have trouble with.
I managed to hacked myself to build a habit of 30 minutes a day of spaced repetition, building off my momentum to consistently do khanacademy an hour a day(A project which I finished a while ago) in conjunction with my time tracking tool. The key seems to be a bunch of small repeatable goal that allow for small wins. It works for me, though I don't know if it will work for anyone else. Plus, it's only anecdotal, so take it with a grain of salt.
(I wish there is a accessible book that rigorously about which habit building strategies work according to scientific research)
I largely credit spaced repetition with my improvement as a chess player from a rating of 1800 to 2000, something pretty unusual for a middle-aged adult. I wrote about about my experience at http://dfan.org/blog/2013/07/07/mnemosyne-part-3/.
This is one of the greatest pieces I've ever read.
The anecdote where Hamming went to Bode's office to talk about Turkey was like a lightning ! Is it possible to see so many great names in one paragraph !
I've always seen some analogies in things with half-life and the charge and discharge of capacitors.
I'll give an example: I've always thought about learning as a capacitor charging through a resistance, and thought about the RC constant. In my mind, it takes relatively a short period of time to get to 63.2%, but it takes a very long time to get from 63.2% (it will never reach that, since it's an exponential).
What this tells me about learning ? Pick a field and then decide how much do you want to master it. Then know that it'll take you longer to get from 63.2% to 100% (which you will never achieve). An example would be the fact that English is my fifth language, which I only started speaking in 2009. Do I want to be an English major and write in a Kipling way, or do I want to have a command good enough to get my message to you (and possibly learn other languages) ?
It depends, but I find it a fascinating subject.
The other funny thing is that I used a mix of cramming and spaced repetition to study: I used to skip college for a year, and then get a month and go hard-core. Five courses. I'd draw a pentagon with the five courses and study two courses a day. 1,2 - 3,4 - 5,1 - 2,3 - 4,5 - 1,2 ...
You'll notice that I go back to each module each other day. The results I got from that are just astounding, as I was capable to memorize entire pages, chapters and I would see them. I would know exactly where I've written my notes on which pages. I aced the final exam of physics and had the best grade with questions that were never asked before (it was theory, and no 'values' or 'calculations' were there). I was writing things I've never written before, but I knew my maths checked out because I just knew. These weren't exercises or something.. I've been accused of cheating many times. A part of me found it insulting, the other part found it flattering as you only accuse someone of cheating if what he did was an accomplishment.
I recently read this study of Halo players from Microsoft, and the authors found that spacing out your play made your gain rating faster per game played, BUT if you space it out too much, then you'll simply be outmatched by players who play a lot more. In other words, if you want to improve at Halo efficiently, space out your play more, but if you want to improve in sheer absolute skill, you will want to simply play more games.
I've been considering making spaced repetition software to help reteach and remind me of technical concepts I've learned over college that I don't recall too well. Stuff like fluid mechanics, microelectronics, thermodynamics, control systems, complex analysis, computer architecture, etc. Basically I have access to a lot of relevant course notes I took over the years (and digitized), but I don't remember it, so I would use SRS on this material to relearn it. I was thinking 20 mins a day would be a good amount of time for this.
But wait, let's take a step back. Does it really matter if I can recall all this engineering material? Supposedly my brain "knows" enough of it such that if there were a big project in a field requiring that knowledge, then I could relearn it all, because I've been trained to think in a way pertinent to that material. Is my time better spent just learning new topics and building things, which is what I currently do? Why dwell on having perfect recall? It only seems to benefit test takers.
I would love to hear counter-arguments for why SRS for engineering materials would be a good idea for those no longer in school, maybe then I'll be motivated to build some gamified spaced repetition software. It'll need to be fun and reward-based, imo.
I've been thinking about this for a while--I created a spaced repetition system to digitize and learn material in college (alas, I don't use it anymore). I have a philosophical approach to the question here (also cited in Gwern's article): http://blog.learnstream.org/2012/08/spaced-repetition-in-nat...
Your distinction between learning facts and learning ways of thinking is interesting. Most of the spaced repetition research and design is around fact recall--what's called "paired associates" because you recall some response paired with some stimulus. I don't have any evidence for it, but I'd say that the schema can be lost just as well as facts can be.
One thing I will say is that spaced repetition is generally known for the repetition--that you retain something over time, and the spacing is more thought of as convenience. But the truth is that spacing is also part of better learning--when recall is difficult it's learned better (called "desirable difficulties"). I thought of that because Robert Bjork does that research with skills as well as facts. (See http://www.wired.com/geekdad/2012/01/everything-about-learni... for an overview.)
Whether you should use SRS for any given domain is completely dependent on your specific circumstances. The answer could easily be different for you and a classmate of yours who fits the description above.
But I'd say that if there's some are you'd like to work in that requires specific knowledge that fits well within SRS, why would you not? As opposed to trying to learn and recall anything that might possibly be useful, ever.
I have been experimenting with spaced repetition algorithms for around two years now, and I've found that for me, an a simple exponent with base of 2.71828 (e) to be optimal for learning short question/answer pairs.
This means, every time I successfully recall the answer to a question, I increase the interval of when I will be challenged with it again by about 2.71828 times since I had seen it previously.
This is with a recall failure rate of about 5%, which I'm trying to maintain. (If it's nearing 0, then you're probably wasting time being challenged too frequently.)
This is on a data set used daily for 30 minutes, reviewing about 50 older questions and encountering around 3 new ones per day.
> At least with physical fitness there isn’t a precisely dismaying number indicating how far behind you are!
Many sports are precisely measurable and in such sports, the measurements determine who has won, to various resolutions. Shortest running time over a given distance, heaviest weight snatched, furthest distance jumped and so on.
Athletes in such sports can very accurately track their performance over time. As of yesterday I demonstrated that I can clean & jerk 25kg less than my lifetime record, for example.
I think the better analogy would be if you had some sort of workout schedule and you had missed n workouts. Just bad performance doesn't indicate much: if I score a 2 on all my flashcards today, that's bad, but I'm not 'behind' since the cards' next review will just be moved up as necessary to compensate.
FluxCards provides spaced repetition learning with easy to edit and create own cards. You can sneak the answer line by line, you can finger-paint and the stats are fast and meaningful.
It is free to use with 0.5, 1, 2, 4, 8, 16 days intervals. Beyond that you need the premium version but your cards are yours, so if you consider it not worth the fee, you can export your cards to SD and use them in Anki.
And yes, spaced repetition rocks. I can only encourage any flash cards programmer to implement it, as all these apps that implement flash cards but no spaced repetition learning are just not worth your time.
I use SRS (Mnemosyne) for memorising traditional Irish tunes. It feels like each tune is a bit too much of a large chunk to remember at one time for best use of an SRS (you're supposed to chop the information up into the smallest element possible), but the benefits of the system are amazing. Before I started using SRS I'd just practise the tunes I liked and already knew!, with the new tunes, and anything in the middle got forgotten about. Now mnemosyne forces me to practise tunes that I don't know, and the number of tunes I'm fluent in has increased dramatically.
I'm also building a language learning startup that uses SRS. If you're interested in trying the system out in beta, or being notified when it's ready for general consumption, email me at [email protected] and let me know which language you want to learn.
at kenHub[1], we built an online trainer for anatomy. It uses a game-like sequence of exercises designed to teach and test you at the same time about anatomy structures and their terminology. We've implemented a very basic spaced-repetition algorithm, but there's always more to improve.
One interesting thing we've found early on, was that when we encouraged people to actually "take a break", not to repeat too much, but instead let your brain process and come back to it later, people didn't like it. They wanted the ability to keep training and training. This resonates well with what the article describe about students' cramming tendency. So some time you have to manipulate even a solid algorithm, to fit with reality.
I combined spaced repetition with machine learning to attempt to predict memory recall for my thesis. Turned out fairly effective, correctly predicting recall up to 70% of the time.
Spaced repetition is pretty amazing. Adding records feels like you're entering them into your brain, provided you use it every day. I managed to go from knowing almost nothing to scoring 99% in a Japanese kanji exam once, entirely thanks to Anki.
SRS software (I mostly used Anki) was invaluable to me when I was learning Japanese, both for vocabulary and the kanji.
I always wanted to make my own SRS, one that had ease of use , stability and data integrity at its core rather than cutting edge features. This year I finally set aside time to make it: feel free to try it out and let me know what you think. It is very simple (there is only one deck), but multiple decks will be the next feature to be added.
iikanji looks good, but have you seen wanikani? The concept seems similar.
http://www.wanikani.com/
Some feedback: I know how to read all of the stuff I tried (the first 10 kanji), mostly thanks to wanikani. But I haven't studied much gramar, so I'm not always sure what the sentences mean. You should probably add translations for these in the back of the cards.
I'm not really sure why you would want that. A CLI spaced repetition means you make it much harder to display images and LaTeX, which are going to be used in a lot of cards.
It's great for learning facts and exploring a subject. Spaced repetition is fascinating. Especially in the realm of language learning. Vocabulary acquisition can be optimised by deciding the right order of words to maximise your ability to read a foreign language text. gwern has a link at the bottom of the article on modelling vocabulary lists as a travelling salesman problem. Feeding this into SRS Flashcard program is powerful. I guess one of the big problems ( if speed is your goal) is the creation of the cards themselves. The time to create the card is much greater than the time to learn the card and can be time a sink if you are trying to learn your first 3000 words of language.
I've been using Anki for learning Japanese. I've particularly appreciated the decks that include audio of complete sentences and have found that it has greatly improved my ability to understand spoken language.
One thing that I've wondered is whether it is a good idea to try to think up mnemonics for tricky words when using spaced repetition. It certainly seems effective in helping recall but I worry that I might train my brain to always rely on going via an indirect path to get to the information and that this will slow down my ability to understand real language.
From my personal experience, using such a method doesn't cause problems. I studied the kana using Heisig's Remembering the Kana, where it connects each character and pronunciation to a mnemonic story. I'm long past the point where I actually need the stories to read kana, the sounds are simply tied to the characters now, with the stories available for recall if i have a brain flub and can't remember one for a bit.
[+] [-] Danieru|12 years ago|reply
I've gotten into a rhythm, four new sentences per day which at my memorization rate works out to ~60 cards to review a day. Thus 20-30 minutes per day of review. It has become such a part of my day that skipping a day makes the day feel like a massive failure.
Its interesting how 4 cards can translate into 60 even with high memorization rates: for every card you add you'll see it again in 2 days, ~4 days, ~8 days, ~2 weeks, ~1 month, ~2 months, ~4 months, ~8 months, ~1 year... A review session is thus made up of multiple standing waves. The effect is noticeable you'll see a workload increase as the next standing wave of review starts. Right now some reviewed cards are getting scheduled next in ~8 years!
[+] [-] jliechti1|12 years ago|reply
I think sentences are also a much better way to learn than learning single vocabulary words, because they show the usage of the word in context. This is very important for speaking naturally and idiomatically.
Once you get beyond a beginner level, choosing a direction of study can be a little difficult. My technique is to find sentences or phrases from conversations I have with people, books/articles I read, and from movies/television shows - and then input it into the SRS. Preferably, each sentence has 1 new novel phrase or vocabulary word. Although this can be time consuming it gives a huge benefit. Let me give an example:
So, in this case the second last two characters made up the word that I was unfamiliar with, so I put that on the answer side. The huge benefit is that I found this sentence in a book I am reading - so every time it comes up in the SRS, I not only recall the sentence, I recall the context of the story (in this case a dialogue between two characters), and this has a double reinforcing effect. In the same way, that if a TV or movie character says a sentence over and over again in a video, it will cause you to remember the scene of the sentence, and at which part of the story the movie was in at the time.The contrast of this method would be to find "pre-made" decks of cards for sentences in the language you are learning, but I find that it is harder to remember the items when the sentences are not as meaningful to you.
A key point for using SRS effectively is to use to review material already learned, not to learn new never before seen material. The article itself even mentions this.
(Also, Mnemosyne is my SRS of choice http://mnemosyne-proj.org/, but Anki is a good choice too.)
[+] [-] Osmium|12 years ago|reply
[+] [-] jamesaguilar|12 years ago|reply
[+] [-] Sukotto|12 years ago|reply
I downloaded a nice kana set (with audio) but somehow they just didn't click. Anki eventually classed most of them as leeches and I stopped trying.
[+] [-] riggins|12 years ago|reply
[+] [-] steveridout|12 years ago|reply
Using flashcards to learn mathematics, computer programming, physics, or anything requiring more creative or analytical thought is not so useful. For these subjects the number of raw facts required is usually smaller, and the thought processes, concepts and problem solving ability are much more important.
However, spaced repetition could be beneficial to these more creative and analytical subjects if it was applied more dynamically. In this case the units to learn could be the rules, laws, concepts which can manifest themselves in different ways within different problems, instead of being static flashcards. Does anyone know of software which works in this way?
My interest in this stems from http://readlang.com, a site I'm working on at the moment for learning foreign languages. I use SRS to keep track of words, but in future it could be nice to think about applying SRS to grammatical concepts too.
[+] [-] kiba|12 years ago|reply
My programming flash card usually consists of evaluating pieces of code like:
Question -->
Evaluate the following:
puts "Hello world"
Answer --> Hello world
I cannot memorize the solution as if I memorize the answer to multiplication problems, which is just straight rote memorization. I memorize the process for code evaluation instead. In any case, I do not think anki and similar spaced repetition flash card software are in any way a replacement for actually just writing code and solving lot of different problems. They are always complementary to learning the materials in other ways.
[+] [-] thinkersilver|12 years ago|reply
[+] [-] sheepmullet|12 years ago|reply
Every time I learn about a new topic/concept/technique I will add a card to my anki deck with the topic name. E.g. Today I have added JS Promises.
Then when a topic comes up for review I spend 10-15 minutes reviewing my knowledge and trying to probe my mind for parts I do not understand and trying to link it to, and see how it can be used with, other related techniques.
After reviewing I then spend 30 minutes to 1 hour investigating the technique in more depth. At this point I'll try and implement an aspect of it in code or see if I can answer a few problems people are having with the topic on stack overflow.
Then I will add a few notes to the card detailing what I have done so that I can make my next "review" of the card more challenging.
All up each review for a card takes close to 1.5 hours. I tend to have 1-2 cards come up per day to review.
I find it has given me a broad, and usable, understanding of software engineering practice and principles and has made me a better developer. The downside is I feel you will never become a master at these topics using this technique as the time investment per topic is just not enough.
I started using it because I found I had an experts understanding of some topics and a very weak understanding of other fundamental areas of computer science and software engineering. E.g. 1000+ hours of crafting complex regular expressions but lacking a proper understanding of the relational model and how a database table join works.
I would be really interested to see if there is any literature that examines using SRS in this manner (as opposed to learning simple facts).
[+] [-] tgb|12 years ago|reply
And above all, it's just another chance to look over things you already should know. I won't claim that it's more or less useful than reading a textbook for the equivalent amount of time or doing problems, but it's certainly more useful than doing nothing.
P.S. Anki has very nice LaTeX support! I don't know about the others, but this is invaluable for math.
[+] [-] chad_oliver|12 years ago|reply
[+] [-] natural219|12 years ago|reply
[+] [-] Dewie|12 years ago|reply
I thought you were wrong for a second, since there are a lot of definitions and theorems that are probably good to know on the fly. But then I thought about learning about the meaning of greek letters; okay, but in what context...
[+] [-] bayesianhorse|12 years ago|reply
Problems arise in practical situations when you want to pass an exam. Or a couple of them in a certain period of time.
Proponents of SR will say that it takes at least 6 times the amount of time/effort over learning "short term". So if you are learning for an exam, you better have plenty of time to spare. And you better start early, like 6 months before the exam. All that makes it pretty useless in a lot of situations I've been in.
Short term learning isn't as bad as it looks. If you really "cram" for an exam, and you think you forgot all of it a couple of days or weeks later, that's not what happened. Instead, your brain is now primed for the material, and when you come into contact with it later, a lot of information is still there and re-learning will be easier.
[+] [-] kiba|12 years ago|reply
This is a bit of a strawman. If you need to pass an exam quickly, you should and will "cram". Afterward, you can resume your spaced repetition learning.
[+] [-] _greim_|12 years ago|reply
[+] [-] mistercow|12 years ago|reply
[+] [-] biot|12 years ago|reply
[+] [-] gwern|12 years ago|reply
This particular page is one of the most citation-stuffed pages on my site, and I read everything I link if the fulltext is available (which is currently almost all of them after my last batch of requests)... I know I spent at least 40 hours compiling the core of the literature review for the LW contest, and I feel like that was less than half the time I spent writing the rest of it, updating it with new papers, that sort of thing, so I would guess this took somewhere upwards of 80 hours.
[+] [-] steveklabnik|12 years ago|reply
[+] [-] mikevm|12 years ago|reply
Anyways, I ended up dumping SuperMemo because once my item database grew I became too lazy to spend 30 minutes (or more!) each day going through this drill. I might get back to it at some point, though.
Edit: By the way, SuperMemo offers a few more interesting tools.
One is "Incremental Reading", where you can import an article or text file into SM and SM will let you incrementally process it. For example, it might pop up as part of your repetition routine, and you can read a few paragraphs (you read as much as you want), and pick out the important parts of what you've read (parts you want to remember). Once you're done reading, you continue and the whole text will come up again and again, until you finish reading it and then you can dismiss it. The parts you cut out from the text become separate learning items and they too will come up in your daily routines, which let you further work on them until you ultimately reduce them into a Question & Answer form.
Another tool is sleeping monitoring. SM will let you track your sleep data, and it will tell you how your sleep affects your memory performance.
[+] [-] kanzure|12 years ago|reply
When I started, it wasn't obvious, but clearly there is an upper limit on the rate of new entries that the system can support. Plus, you have to factor in time spent doing data entry. I think these factors combine to make for a life-time limit on the number of entries. Suppose you use SuperMemo for 20 years (7300 days) at 2 hours per day (14600 hours) and 100 new entries per day. That's only 1.4 million unique entries.. (this analysis is severely lacking and deserves more thought).
Insert here some humbling story about the benefits of improving meta-aspects of how and what you pick to spend time on. Obviously, inserting everything you ever come across into SuperMemo is a bad idea.
Anyway, then I realized this was all stupid because I remembered everything anyway. So now I'm off SuperMemo. Hooray memory!
[+] [-] kiba|12 years ago|reply
I review all the due items in less than 30 minutes, always. Then I fill it up to 30 minutes with new materials, sometime going a few minutes beyond 30 minutes, because the materials sometime requires some time to learn.
[+] [-] iandanforth|12 years ago|reply
[+] [-] kiba|12 years ago|reply
(I wish there is a accessible book that rigorously about which habit building strategies work according to scientific research)
[+] [-] johtso|12 years ago|reply
https://www.beeminder.com/
https://github.com/muflax/beeminder-anki
[+] [-] dfan|12 years ago|reply
[+] [-] Jugurtha|12 years ago|reply
The anecdote where Hamming went to Bode's office to talk about Turkey was like a lightning ! Is it possible to see so many great names in one paragraph !
I've always seen some analogies in things with half-life and the charge and discharge of capacitors.
I'll give an example: I've always thought about learning as a capacitor charging through a resistance, and thought about the RC constant. In my mind, it takes relatively a short period of time to get to 63.2%, but it takes a very long time to get from 63.2% (it will never reach that, since it's an exponential).
What this tells me about learning ? Pick a field and then decide how much do you want to master it. Then know that it'll take you longer to get from 63.2% to 100% (which you will never achieve). An example would be the fact that English is my fifth language, which I only started speaking in 2009. Do I want to be an English major and write in a Kipling way, or do I want to have a command good enough to get my message to you (and possibly learn other languages) ?
It depends, but I find it a fascinating subject.
The other funny thing is that I used a mix of cramming and spaced repetition to study: I used to skip college for a year, and then get a month and go hard-core. Five courses. I'd draw a pentagon with the five courses and study two courses a day. 1,2 - 3,4 - 5,1 - 2,3 - 4,5 - 1,2 ...
You'll notice that I go back to each module each other day. The results I got from that are just astounding, as I was capable to memorize entire pages, chapters and I would see them. I would know exactly where I've written my notes on which pages. I aced the final exam of physics and had the best grade with questions that were never asked before (it was theory, and no 'values' or 'calculations' were there). I was writing things I've never written before, but I knew my maths checked out because I just knew. These weren't exercises or something.. I've been accused of cheating many times. A part of me found it insulting, the other part found it flattering as you only accuse someone of cheating if what he did was an accomplishment.
[+] [-] luckluckgoosed|12 years ago|reply
http://jeffhuang.com/Final_HaloLearning_CHI13.pdf
[+] [-] zxcvvcxz|12 years ago|reply
But wait, let's take a step back. Does it really matter if I can recall all this engineering material? Supposedly my brain "knows" enough of it such that if there were a big project in a field requiring that knowledge, then I could relearn it all, because I've been trained to think in a way pertinent to that material. Is my time better spent just learning new topics and building things, which is what I currently do? Why dwell on having perfect recall? It only seems to benefit test takers.
I would love to hear counter-arguments for why SRS for engineering materials would be a good idea for those no longer in school, maybe then I'll be motivated to build some gamified spaced repetition software. It'll need to be fun and reward-based, imo.
[+] [-] cicatriz|12 years ago|reply
Your distinction between learning facts and learning ways of thinking is interesting. Most of the spaced repetition research and design is around fact recall--what's called "paired associates" because you recall some response paired with some stimulus. I don't have any evidence for it, but I'd say that the schema can be lost just as well as facts can be.
One thing I will say is that spaced repetition is generally known for the repetition--that you retain something over time, and the spacing is more thought of as convenience. But the truth is that spacing is also part of better learning--when recall is difficult it's learned better (called "desirable difficulties"). I thought of that because Robert Bjork does that research with skills as well as facts. (See http://www.wired.com/geekdad/2012/01/everything-about-learni... for an overview.)
[+] [-] dwc|12 years ago|reply
But I'd say that if there's some are you'd like to work in that requires specific knowledge that fits well within SRS, why would you not? As opposed to trying to learn and recall anything that might possibly be useful, ever.
[+] [-] pixelcort|12 years ago|reply
This means, every time I successfully recall the answer to a question, I increase the interval of when I will be challenged with it again by about 2.71828 times since I had seen it previously.
This is with a recall failure rate of about 5%, which I'm trying to maintain. (If it's nearing 0, then you're probably wasting time being challenged too frequently.)
This is on a data set used daily for 30 minutes, reviewing about 50 older questions and encountering around 3 new ones per day.
[+] [-] jacques_chester|12 years ago|reply
> At least with physical fitness there isn’t a precisely dismaying number indicating how far behind you are!
Many sports are precisely measurable and in such sports, the measurements determine who has won, to various resolutions. Shortest running time over a given distance, heaviest weight snatched, furthest distance jumped and so on.
Athletes in such sports can very accurately track their performance over time. As of yesterday I demonstrated that I can clean & jerk 25kg less than my lifetime record, for example.
[+] [-] gwern|12 years ago|reply
[+] [-] hngiszmo|12 years ago|reply
FluxCards provides spaced repetition learning with easy to edit and create own cards. You can sneak the answer line by line, you can finger-paint and the stats are fast and meaningful.
It is free to use with 0.5, 1, 2, 4, 8, 16 days intervals. Beyond that you need the premium version but your cards are yours, so if you consider it not worth the fee, you can export your cards to SD and use them in Anki.
And yes, spaced repetition rocks. I can only encourage any flash cards programmer to implement it, as all these apps that implement flash cards but no spaced repetition learning are just not worth your time.
[+] [-] jms|12 years ago|reply
I'm also building a language learning startup that uses SRS. If you're interested in trying the system out in beta, or being notified when it's ready for general consumption, email me at [email protected] and let me know which language you want to learn.
[+] [-] gingerlime|12 years ago|reply
One interesting thing we've found early on, was that when we encouraged people to actually "take a break", not to repeat too much, but instead let your brain process and come back to it later, people didn't like it. They wanted the ability to keep training and training. This resonates well with what the article describe about students' cramming tendency. So some time you have to manipulate even a solid algorithm, to fit with reality.
[1]https://www.kenhub.com
[+] [-] jordwest|12 years ago|reply
Spaced repetition is pretty amazing. Adding records feels like you're entering them into your brain, provided you use it every day. I managed to go from knowing almost nothing to scoring 99% in a Japanese kanji exam once, entirely thanks to Anki.
My thesis is available here: https://github.com/jordwest/Thesis-Report/raw/master/report....
The online SRS I built is also available: https://github.com/jordwest/Membit
[+] [-] styrmis|12 years ago|reply
I always wanted to make my own SRS, one that had ease of use , stability and data integrity at its core rather than cutting edge features. This year I finally set aside time to make it: feel free to try it out and let me know what you think. It is very simple (there is only one deck), but multiple decks will be the next feature to be added.
http://benkyo.co
On top of that platform is an implementation of a rather interesting way to learn Japanese reading which it would also be great to have feedback on:
http://benkyo.co/iikanji/
[+] [-] Ideka|12 years ago|reply
[+] [-] narwally|12 years ago|reply
[+] [-] fsiefken|12 years ago|reply
http://sachachua.com/blog/2013/02/rediscovering-emacs-featur...
[+] [-] gwern|12 years ago|reply
[+] [-] Mithaldu|12 years ago|reply
[+] [-] thinkersilver|12 years ago|reply
[+] [-] peterderivaz|12 years ago|reply
One thing that I've wondered is whether it is a good idea to try to think up mnemonics for tricky words when using spaced repetition. It certainly seems effective in helping recall but I worry that I might train my brain to always rely on going via an indirect path to get to the information and that this will slow down my ability to understand real language.
Has anyone seen any research along these lines?
[+] [-] dunstad|12 years ago|reply