Looks like a great start, played around with it a bit yesterday and today, I've basically been doing the same with my own CLI but the UI you came up with helps a great deal with navigation and resuming learning :)
One issue I found is the typical "LLM accuracy" issue, with seemingly no recurse. I tried to generate some courses for topics I already know well, just to review how accurate it is, and while popular subjects (ex: "Electronic Music Fundamentals") it gets most of the details correct, less popular subjects (ex: "Scene Transitions with Octatrack") are riddled with errors (both in the "docs" and the quizes/exercises), and I cannot find a way of correcting/adjusting/reporting the errors.
Yeah it's still hard to deal with LLM gaps (fwiw Study mode would also be prone to this). I do try to catch the super obvious stuff and put up a disclaimer but it's far from perfect.
I had some prototypes basing the generations in websearch but the APIs are still super expensive on that front + the models tend to overindex on the results.
There are both in-document quizzes and larger exams (at a course level).
I've also been playing around with adapting content based on their results (e.g. proactively nudging complexity up/down) but haven't gotten it to a good place yet.
diggan|7 months ago
One issue I found is the typical "LLM accuracy" issue, with seemingly no recurse. I tried to generate some courses for topics I already know well, just to review how accurate it is, and while popular subjects (ex: "Electronic Music Fundamentals") it gets most of the details correct, less popular subjects (ex: "Scene Transitions with Octatrack") are riddled with errors (both in the "docs" and the quizes/exercises), and I cannot find a way of correcting/adjusting/reporting the errors.
tootyskooty|7 months ago
I had some prototypes basing the generations in websearch but the APIs are still super expensive on that front + the models tend to overindex on the results.
danenania|7 months ago
Have you considered using the LLM to give tests/quizzes (perhaps just conversationally) in order to measure progress and uncover weak spots?
tootyskooty|7 months ago
I've also been playing around with adapting content based on their results (e.g. proactively nudging complexity up/down) but haven't gotten it to a good place yet.