I used to think I had this problem, too. I developed an elaborate categorizing and indexing scheme. I tried to apply it outside of my personal knowledge, creating a crawler/indexer for research and web sites in one my areas of interest. I thought "if only we organize things better we can change the world!"
I realized over time that the collection wasn't the hard part. It was the categorizing and simplifying. The author hits on it a few times:
"It is the extraction and organization of the information that takes time...
I know full automation is not feasible, since the imposition of meaning onto the raw information is something that I must do, not the computer."
In my experience, the simplifying is really where you get all the gain.
What is simplifying?
It's distilling a complex research paper into a few key data points.
It's naming files well so that you can search them with Spotlight.
It's learning to write more clearly.
etc.
That last one -- clear writing -- should have been obvious. Good writers manage to convey so much information in so little space. How do they do it?
The parallel to programming should be obvious. When you name things well, they become very easy to find and use.
We can expand this to a bigger point: if you really want to get smarter, you need to be learning to simplify because your brain can only hold so much at once.
It's like learning the law of gravity rather than cataloging every time an apple falls from a tree.
The following unintuitive conclusion arises: you should be looking to make your "Personal Knowledgebase" more difficult to grow because it forces you to go through the simplifying process sooner.
I stole this idea from someone on HN: anytime I find an idea or quote that I think it important, I clip it into a Word Document and print it out. I keep these in a binder. I have gone back to these notes so many more times than anything that I have in any digital form. More significantly, these bits of information have influenced my life more often and more deeply.
This could be viewed as layered ontologies, moving from the largest dataset to the most abstracted. Our categories are necessarily personal, but they eventually overlap with generic ones. Presently we don't have good tools to associate personal categories with those of larger datasets, because categories span apps, platforms, orgs & license regimes.
Classified data
Corporate data
Licensed data
Books
Open internet
Personal notes/kb
Human brain
Writing is the best and hardest way to think. "Drop by drop," as I've heard say.
I love the binder practice. I got a printer about a month ago, printed a bunch of files (notes, incomplete songs, etc), deleted them, then reduced the mess down to a few pages. That was a productive day.
I submit that we have a false equivalency—nay, a false superiority of digital files over paper. After all, that's what we're all about here, right?
As you say, the constraints of paper are a feature, not a bug. I would have made this comment shorter, but I didn't have time.
> anytime I find an idea or quote that I think it important, I clip it into a Word Document and print it out. I keep these in a binder. I have gone back to these notes so many more times than anything that I have in any digital form.
I do the same but I do keep it in digital form because I just hate paper. There is a browser plug in called 'scrapbook' that works very well for me, I've tied it to a hotkey (shift-ctrl-B) to capture whatever is highlit on the screen without further confirmation or other interaction. I periodically dump the scrapbook and distill it.
I like it because it keeps the data locally rather than on some cloud service.
We had these things called RedBooks at MetaDesign in SF. A huge library of notes, sketches, diagrams and final work etc from each project. There is something about the tangability of a physical book when you want to retrieve information thats just much more intuitive than having some complex structure on some hard disk somewhere. It's simply the wrong way to use computers.
The force of digitalizing information is the ability to retrieve contextual information from a fuzzy collection of knowledge.
Exactly, I've posted similar remarks on HN: I keep little quotes and principles in my main "notes" file so that they reach my eyes frequently. I've also recently made the move to formalize my personal project data a little(Fossil repo, add tickets to build a todo list). I still do everything digitally, but it's increasingly with more of a process - a process that itself aims to be as small and transparent as possible.
Is it impossible a technological solution exists? There has been a lot of research on automatically categorizing documents, or reducing them to a few dimensions.
Even if it works well, I'm not sure how much utility this information would provide.
I'm a bit late to the party, but I agree with your sentiment that distilling (or synthesizing) is key to a long-term personal knowledgebase. I don't use paper, but I often refer to my digital notes.
I use an open-source mindmapping program called Freeplane to record most of my knowledge and thoughts: concepts I've encountered, products I've evaluated, snippets and webpages I've enjoyed, plans I've made, pieces of my writing. I have mindmaps like "Personal", "Ideas", "Projects", "Coding", "Devops", and "Ruby on Rails". I learned about Freeplane from a friend a few years ago and it's been amazing. The keyboard shortcuts are very fluid (use them!) and the program is very fast since it's text-based.
I make sure I summarize the takeaway from each link or snippet I put into my mindmaps - what I want is information that my brain can directly use when I refer to it in the future. If I want more detail, I can refer to the source snippet or page.
Regarding software, I use Freeplane to record the majority of the thoughts and information I want to preserve, and I think it works well for me. It doesn't have the sheer freedom of pen on paper - which I still use for mapping out thoughts that I'm not quite sure about or have complex relationships - but it serves me well in 1) categorizing random scribbles and steps into meaningful subcategories, and 2) crystallizing the final, synthesized thoughts I have on a matter.
Plus I keep my map files in my Dropbox so they're synced to all my computers (although I haven't set up the software to view them on my phone). Scalable, easily reorganized and expanded, cross-connectable, very fast to input, synced and backed up, free - what's not to like about software?
I agree with you on the usefulness of mapping for testing. It's definitely has helped me a lot in problem-solving. I'll throw out a few nodes that I think I need to investigate, explore each one a bit, write more items to consider, and whittle down or branch out as necessary. So after a while I resolve all the branchy, bushy sub-issues and have a reasonable game plan. Sometimes I dive into code halfway, but switch back to the map to record where I am and add new issues that come up that I need to resolve.
I also record most of the coding methods I find while working on tasks, in general form. So my mindmaps are also a web of how-to notes or a gigantic cheatsheet that details how to achieve any effect that I've previously worked through: from comparatively minor ones like the syntax for Rails migrations or opening a new window in JS, to larger ones like how to set up a Rails+postgres+nginx stack on Ubuntu, recording every action taken and issue encountered along the way. Comparisons of tools and databases and frameworks, mysterious bugs that I've run across, Sublime Text shortcuts - they all go into the maps.
I'm not sure I need to record every thought that goes through my head like the author suggests, but I think there's a lot to be said for keeping a comprehensive, organized knowledgebase.
=========
edit: I should also mention that I use Anki for language learning. However, I don't find it necessary to use it to memorize other types of knowledge, because I don't actually need to memorize much knowledge - I only need to categorize and store it for easy retrieval.
For language learning, Anki also becomes less useful as you get better in a language - at a point, extensive reading becomes more important, and it acts as a "natural SRS". (This conclusion is from discussions on Chinese-Forums.com.)
simplification helps only in the way you start looking at things later. But then when you communicate to another person, he still needs all the data to come to the same level of simplification you achieved. Being able to organize the way your brain looks at it at some point in time and later reconfiguring as things evolve is what may be needed.
I'd very much appreciate a system like the one described as well. And I've got some further ideas. When tagging, suggested tags should be very smart - like Amazon/Netflix recommendation system smart. It should be backed with a Bayesian categorizer or similar. This sort of categorization engine could also be used to recommend hyperlink targets between sources when a term is selected.
It definitely needs to integrate with the web browser. Highlighting a web page extract and saving it should both save the extract to the knowledgebase and also mirror the entire page in the KB so that later you can hit a 'context' button and be given the original source it came from. Trust as little as possible that the content will remain available and unchanged.
I'd also like for there to be a 'bulk' area where things like entire books for future reference material could be saved reachable by the searching functionality.
An automatic chronological index would also be helpful. If I save a page about fractal image compression and from there move to wavelet compression, being able to see the history of what order the topics were visited in would be beneficial.
Integration with learning assistance systems would be nice as well, things like the various apps that use intelligently timed repetition of information in order to help with memory.
I do think that building from a wiki could be a possible start. My biggest concern with using a wiki or other web based system is the terrible layout options forced upon us by HTML and CSS. They are designed such that arbitrary layout is quite difficult, and arbitrary layout would be fairly necessary for this.
This is a great text; saving to re-read another time.
Personally, I use Dropbox + org-mode based personal wiki. I store each important topic in its own file, and the whole thing gets synced between all my machines. It basically looks like this:
I heavily use links between documents, to outside sources, and back-up anything important by downloading PDFs/pictures and saving plaintext in an org file.
My current pain points:
- internal linking - I try to avoid linking from .org file to another, because I expect that if I move a piece of text to another file (or even subheading, for heading-relative links), I won't be able to find all the links that broke in the other files. So I created a "dispatcher" file, that maps links to destinations, so when I move something, I only have one place to check. Unfortunately, this requires two clicks to traverse to your destination, and is basically annoying. So I'm looking for another solution for broken incoming links.
- mobile viewing - I don't use the org-agenda based flow that is apparently required for org-mode, and the Android is too stupid to let Dropbox open my org files (without workarounds that require a lot of tapping on the screen), because there's nothing registered to open files with .org extension sigh. I'm yet to find a good solution for this (even just a simple text editor that could register itself as an app for .org files would be immensely helpful).
- web viewing - sometimes I'd like to browse my personal wiki from someone else's machine, or maybe even link to a particular page; for that I'd love to have a web-viewable version of my wiki. I want it to work seamlessly, i.e. without having to manually regenerate or commit everything. This could be doable with Dropbox and a bit of org-export scripting, but I'm yet to get around to do it.
- web writing - sometimes I'd like to dump some notes into my wiki from someone else's machine; I haven't figured out how to make it work.
That's a very extensive list of org files you have there :)
I bypass org-agenda for mobile viewing by scheduling a job to call org-html-export-to-html on each .org file. Dropbox mobile works well for viewing the exported HTML. There is zero hassle once the script is running.
Here's the rake script I use: https://gist.github.com/shoover/d75a58074be9894bfc54. The core emacs batch command for html export is simple enough. The rake-isms are there to find all the .org files in the given directories and check timestamps. There is optional elisp to deal with loading my latest org-mode and not the built-in one. The INDEX business was added to deal with MobileOrg but I no longer use that. Read-only HTML in Dropbox is good enough for me.
Agreed that internal linking is a pain point. I just use `C-c l` and suffer the dead links.
Agreed regarding remote editing. Gmail drafts "transport" are what I end up using. It's not terrible but could be better.
After years of being a paying Evernote customer, I stopped using it several months ago. I am actively looking for alternatives. I found that I was spending too much time curating information in Evernote for the value I got from looking up material for reference.
One thing that I have been experimenting with is keeping a few top level subject directories in Dropbox and making notes in plain text markdown files. If I use a .txt file extension, then available iOS and Android editors that work with Dropbox give me coverage across tablet, phone, and laptop. Spotlight search on my laptop helps me find notes but finding the right information on mobile devices is a problem.
I also write notes in Google Drive and store potentially useful PDFs and purchased books in GD, and rely on GD search to find things.
I have tried org-mode but I no longer "live" in Emacs so org-mode is not ready at hand as it would have been ten or twenty years ago when I used Emacs for everything.
I have also struggled for a long time to get what I want out of existing personal knowledgebase solutions, despite loving many individual elements of the products that are already out there like Evernote, Workflowy and Dropbox/.txt files etc. This pain was particularly acute in college while drinking from the firehose of information, and not having a tight solution, such that years on so much value was simply lost and forgotten. I have been building the product I wished I had - Naked Knowledge. It would be great to get interested parties' feedback. You can follow us and sign up for the beta here if you like: https://nakedknowledge.com
We have had very similar experiences with Evernote, then. I also used Google Notebook before the (in my opinion very efficient) product was discontinued. Google Docs on the other hand forces you to open entire documents. This doesn't work well when the most frequent use case is a quick-and-dirty append.
This may sound like a "solve everything with git" type comment, but honestly, I have found that keeping a set of markdown files in a git repository (frequently pushed to github) works well for me. I can have a quick script that appends a default journal file for thoughts I don't want to spend time organizing immediately. For everything else, I open up the relevant .md file and add a bullet point.
So far it's working better than anything I've experimented with over the last ten years.
I've been using Arena for the last year or two: http://are.na.
It's conceptually based on blocks and channels, where a block is text, pdf, video, audio, image, etc. and a channel is an ordered set of blocks and other channels. Channels can be public, private, or private with collaborators. It's web-based and has a good API(1), which I use as a lightweight CMS for my own website. They plan(2) to eventually charge for a pro version, but for now they are accepting rewarded donations.
Have you looked at http://www.notebooksapp.com/? It stores data in plain text, supports markdown, will sync with Dropbox & WebDav, and does full text search & tagging. Available for OSX, iOS & Windows.
Agree with you. I have over 5000 notes on Evernote, but I only use its shortcut to write several content(about 10 notes). Now I am trying Weavi, but uploading files is not convenient like Dropbox. Fortunately, I find my files are less and less as I use more and more mobile devices. So what is need is pure content management not file management. If Weavi supports private records, and an extension like Evernote Clipper, it would be my best choice.
If what you have is mainly text, I highly recommend Workflowy. I have a top-level header called "Brain Dump" where I put any thoughts I have during the day, then I move them into subcategories when I have time.
The combination of Workflowy (with tagging), Pinboard, and Dropbox feels "right" to me.
God this is hard. I've been trying to get at a solution to this exact problem for a very long time as well. It needs to be there when I'm walking down the street, sitting in a busy meeting, or at home in bed. A hybrid of mind-mapping, notes, and a document database. I'd also like to be able to publish and organize documents for others to view.
Org-mode is incredible. I use it to capture everything from vocabulary (which is automatically put into org-drill for spaced repetition), to my ideas, todos, shopping lists, EVERYTHING. Capture can happen on phone or laptop and syncs seamlessly.
Agenda for captured todos syncs with gcal. Publishing / printing to pdf / web / latex no problem. Index or search? No problem. Inline images and video, figured out. Snippet capture from web browser? Done.
Plus the incredible power to modify all this using emacs lisp. It's not python but it's powerful and every other emacs mode uses it.
There zillions of modes and tweaks to the emacs environment that add to this.
Org-mode and emacs are the single most powerful organizational tools I've ever encountered.
I have one, but it's not public -- at least, not yet. One of the hardest things to get right here is search. It's easy to drop in a full-text search solution, quite another thing to design search that works well.
My system is a tiny app that automatically grabs pretty much everything I do on the Internet and indexes it. Github stars, HN posts, Facebook likes, bookmarks, you name it.
I can optionally tag or add my own sticky-notes to anything, and there's some other neat tricks that make fuzzy searching a lot easier. Human memories are very story-based, and so if I only remember that I looked at something important about ducks when I was at a Meetup, I can actually search for that.
I'm considering productizing this -- email's in the profile if you'd like to chat. :)
FWIW - I’ve always thought these guys were close to something (but not there yet) - https://gingkoapp.com/p/future-of-text - the idea of a quick dive-down/up through knowledge with potentially tagging/linking thrown in would seem pretty stellar from a knowledge management system.
Semi-automated organization is difficult, not because we don't have the algorithms to classify things into categories, but because everyone has a different idea about 1) which things belong where and 2) how much of it should be automated.
This is why everybody else's organization scheme feels alien to us, and why our own organization scheme, no matter how much sense we think it makes, will inevitably feel alien to everybody else.
If you build a tool to automate part of the process, and then release it to the whole wild world, people will complain that it files things in all the wrong places. If someone else builds a similar tool, and you use it, you will complain that it files things in all the wrong places. Because each of us not only has a different ontology, but also has different ideas about how we should approach ontology in the first place.
You could make your tool highly flexible so that it can accommodate all sorts of different ontologies, but then everyone will complain that there are too many options. It really sucks.
Oh well, I guess that's why we've ended up with a gazillion incompatible tools, each of which is probably good enough for whoever built it, but none of which really suits anybody else's needs.
"The app also allows you to connect things freely thus letting you express both your rational and irrational self. There are no universal categories or connections between different things - rather it's about an individual's own "ontology" that's created through usage. The "associative ontology" evolves continuously both through the actions of the individual and other people using the system. "
I don't think 'mistaken' associations need to be considered a bad thing, in fact I think they are a positive. Those things might stimulate thoughts and ideas not previously considered.
OneNote isn't a bad choice, but it doesn't hit all the marks here. It does shine in a few places for me:
- Search: Search is very very fast and also does OCR of text in images
- Cross-platform: Works on Windows desktop and on Android/iOS phones
- Online: Can also share OneNote pages as a link to those people who don't have OneNote
I'd be surprised if the format isn't propietary, and there's no CLI. But if those things aren't important to you, then basic usage of the application is awesome.
(My minor nit with it is that navigating Sections can be difficult, and "finding where you are" in a large notebook is also harder than it should be).
What I would like is an open source browser that autorecords all my non incognito browsing history, including videos, etc within a transparent database / file format.
Even with Google, I often have problems finding interesting things that I've read in the past.
How it would work: For each page, any time the DOM changes, that is saved as a revision of the document. This way, I can get back to any state of a website without javascript complexity by browsing the revision list.
Bookmarks / Favorites would still be useful for saving notable stuff, and ideally there would be a powerful data management facilities for search (eg. Solr), deletion and merging datasets. A future project, perhaps...
I've wanted a similar thing for a very long time. In effect, an infinite web browser that remembers everything. Rather than 'surfing' the web it would be more like mapping it out and exploring it, with more than just your memory to return to. I like the idea of a vertical tab bar on the side, and I think if the user interface were done right, one where the tabs of things which interest you never actually 'close' (closing a tab would be equivalent to dismissing something as irrelevant or uninteresting) but the list of tabs just continues growing, scrolling off the top of the screen but recallable (and searchable) at a moments notice. It would be major rethinking of how a browser works and what it does and it would possibly get confusing with things like web apps (shoehorning applications into a document presentation system bites us again). I imagine lots of people would presume a proposal like this, keeping everything you ever browse or at least a very significant subset of it, as slightly insane... but that just makes it more alluring to me.
I dont know about all the other items on the wishlist but you might be able to find what you were doing/watching when.
https://github.com/gurgeh/selfspy
I would just like to dump tons of stuff into it. Scraps of thought, sketches, images, archived web pages, pdfs etc... then have it organise itself, finding links and connections between things, perhaps with some way of shaping and guiding it. I just don't have the patience to tag things or organise things into a hierarchy. I inevitably end up with a huge blob of 'uncategorised' stuff.
I wouldn't mind if the algorithm for making connections made some mistakes - the important thing would be helping you connect things together in novel and unexpected ways, based on your curated collection of stuff. Serendipity is super important.
I'm surprised he doesn't talk about Zotero. https://www.zotero.org/
Especially for a grad student, it seems like a better option than any of the other things he's tried/discussed.
While you're at it, please add a spaced repetition learning feature à la Anki. Our brains are the best PKB, we just need the right tools to efficiently upload data to it (anki)
Org-mode has all of this. Non-proprietary? Easy capture? Indexing? Search? Agenda? Mobile-capture and sync? Org-drill for spaced repetition? Integration with snippet capture in browser?
If there was a cross-platform, open database that could be used by commercial or open-source apps, then we could have a competitive market on "organizing UX" without worrying that a policy change at our favorite indie developer or global conglomerate would orphan the data. E.g. WebDAV, CalDAV, OPML, a dublin core metadata file which accompanies artifact files (photo, pdf, warc), sqlite db, camlistore, git-annex, ..? Currently using a combination of these tools for cloud-neutral kb:
Mind mapping: iThoughts HD on iOS, export to PDF
Idea capture: OmniFocus on iOS, export to XML
Contextual & tag search with preview snippets: xapian + recoll on Linux
Web clipping: Print To PDF (wkhtmltopdf) on Firefox or Epub
File tagging: Calibre on Windows, plus SumatraPDF for viewing epub, pdf, djvu
OCR: Abbyy on Windows
Nepomuk was/is kind of a solution like that, though mostly targeted at the desktop. It would store all the data in an RDF database (with layers to speak other protocols, like IMAP) using standard formats, and applications would interact with it instead of having their own storage layer.
As for the research part, I think one major issue is that the workflow is too personal and not (necessarily) compatible with other people.
In our research group we ended up doing annotated bibliography in a common repo. bibtex format and git. This way the kbr grows over the time with distributed effort.
There is a common shared structure. You have to provide well written bibtex ref and a paragraph or 2 of annotations. The first paragraph has a strict structure where essentially you reply to a few questions. Other paragraphs can be added freely by each person (with your initials).
Anyway, this is just specific to research and a subset of the kbr, but I'd think also to ways where what you collect can be useful to others, and viceversa others can add to your kbr without generating mess.
IMHO I don't think we lack knowledge capture and retrieval tools as much as we lack integration between those tools. I use Zim as a desktop wiki and todo list tool, Freemind as a mind mapper, TaskJuggler 3 as a project planning tool and Zotereo as a reference capture tool - and they are all very good at what they do but they just don't work together very well. A high level of integration between tools like this would be the killer app in this space. I have hopes that Camlistore would be part of this solution.
There is a real opportunity here for evernote and even google. What he needs a personalized wikipedia, a "facebook" of googleable archive of his selected internet content.
If facebook/myspace are the social networks for people's offline life, how about a social network for people's brain in online life.
It is much more than a personal knowledgeabase per se. This may very well be a knowledge graph in google's scale but with personalized context and relativity.
I use the Outline View in MS Word to organize my bookmarks. I use 4 levels: Heading - Subheading -- Link --- Text. (Note that each level is a separate line in MS Word Outline View.)
I'm Alex, the author of the blog post. I'm so honored that my post made it here to HN. :) Obviously the itch I'm trying to scratch is one that a lot people are.
The suggestions here are very helpful. I think all of us have learned to cobble together a suite of tools to do what we want, but it seems that few are totally satisfied. I don't think what I've proposed, or what other have proposed is insurmountable - we do far more complex things with computers. But given the idiosyncrasies of our individual workflows, maybe it's unrealistic to think we'll find a solution that everyone likes.
I'm going to compile a lot of the suggestions from this thread and around the web in a follow up post on the blog.
I want to quickly address some of the comments on Anki. I've been using Anki for a few years now, mostly to handle the massive amount of knowledge in medical school. I've written about it here. Anki has caught on in med school significantly.
There is no much interest in Anki for knowledge management (and retention), that I'm working on an eBook for med students about it - http://www.learningmedicinebook.com/
I used to think that everything I came across should live in my brain. And so, I went a little crazy with Anki in the beginning, capturing EVERYTHING I read. That quickly wore me down, and I had to become more discriminating. A lot of novice Anki users fall into a similar trap. I realized that not everything is worth occupying my headspace. I've been trying to come up with criteria for what should and shouldn't live in my head, but regardless, I've come to point where I want to offload most of the heavy lifting to my PKB. The really high yield bits from my PKB, however, will become Anki cards, so that the most important things I remember, and serve as 'crumbs' back to the details in my PKB. I'm going to flesh this out further in future posts. I love the enthusiasm here for spaced repetition. It's very powerful.
Last thing, regarding the comments on a better collaborative environment for Anki decks. I completely agree! I love Anki as much as the next guy, but there are some serious deficiencies. Collaboration being one of the them. I still think Anki is the best in class right now, but there is a new tool on the horizon that I'm excited about, and I think it will overtake Anki eventually.
It's called Memorang: https://www.memorangapp.com/. I'm enthusiastic about this app, and while it doesn't have everything I need yet (I think the scheduling could be better), the collaborative environment is excellent. It's worth checking it out. Perhaps one day I can integrate with my PKB.
Anyway, thanks again for checking out my post. The discussion this post has spurred is very fruitful.
I've had some success using Confluence hosted from Atlassian. The front-end is well-polished and the document management features work well (mostly drag and drop). There's also a back-end API to extract JSON and XML data. Paying $10/month for the hosted service avoids having to spend time on maintenance. Confluence clearly doesn't meet all of the author's requirements, but it's a nice trade-off for someone interested in a simple pkb.
[+] [-] sirgawain33|11 years ago|reply
I realized over time that the collection wasn't the hard part. It was the categorizing and simplifying. The author hits on it a few times:
"It is the extraction and organization of the information that takes time... I know full automation is not feasible, since the imposition of meaning onto the raw information is something that I must do, not the computer."
In my experience, the simplifying is really where you get all the gain.
What is simplifying?
It's distilling a complex research paper into a few key data points. It's naming files well so that you can search them with Spotlight. It's learning to write more clearly. etc.
That last one -- clear writing -- should have been obvious. Good writers manage to convey so much information in so little space. How do they do it?
The parallel to programming should be obvious. When you name things well, they become very easy to find and use.
We can expand this to a bigger point: if you really want to get smarter, you need to be learning to simplify because your brain can only hold so much at once.
It's like learning the law of gravity rather than cataloging every time an apple falls from a tree.
The following unintuitive conclusion arises: you should be looking to make your "Personal Knowledgebase" more difficult to grow because it forces you to go through the simplifying process sooner.
I stole this idea from someone on HN: anytime I find an idea or quote that I think it important, I clip it into a Word Document and print it out. I keep these in a binder. I have gone back to these notes so many more times than anything that I have in any digital form. More significantly, these bits of information have influenced my life more often and more deeply.
[+] [-] walterbell|11 years ago|reply
[+] [-] gavinpc|11 years ago|reply
I love the binder practice. I got a printer about a month ago, printed a bunch of files (notes, incomplete songs, etc), deleted them, then reduced the mess down to a few pages. That was a productive day.
I submit that we have a false equivalency—nay, a false superiority of digital files over paper. After all, that's what we're all about here, right?
As you say, the constraints of paper are a feature, not a bug. I would have made this comment shorter, but I didn't have time.
EDIT Here you go, from right now: https://news.ycombinator.com/item?id=8272394
[+] [-] jacquesm|11 years ago|reply
I do the same but I do keep it in digital form because I just hate paper. There is a browser plug in called 'scrapbook' that works very well for me, I've tied it to a hotkey (shift-ctrl-B) to capture whatever is highlit on the screen without further confirmation or other interaction. I periodically dump the scrapbook and distill it.
I like it because it keeps the data locally rather than on some cloud service.
[+] [-] ThomPete|11 years ago|reply
We had these things called RedBooks at MetaDesign in SF. A huge library of notes, sketches, diagrams and final work etc from each project. There is something about the tangability of a physical book when you want to retrieve information thats just much more intuitive than having some complex structure on some hard disk somewhere. It's simply the wrong way to use computers.
The force of digitalizing information is the ability to retrieve contextual information from a fuzzy collection of knowledge.
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] chipsy|11 years ago|reply
[+] [-] Houshalter|11 years ago|reply
Even if it works well, I'm not sure how much utility this information would provide.
[+] [-] creamyhorror|11 years ago|reply
I use an open-source mindmapping program called Freeplane to record most of my knowledge and thoughts: concepts I've encountered, products I've evaluated, snippets and webpages I've enjoyed, plans I've made, pieces of my writing. I have mindmaps like "Personal", "Ideas", "Projects", "Coding", "Devops", and "Ruby on Rails". I learned about Freeplane from a friend a few years ago and it's been amazing. The keyboard shortcuts are very fluid (use them!) and the program is very fast since it's text-based.
How it looks (at top level): http://imgur.com/QRkJ9vh
I make sure I summarize the takeaway from each link or snippet I put into my mindmaps - what I want is information that my brain can directly use when I refer to it in the future. If I want more detail, I can refer to the source snippet or page.
See the reviews for Freeplane, they're almost all 5 stars: http://sourceforge.net/projects/freeplane/reviews
Quoting myself from a year ago:
========
Regarding software, I use Freeplane to record the majority of the thoughts and information I want to preserve, and I think it works well for me. It doesn't have the sheer freedom of pen on paper - which I still use for mapping out thoughts that I'm not quite sure about or have complex relationships - but it serves me well in 1) categorizing random scribbles and steps into meaningful subcategories, and 2) crystallizing the final, synthesized thoughts I have on a matter.
Plus I keep my map files in my Dropbox so they're synced to all my computers (although I haven't set up the software to view them on my phone). Scalable, easily reorganized and expanded, cross-connectable, very fast to input, synced and backed up, free - what's not to like about software?
I agree with you on the usefulness of mapping for testing. It's definitely has helped me a lot in problem-solving. I'll throw out a few nodes that I think I need to investigate, explore each one a bit, write more items to consider, and whittle down or branch out as necessary. So after a while I resolve all the branchy, bushy sub-issues and have a reasonable game plan. Sometimes I dive into code halfway, but switch back to the map to record where I am and add new issues that come up that I need to resolve.
I also record most of the coding methods I find while working on tasks, in general form. So my mindmaps are also a web of how-to notes or a gigantic cheatsheet that details how to achieve any effect that I've previously worked through: from comparatively minor ones like the syntax for Rails migrations or opening a new window in JS, to larger ones like how to set up a Rails+postgres+nginx stack on Ubuntu, recording every action taken and issue encountered along the way. Comparisons of tools and databases and frameworks, mysterious bugs that I've run across, Sublime Text shortcuts - they all go into the maps.
I'm not sure I need to record every thought that goes through my head like the author suggests, but I think there's a lot to be said for keeping a comprehensive, organized knowledgebase.
=========
edit: I should also mention that I use Anki for language learning. However, I don't find it necessary to use it to memorize other types of knowledge, because I don't actually need to memorize much knowledge - I only need to categorize and store it for easy retrieval.
For language learning, Anki also becomes less useful as you get better in a language - at a point, extensive reading becomes more important, and it acts as a "natural SRS". (This conclusion is from discussions on Chinese-Forums.com.)
[+] [-] thallukrish|11 years ago|reply
[+] [-] otakucode|11 years ago|reply
It definitely needs to integrate with the web browser. Highlighting a web page extract and saving it should both save the extract to the knowledgebase and also mirror the entire page in the KB so that later you can hit a 'context' button and be given the original source it came from. Trust as little as possible that the content will remain available and unchanged.
I'd also like for there to be a 'bulk' area where things like entire books for future reference material could be saved reachable by the searching functionality.
An automatic chronological index would also be helpful. If I save a page about fractal image compression and from there move to wavelet compression, being able to see the history of what order the topics were visited in would be beneficial.
Integration with learning assistance systems would be nice as well, things like the various apps that use intelligently timed repetition of information in order to help with memory.
I do think that building from a wiki could be a possible start. My biggest concern with using a wiki or other web based system is the terrible layout options forced upon us by HTML and CSS. They are designed such that arbitrary layout is quite difficult, and arbitrary layout would be fairly necessary for this.
[+] [-] TeMPOraL|11 years ago|reply
Personally, I use Dropbox + org-mode based personal wiki. I store each important topic in its own file, and the whole thing gets synced between all my machines. It basically looks like this:
http://imgur.com/Xgpzj5f
I heavily use links between documents, to outside sources, and back-up anything important by downloading PDFs/pictures and saving plaintext in an org file.
My current pain points:
- internal linking - I try to avoid linking from .org file to another, because I expect that if I move a piece of text to another file (or even subheading, for heading-relative links), I won't be able to find all the links that broke in the other files. So I created a "dispatcher" file, that maps links to destinations, so when I move something, I only have one place to check. Unfortunately, this requires two clicks to traverse to your destination, and is basically annoying. So I'm looking for another solution for broken incoming links.
- mobile viewing - I don't use the org-agenda based flow that is apparently required for org-mode, and the Android is too stupid to let Dropbox open my org files (without workarounds that require a lot of tapping on the screen), because there's nothing registered to open files with .org extension sigh. I'm yet to find a good solution for this (even just a simple text editor that could register itself as an app for .org files would be immensely helpful).
- web viewing - sometimes I'd like to browse my personal wiki from someone else's machine, or maybe even link to a particular page; for that I'd love to have a web-viewable version of my wiki. I want it to work seamlessly, i.e. without having to manually regenerate or commit everything. This could be doable with Dropbox and a bit of org-export scripting, but I'm yet to get around to do it.
- web writing - sometimes I'd like to dump some notes into my wiki from someone else's machine; I haven't figured out how to make it work.
[+] [-] shoover|11 years ago|reply
I bypass org-agenda for mobile viewing by scheduling a job to call org-html-export-to-html on each .org file. Dropbox mobile works well for viewing the exported HTML. There is zero hassle once the script is running.
Here's the rake script I use: https://gist.github.com/shoover/d75a58074be9894bfc54. The core emacs batch command for html export is simple enough. The rake-isms are there to find all the .org files in the given directories and check timestamps. There is optional elisp to deal with loading my latest org-mode and not the built-in one. The INDEX business was added to deal with MobileOrg but I no longer use that. Read-only HTML in Dropbox is good enough for me.
Agreed that internal linking is a pain point. I just use `C-c l` and suffer the dead links.
Agreed regarding remote editing. Gmail drafts "transport" are what I end up using. It's not terrible but could be better.
[+] [-] benaiah|11 years ago|reply
[0]: https://github.com/kelvinh/org-page
[+] [-] mark_l_watson|11 years ago|reply
One thing that I have been experimenting with is keeping a few top level subject directories in Dropbox and making notes in plain text markdown files. If I use a .txt file extension, then available iOS and Android editors that work with Dropbox give me coverage across tablet, phone, and laptop. Spotlight search on my laptop helps me find notes but finding the right information on mobile devices is a problem.
I also write notes in Google Drive and store potentially useful PDFs and purchased books in GD, and rely on GD search to find things.
I have tried org-mode but I no longer "live" in Emacs so org-mode is not ready at hand as it would have been ten or twenty years ago when I used Emacs for everything.
[+] [-] jereme|11 years ago|reply
[+] [-] hliyan|11 years ago|reply
This may sound like a "solve everything with git" type comment, but honestly, I have found that keeping a set of markdown files in a git repository (frequently pushed to github) works well for me. I can have a quick script that appends a default journal file for thoughts I don't want to spend time organizing immediately. For everything else, I open up the relevant .md file and add a bullet point.
So far it's working better than anything I've experimented with over the last ten years.
[+] [-] zachrose|11 years ago|reply
It's conceptually based on blocks and channels, where a block is text, pdf, video, audio, image, etc. and a channel is an ordered set of blocks and other channels. Channels can be public, private, or private with collaborators. It's web-based and has a good API(1), which I use as a lightweight CMS for my own website. They plan(2) to eventually charge for a pro version, but for now they are accepting rewarded donations.
(1) http://dev.are.na/documentation/channels (2) http://future.are.na/
[+] [-] walterbell|11 years ago|reply
[+] [-] Nadiney|11 years ago|reply
[+] [-] SnacksOnAPlane|11 years ago|reply
The combination of Workflowy (with tagging), Pinboard, and Dropbox feels "right" to me.
[+] [-] darkxanthos|11 years ago|reply
I wish I had a solution.
[+] [-] Ixiaus|11 years ago|reply
Agenda for captured todos syncs with gcal. Publishing / printing to pdf / web / latex no problem. Index or search? No problem. Inline images and video, figured out. Snippet capture from web browser? Done.
Plus the incredible power to modify all this using emacs lisp. It's not python but it's powerful and every other emacs mode uses it.
There zillions of modes and tweaks to the emacs environment that add to this.
Org-mode and emacs are the single most powerful organizational tools I've ever encountered.
[+] [-] donw|11 years ago|reply
My system is a tiny app that automatically grabs pretty much everything I do on the Internet and indexes it. Github stars, HN posts, Facebook likes, bookmarks, you name it.
I can optionally tag or add my own sticky-notes to anything, and there's some other neat tricks that make fuzzy searching a lot easier. Human memories are very story-based, and so if I only remember that I looked at something important about ducks when I was at a Meetup, I can actually search for that.
I'm considering productizing this -- email's in the profile if you'd like to chat. :)
[+] [-] ckluis|11 years ago|reply
[+] [-] porter|11 years ago|reply
[+] [-] walterbell|11 years ago|reply
[+] [-] kijin|11 years ago|reply
This is why everybody else's organization scheme feels alien to us, and why our own organization scheme, no matter how much sense we think it makes, will inevitably feel alien to everybody else.
If you build a tool to automate part of the process, and then release it to the whole wild world, people will complain that it files things in all the wrong places. If someone else builds a similar tool, and you use it, you will complain that it files things in all the wrong places. Because each of us not only has a different ontology, but also has different ideas about how we should approach ontology in the first place.
You could make your tool highly flexible so that it can accommodate all sorts of different ontologies, but then everyone will complain that there are too many options. It really sucks.
Oh well, I guess that's why we've ended up with a gazillion incompatible tools, each of which is probably good enough for whoever built it, but none of which really suits anybody else's needs.
[+] [-] walterbell|11 years ago|reply
Random is an iOS app/svc attempting to infer "associative ontologies" from navigation patterns, http://www.datascienceweekly.org/data-scientist-interviews/r...
"The app also allows you to connect things freely thus letting you express both your rational and irrational self. There are no universal categories or connections between different things - rather it's about an individual's own "ontology" that's created through usage. The "associative ontology" evolves continuously both through the actions of the individual and other people using the system. "
[+] [-] bjz_|11 years ago|reply
[+] [-] bentcorner|11 years ago|reply
- Search: Search is very very fast and also does OCR of text in images
- Cross-platform: Works on Windows desktop and on Android/iOS phones
- Online: Can also share OneNote pages as a link to those people who don't have OneNote
I'd be surprised if the format isn't propietary, and there's no CLI. But if those things aren't important to you, then basic usage of the application is awesome.
(My minor nit with it is that navigating Sections can be difficult, and "finding where you are" in a large notebook is also harder than it should be).
[+] [-] nextos|11 years ago|reply
It's has two distinct advantages: the filesystem is the database (plus a plain-text one!), and everything is scriptable using Emacs Lisp.
[+] [-] grizzles|11 years ago|reply
Even with Google, I often have problems finding interesting things that I've read in the past.
How it would work: For each page, any time the DOM changes, that is saved as a revision of the document. This way, I can get back to any state of a website without javascript complexity by browsing the revision list.
Bookmarks / Favorites would still be useful for saving notable stuff, and ideally there would be a powerful data management facilities for search (eg. Solr), deletion and merging datasets. A future project, perhaps...
[+] [-] otakucode|11 years ago|reply
[+] [-] hugogee|11 years ago|reply
[+] [-] yellow_and_gray|11 years ago|reply
https://www.linkedin.com/company/webmynd-corp
http://techcrunch.com/2011/01/26/webmynd/
[+] [-] bjz_|11 years ago|reply
I wouldn't mind if the algorithm for making connections made some mistakes - the important thing would be helping you connect things together in novel and unexpected ways, based on your curated collection of stuff. Serendipity is super important.
[+] [-] ColinCera|11 years ago|reply
[+] [-] eik3_de|11 years ago|reply
[+] [-] Ixiaus|11 years ago|reply
Every single one checked.
[+] [-] walterbell|11 years ago|reply
[+] [-] icebraining|11 years ago|reply
[+] [-] ecesena|11 years ago|reply
In our research group we ended up doing annotated bibliography in a common repo. bibtex format and git. This way the kbr grows over the time with distributed effort.
There is a common shared structure. You have to provide well written bibtex ref and a paragraph or 2 of annotations. The first paragraph has a strict structure where essentially you reply to a few questions. Other paragraphs can be added freely by each person (with your initials).
Anyway, this is just specific to research and a subset of the kbr, but I'd think also to ways where what you collect can be useful to others, and viceversa others can add to your kbr without generating mess.
[+] [-] flarg|11 years ago|reply
[+] [-] bitcuration|11 years ago|reply
If facebook/myspace are the social networks for people's offline life, how about a social network for people's brain in online life.
It is much more than a personal knowledgeabase per se. This may very well be a knowledge graph in google's scale but with personalized context and relativity.
[+] [-] platz|11 years ago|reply
[+] [-] rasengan0|11 years ago|reply
[+] [-] pdm55|11 years ago|reply
An example:
Education - Assessment -- http://www.nytimes.com/2014/09/07/magazine/why-flunking-exam... --- test and test often to focus the learners' minds
Business - Insights --http://thecodist.com/article/lessons_from_a_lifetime_of_bein... --- find dumb people with lots of money
I find this to be a simple, easy system to organize what I find on the internet.
[+] [-] achamess|11 years ago|reply
I'm Alex, the author of the blog post. I'm so honored that my post made it here to HN. :) Obviously the itch I'm trying to scratch is one that a lot people are.
The suggestions here are very helpful. I think all of us have learned to cobble together a suite of tools to do what we want, but it seems that few are totally satisfied. I don't think what I've proposed, or what other have proposed is insurmountable - we do far more complex things with computers. But given the idiosyncrasies of our individual workflows, maybe it's unrealistic to think we'll find a solution that everyone likes.
I'm going to compile a lot of the suggestions from this thread and around the web in a follow up post on the blog.
I want to quickly address some of the comments on Anki. I've been using Anki for a few years now, mostly to handle the massive amount of knowledge in medical school. I've written about it here. Anki has caught on in med school significantly.
http://drwillbe.blogspot.com/2011/08/anki-guide-for-medical-...
There is no much interest in Anki for knowledge management (and retention), that I'm working on an eBook for med students about it - http://www.learningmedicinebook.com/
I used to think that everything I came across should live in my brain. And so, I went a little crazy with Anki in the beginning, capturing EVERYTHING I read. That quickly wore me down, and I had to become more discriminating. A lot of novice Anki users fall into a similar trap. I realized that not everything is worth occupying my headspace. I've been trying to come up with criteria for what should and shouldn't live in my head, but regardless, I've come to point where I want to offload most of the heavy lifting to my PKB. The really high yield bits from my PKB, however, will become Anki cards, so that the most important things I remember, and serve as 'crumbs' back to the details in my PKB. I'm going to flesh this out further in future posts. I love the enthusiasm here for spaced repetition. It's very powerful.
Last thing, regarding the comments on a better collaborative environment for Anki decks. I completely agree! I love Anki as much as the next guy, but there are some serious deficiencies. Collaboration being one of the them. I still think Anki is the best in class right now, but there is a new tool on the horizon that I'm excited about, and I think it will overtake Anki eventually.
It's called Memorang: https://www.memorangapp.com/. I'm enthusiastic about this app, and while it doesn't have everything I need yet (I think the scheduling could be better), the collaborative environment is excellent. It's worth checking it out. Perhaps one day I can integrate with my PKB.
Anyway, thanks again for checking out my post. The discussion this post has spurred is very fruitful.
[+] [-] neovive|11 years ago|reply