Ask HN: How do you manage/organize information and knowledge in your life?

[ay1n]

This is a general question, but I didn't want to ask specific one because I'm still trying to understand the bigger picture.

We "collect" data all the time, and sometimes it helps us learn something. On one hand we have things that were created by others: funny images, movies, articles on the web, books. On the other we have things creted by us: personal insights, thoughts, snippets of code. How do you manage/organize all of this? Do you have apps/tools for organizing specific type of information and if so, why this particular one? Do you have ideas how to organize/manage this efficiently?

But we're not only collecting information [0], we also learn from it. Most people store all this knowledge in their heads for their whole life, but I think it's not efficient (memory loss, can't search, not always reliable etc.), some people create some kind of personal knowledge bases (e.g. personal dokuwiki/mediawiki on localhost). How do you manage/organize things you learn?

It may seem that I should create 2 different topics, but for me both concepts are connected[1]. What I'm trying to find is an efficient solution to managing (almost) all the information that enters (and already is in) my life. I don't think that there exist a good app for that, I know that there are some that solve parts of this problem (evernote, wikis etc.), but I'm more interested in your ideas on the whole topic, how to approach this problem, where to look, how to think about this etc.

I'm curious about your solutions, ideas and "setups" for this problem(s). If you have any resources (books, research papers etc.) about the topic, I'd love to learn from them. Thank you for your time.

[0] as in bits on the disk, learning can be viewed as collecting new information, I guess

[1] I liked quote from a book, some code from LLVM gave me an insight into a compiler design etc.

[thaumaturgy]

1. Live simply. As I've gotten older I've found more and more value in not having too many things going on. Gradually I'm sleeping better, eating better, reducing stress, and getting more exercise, all of which is important for the next thing:

2. Memorize as much as I can. It's an exercise; I memorize phone numbers, schedules, people's names, trivia, all kinds of stuff. I've never found anything that matches the flexibility and utility of my own brain. I should use the best tool I have, and that's in my head. Technology is unreliable and constantly changing and difficult to organize and search. I've been practicing this for long enough that now I'm pretty good at it.

3. For everything else, I use a few simple systems: a few sheets of paper to the right side of my desk for scribbling and note-taking (meant to be discarded after a day or two), a pile of to-do to my left, a tab open in my text editor labeled "notepad" for longer-term stuff, and a well-organized directory of documents on my laptop with subfolders like "projects", "writing", "sysadmin", etc. -- I try to keep this directory as small as possible by dedicating time here and there to either finishing or pruning projects.

I disagree that keeping knowledge in your head isn't efficient. I think a lot of people just don't practice it enough. Smartphones and computers and everything else make it really easy to not bother. But, my brain is always with me, doesn't require batteries (well...), can store any type of information I want, and can instantly recall it without having to craft some kind of search query or organize the information in a rigorous way. It is exactly the kind of database storage we all wish we had. It never changes data formats, it never tries to get acquired by a bigger company and then shut down, and it gets reception everywhere I go. If my brain were an electronic tool, I would want to use it all the time. And, the more I use it, the better it works.

(edit: oh yeah, and pinboard. Looove pinboard.)

[meesterdude]

One of the problems of keeping everything in your head is trust. People forget! Maybe you have a great memory, but eventually it'll forget things. And it should! otherwise there won't be room for any new stuff. I don't think we have infinite storage available; at some point things have to be erased to make room.

But also, keeping that stuff in a system you can trust frees your brain to do other things. Instead of falling asleep reminding yourself to buy milk tomorrow, you can leave it to the system to remember. The system being anything - a notebook, an app, or what have you. There is some good evidence that this kind of delegation allows for more high-level thinking to come about, since your brain is more free to do other things.

Now, as you've touched on - sometimes technology screws you one way or another, and that trust is broken. So if you have no redundancy or backup or plan for that dependency, maybe you'd be better off keeping things in your head.

But you are human and your head is human by extension. it is faulty, imperfect, and not nearly as good at storing things as pretty much anything else. Even if its good now, it won't necessarily always be, and isn't in such good shape for a lot of folk. So I would not paint it in such a rosy light.

I'm curious, Have you done any experimentation with Method of loci?

[bsilvereagle]

> 2. Memorize as much as I can.

I agree that you should rely on your memory. However, when you die, all that information you have is gone to the world. That's why I started writing things down, for when I'm gone.

Take a look at Roy Underhill, if he hadn't started writing books and opened a school, all of the woodworking knowledge he has acquired would be gone forever (or until someone else dedicated a lifetime to finding it all over again).

[toomuchtodo]

Having turned 32 this year, I can relate with all of your points.

Make life as simple as possible ("simplicity is the ultimate sophistication"). Use paper when necessary.

Also, try hard not to multi-task. It's been proven the human brain isn't built to handle it.

[ay1n]

I agree on living simply, everyday I'm trying to do this better.

I also try to use my memory as often as possible and I think it's pretty good (I'm learning mnemonic techniques which are really fun to implement in everyday life). Yes, mind itself is reaaaallly good system, but (at least for me) not all my insights, ideas, notes etc. about particular topic are available on demand, but only some part of them. And if I don't think about one thing very often it can be lost forever. Nevertheless, thank you for taking time to write about your system and giving different perspective.

[read]

One special trick I've used is to write an idea down when I have it. Another is to write more often to unearth my ideas. What's more valuable that storing existing knowledge is generating new knowledge.

You won't reach your maximum potential if you only try keeping knowledge in your head.

  The power of the unaided mind is highly overrated.
  Without external aids, deep, sustained reasoning is
  difficult.

  - Don Norman

[TannerLD]

What system and/or application(s) do you use to help you remember things?

[ay1n]

There is a 2000 character limit in the question box, so here is what I'm using now:

- pinboard for managing bookmarks (database of things that may be useful sometime; probably never) & reading list for articles

- I'm testing tagspaces (http://www.tagspaces.org/) for local files organization (mostly tagging research papers and books; didn't like Mendeley)

- cardav & caldav from owncloud for contacts and events

- anki as a memorization tool (spaced repetition) - from languages to my own mistakes (i.e. "lessons learned", so I won't repeat them)

- for insights, notes, ideas, things I've learned & everything else I use personal wiki (media wiki) on localhost. This is the biggest part of my "system", I have there entries like things to buy someday, current project's notes/resources, useful scripts, configuration snippets, notes from books, journeys, analysis of my own behaviour, personal journal, ideas for startups etc. But it's hard to organize, it becomes a mess very easly after some time. Also, I can't use it on mobile (I don't want to put all this on the web, there is a lot of personal info), it takes time to add new thing/entry (I need to think to which page new piece of information belongs etc.).

- simplified version of gtd as a meta-system managing this system and for projects/things to do

[frik]

I tried a personal wiki (MediaWiki and SharePoint) on localhost too and encountered the same problems (hard to reorganize, secure mobile access).

So I switched back to plain text, using the Markdown syntax format. Sync works fine with IMAP based email account (note apps on iOS & Android support that too). For visual complex documents I use a WYSIWYG HTML editor and the HTML format. SVN/Git repo to preserve the edit history. And I coded an desktop/enterprise search engine myself to search through PDFs, HTML and various Office formats - similar to the discontinued Google Desktop search and some enterprise search software.

[ivansavz]

I have totally given up hope on classifying all the interesting things I find on the web each day, so I use an uncategorized save all strategy. I save all opened and downloaded files (mostly PDFs) to ~/Desktop/ and I also bookmark things there (by dragging the URL icon onto the desktop). After a few weeks the desktop becomes a complete mess, so I use a script[1] that puts files into subfolders (by extension), and then I put away this "complete week of research" archive and start from a clean slate (except for one or two active projects dirs).

It's a bit time-intensive to find things, but it's not impossible: let's just say the system is optimized for write efficiency and not read efficiency ;)

[1] https://gist.github.com/ivanistheone/9daa23ae2a7abb472cb2

[cmpb]

I like this idea since it solves the issue of easily persisting things you may want to keep around, but wouldn't it limit the types of things you can persist (e.g. can you easily persist ideas/snippets this way)?

[markbao]

I mostly use Simplenote for ideas and loose notes, and Quip/Evernote for more structured notes. All three sync to all my devices, and while neither are perfect for my needs, they are OK.

The problem comes from keeping everything organized (impossible with Evernote and Simplenote with their lack of structure, impossible with Quip since it's docs/spreadsheets-only) and in a way that works with my mind and workflow.

I'm working on sketching out what a unified personal knowledge management product might look like, which combines a kind of "inbox" of sorts of resources and notes coming in, and also a "personal wiki" with structured docs, and a number of people I know are also thinking about what a 'perfect' knowledge management system would look like. Let me know if you want to bounce around some ideas around.

[paglia_s]

What I would like is a way to organize not only written notes but files too, keeping them where they are in my disk or dropbox/google drive without having to duplicate them in another online service

[napoleond]

I've also been thinking about this problem a lot. On my phone now so I can't expand but I would also love to discuss this further with anyone. (Email in profile.)

[ay1n]

I'm very interested, will send you an email when I'll finish processing all the information from this thread ;).

[MichaelGG]

1. A paper notebook. I don't often actually refer back to it, but just the process of sketching things out seems to make them stick. More importantly, once written down, my subconscious starts working on another idea.

2. Poor solution, but I email myself notes a lot. Usually from my phone, then I drop them into a folder in Outlook when I get back to my desk.

3. OneNote, for more long term collections. Microsoft did an amazing job with this product.

[alanclimer]

I use TextWrangler and several text files along with Chrome bookmarks. I'm minimalist and run a very simple operation so this may not be appropriate for many.

To organize the text files; over time I build an index of categories at the top in CAPS, and each category heading below is also in CAPS. Then when I save / retrieve / cull information I search the category "Case sensitive" to locate it quickly. Once major groupings can be identified and corralled, I separate those into independent text files. It's work.

For Chrome bookmarks I build similar categories but this can get unwieldy if not maintained and subdivided on the regular.

If it matters; all my local files reside in one of two folders (or downstream of them). One is for current "in flux" files & the rest goes in the other "archive" folder. I do encrypted backups on the "in flux" often and the "archive" far less often to external drive(s) & the cloud/online. I have a third "clients" folder but all the files there are temp and go back to their respective servers and I don't backup any of it.

I concur with sp3n concerning over-collecting, often I get back to something and it's already obsolete or maybe not at all. As a result I end up in a data cull session from time to time.

Don't like paper and would love to migrate all to 100 percent online one day.

[aikah]

By the way,is there a solution for windows or linux that would take SQL like queries and allows one to search among files in a computer. Something like :

    SELECT All *.jpg as image FROM /myfolder/** where image.creationDate > yesterday and image.size < 100 and image.filename LIKE TRIP% ;

Then either display the result in a window or as text.

Something that would combine find,grep+pipes into something more "userfriendly".

[renaudg]

You've just described MacOS X's 10 year old Spotlight feature : http://en.wikipedia.org/wiki/Spotlight_%28software%29

[mappu]

For CLI: Cygwin `find` with the -mtime and -size options?

For GUI: Windows search can do this with the datemodified:* and size:* operators, or just click the Search tab in the Explorer ribbon to see buttons

[bsilvereagle]

I used to use OneNote pretty extensively for nearly everything. Recipes, course notes, thoughts while working on projects, lists of reminders, etc. The main problem with it was that if I wanted to search on something, I'd have to search X number of OneNote notebooks.

I've recently started to use fedwiki instead of OneNote, and things are alrightish. Fedwiki has lots of room for improvement as a wiki, and then as a knowledge store.

I think the ideal personal information/knowledge store would incorporate a tagging filesystem combined with something like OneNote/fedwiki. The tagging filesystem would allow PDFs, movies, etc to show up in searches with fedwiki/OneNote handling the plain text & images. Ideally the client that the user uses is something like fedwiki, where you can have multiple different pages open at once, but also allow you to pull in the PDF/video resources.

[ay1n]

For me the biggest downside of OneNote is that I can't use it on Linux/Unix (Wine doesn't always work). I didn't know fedwiki, looks interesting and I'll look into this, thanks for that.

Do you have categories in which single entries are grouped or you just add new thing and search for it later?

[analog31]

Three things that have helped me.

1. A little bound paper notebook such as a Moleskine.

2. A mind mapping program -- I use FreePlane -- to store links, including links to files on my hard disk. An advantage is that I don't have to get the organization right on the first, second, or even third try.

3. A lot of the information in my life is not digitized, such as most of the sheet music in the world. So I now rely on my cell phone camera to record a lot of that stuff.

Amusingly, when I was in grad school, it was still considered to be an open question whether a person should get their own computer. The university computer store had a little guide, and the most memorable advice -- which certainly rings true in my life -- was: "Don't expect a computer to make you organized. If you have a messy desk, you will have a messy computer."

[andrey-p]

What I do is:

- For meetings, events, social obligations, I carry a small paper-based diary.

- For ideas and thoughts, I write them down. The idea is that the act of writing something down aids recall. For example, story and blog ideas go in a A5 notebook that I carry around with me, and I commit them to a digitally backed up document as soon as possible. Note-to-self lifestyle advice goes in my bedside drawer, to check on if I ever feel like I've forgotten anything.

- For things to learn and interesting articles, I don't do anything and keep my fingers crossed that the salient bits will have rubbed off on me, lurking in my subconscious and subtly improving my life forever after.

I'm not 100% sure that last one works quite that well.

[icco]

This is pretty close to what I do, but I also try and keep track of what I read. To do this, I just bookmark everything I read in pinboard so if I remember some tendril of a thing, I can search for it. In general my memory is horrible, every little bit helps.

[yellow_and_gray]

What would you do to improve how your subconscious works?

[mozillas]

I use Evernote. Because it can handle multiple types of content. The most useful information I keep in there are tutorials. Mostly written by me. How to do things that I only have to do a few times a year. Basically recipes for anything, not just food. The stuff I do repeatedly I just remember or automate.

Passwords are a good example of this. I can deduce them(based on a formula), but I don't want to do that every day, so I use 1Password.

But it's not bad to just forget. There are quotes, links, funny pics in my Evernote that I've never used. So I'm now much more selective in my note taking. It's less stressful.

[galfarragem]

To organize myself I use Secretweapon [1] (GTD for evernote) and Folder-System [2] (an hierarchical folder system to organise personal documents based on GTD, making it easy to predict where something is stored). The initial investment (file renaming) pays off very fast.

[1] http://www.thesecretweapon.org/media/Manifesto/The-Secret-We...

[2] https://github.com/we-build-dreams/folder-system

[hammerandtongs]

org-mode in emacs

Simple text file that's actually more functional then almost any other solution, designed around allowing you to create and adapt new workflows and WILL be available and useful to you for decades.

https://en.wikipedia.org/wiki/Org-mode

[jamesisaac]

I approached this problem and decided to try and solve it for myself just over a year ago. Firstly, I decided that instead of knowledge being segregated by medium (notes -> evernote, bookmarks -> Chrome, etc), it should be organised by purpose.

The most important purpose, I decided, was personal goals. Knowledge/information which is relevant to helping me achieve my own goals is the most important thing I should be focusing on, and should be extremely well organised and easily accessible. Any other interesting info that falls outside that is a bit of a shame to lose, but ultimately just a distraction and clutter. For this purpose, I developed this tool: https://nachapp.com

I believe the next level of information down would be general learning/knowledge. Stuff that doesn't fall under any specific goals, but is still useful information to know and understand (and may in disparate ways tie into core goals). For this, I'm currently using https://pinboard.in, although it's not ideal as it's again limited to a single medium. I have a solution in mind, but haven't started developing yet. If you're interested, feel free to get in touch and I can keep you updated (contact info in profile).

[meesterdude]

I've come to find that everyone has a different solution, and lots of people have different priorities. A client of mine keeps everything in text files. I know another that does everything in excel. Others use some combination of apps and services (dropbox for files, omnifocus for todos, evernote for notes and articles) and that works ok for them too.

After trying several solutions, I ended up building a SaaS to manage everything. I found most things out there are fairly boring, and are not at all as powerful as what I wanted. It's basically a brain for my brain; so i can remain a scatterbrain and it can tell me when its time to water the plants, or if food in my fridge is about to expire. But it also handles all my notes, important files, time-series data, and historic dates.

But really, you have to figure out whats important to you and what system best aligns with that; and in the end you'll likely need to make a few tradeoffs to get something working.

But I think there are definitely some principles you can apply to any system you use; I can't recommend Getting Things Done by David Allen enough. His methodology is great, but even if you don't like it or can't use it for whatever reason, there are oodles of great tips; and it'll make you into a natural project manager / information guru.

[petemir]

Do you have it online? care to share?

[smarks159]

I write a lot and have a lot of text files scattered all over my desktop with ideas, plans and notes. Everytime I start a programming project I also end up with a lot of text files with requirements, design decisions and implementation details. I kept all these ideas in plain text files because, none of the existing tools really fit what I wanted. I ended up creating a program to help with this based on the ideas of Doug Engelbart.[1] The program is still experimental and just deals with text at the moment but I still find it useful.

In terms of research papers, you may want to look at the ideas of Doug Engelbart. The process which you speak of, of collecting information and learning from it, Engelbart termed the CODIAK process. There is a section describing what CODIAK is in this paper[2]. (click on the CODIAK Process link in the table of contents). Engelbart speaks of this process in terms of groups and organizations, but the ideas apply to individuals as well. Engelbart's goal was to create an integrated "knowledge workshop", where all the different programs for organization everything would be integrated together and act as an extension of the human mind to augment people's abilities to collect and digest an ever increasing amount of information and knowledge. There is a lot of work left to be done in this area, but it is an important problem to solve.

[1] https://github.com/smarks159/hyperdocument-system-wiki

[2] http://www.dougengelbart.org/pubs/augment-132811.html#6

[therealmarv]

After trying a lot of software I returned at the end always to Evernote. I always missed good rich text and image support and easy web page saving in all other products. Evernote has a great web clipper (also works for Android with Everclip) which saves sooo much time and great IFTTT integration for interaction with other services. Especially having everything synced and editable on your phone is something the other solutions are missing.

[bonobo3000]

I've been using Kifi[0]. It lets you collect any webpage thats interesting, tag it with multiple tags, and search your whole collection based on tags/content of the page. I highly recommend it.

For example, I want to learn more and more about distributed systems, so when I see an interesting article, i tag it. When i have some time, i go through the relevant tags.

[0] https://www.kifi.com/

[pwelch]

This is something I am constantly trying to get better at.

I have recently started using jrnl: http://jrnl.sh

I really like it because you can export it any many formats which I think would come in handy later to import the data, say a database. I always like the idea of a flat file for right now so you can ack/grep on the command line.

The cons are not great for assets such as images.

Looking forward to reading some other solutions.

[laxatives]

Depends on the level of activity/energy required:

Things with well defined start/end dates or with very high priority/high cost of missing (ie flights, parties, reservations, deadlines): calendar (I use my iphone calendar with alarms if necessary)

Things with less well defined start/end dates: starred emails/browser bookmarks toolbar

Menial things that need to be in the near future (ie groceries, laundry, shopping): Leave something out of place as a reminder, or put in calendar with alarm if urgent

Things with low priority/low cost (ie things to read/learn/listen/watch, fitness goals/accomplishments, future trip plans, long term plans, misc notes): iphone notes

Also keep a notepad at everywhere I work regularly (home desk/office desk) and in my backpack with notes

I also feel its very important to maintain 0 unread emails in all of my inboxes. Makes it much more feasible to stay up to date on everything and avoid missing anything important. 90% of the time I'm awake, I will read any incoming mail within 15 minutes and respond immediately if necessary.