top | item 15473702

The software engineering notebook

357 points| Winterflow3r | 8 years ago |winterflower.github.io

130 comments

order
[+] goo|8 years ago|reply
The commitment to taking good notes has been one of the best decisions of my life -- not just professionally. I use my standard text editor (Textmate 2), and have a "notes" project, with all manner of categories organized by folder. I take notes on articles I have read, take notes on any presentations I watch or educational courses I take, or books I read. Moreover, I now take much better in person notes -- I take them into "scratchpads" and then synthesize them later.

It's indispensable in meetings because I both take notes about the goals of meetings beforehand, take note of any important parts during them, and am able to succinctly review them afterwards as they relate to the original goals, and easily keep track of any action items.

I manage EVERYTHING this way now-- what I am currently working on, notes about what I've done in a day, notes about what I've done on the weekend, notes about books I want to read, movies I want to watch, have watched, fitness routines. It goes on.

At first it was a small but useful knowledge base, and a good plan for how to keep track of things better. Now it's a larger, and even more useful knowledge base, where due to review and organization the important concepts in it have mostly made its way into my long term memory. And the process of taking and using the notes has become a smooth and well-honed process. I keep the notes in dropbox so I have easy access to them on mobile also.

In short: I treat my personal knowledge base as if it were a coding project: using well organized text files. I am constantly adding to them and maintaining them. It is working very well for me.

[+] Koshkin|8 years ago|reply
On the other hand, based on my personal experience, maintaining a constantly growing archive of things, be it notes, bookmarks, emails, etc. turns out to be not very useful. Also, it gradually becomes something that you spend more time on than on other things. I now prefer to be more spontaneous, try to minimize the number of things that drag me into the past, start things as if from scratch, "reset" myself, dismiss the baggage that is often turns out to be nothing more than a dead weight. This "lightweight" approach seems to allow me to move forward faster, learn new things that I might have no plans to learn for a long time, because of a queue of other things on my "TODO list". But again, this may be personal.
[+] djhworld|8 years ago|reply
I'm fairly similar to you but not as maybe not as meticulous

One thing that wasn't working for me was storing TODO style things and "X I want to watch/read/learn/play/do" items in my notes. Eventually I found Trello and it seems a good fit for that style of task (for me)

Everything else is stored in my notes though, or sometimes in paper-based notebooks that I take pictures of occasionally for archiving.

Still need to get better at things I've done at the weekend/daily bullet points etc, I often forget or am too lazy

[+] amelius|8 years ago|reply
What is the read-versus-write ratio of your notes?

I.e., how often do you actually read the things you write down?

PS: Have you ever thought about using a "lifelogging camera" instead? :)

[+] dorian-graph|8 years ago|reply
Totally agree goo! I do something more or less the same as you, just to not the same extent—but I will be soon when I export my 'to watch' list from IMDB and 'to read' list from Instapaper.

Since 2013 I started creating a text file almost every day in a directory structure of '../daily/2017/17-10-14.txt'.

I use it as a journal (I put the current time and wrote whatever it is down), a scratchpad, a temporary todo, habit tracker, etc. It's been super useful for me. I've thought about formalising it, using tags, etc. but have found myself getting along fine.

I used to use Day One (macOS/iOS) religiously but have found that this daily txt log mostly sufficient. What I would like is to expand my macros from getting the current time to getting my current location and weather.

For work, I have a 'log.txt' in which I have a section for each day and copy-forward the previous day's section. While we have our own company tracking tools, etc., this is purely for me and has helped me keep better track of everything I need/want to do without letting anything slip out.

I've found that doing each of these things takes up very little time too—partly due to not formalising it or making it fancy. I've thought about using a vimwiki which I'm a big fan of, and gave it a go once, but I stopped. It was no fault of vimwiki itself, but so far have found the text files sufficient.

For my phone (iPhone), I use https://ia.net/writer/, and I keep all these text files in Dropbox.

Another reason why I've stuck to text files is because of the fatigue of different apps and services disappearing due to whatever reasons.

[+] aiisgood|8 years ago|reply
Can you explain how you organize them? Been trying to do something similar but haven't gotten it quite right yet.
[+] tigershark|8 years ago|reply
I have a diametrically opposite experience instead. For me taking notes is most of the time harmful. If I need to document something it is much better to write a confluence page so everyone in the team can access it. While I'm coding taking notes breaks my flow of thoughts and I lose the momentum. Obviously I do write schematics or structures of the components on which I'm working on, but I don't consider that note taking, it is really part of the designing process and at the end the architecture will go in any case in confluence. Last but not least, I noticed that if I can just read some note easily I don't commit that information to my long term memory, so paradoxically, writing notes actually harms my memorisation process.
[+] hyperpallium|8 years ago|reply
How do you find relevant notes from the past? grep (or a fuzzy-match version) would help, or do you organize them by some system?

The organization of notes in an accessible manner is a challenge for me - perhaps I'm doing it wromg.

[+] epsolos|8 years ago|reply
Would you be willing to talk more about your experience? I too am very interested in this process, but have yet to find one that is successful for me. I would love to hear more about your process.
[+] vram22|8 years ago|reply
I had started a simpler version of your approach a while ago, but how do you manage the overhead if you use it for EVERYTHING?
[+] craftyguy|8 years ago|reply
Good god, man/woman. Do you have time to do anything else besides taking notes?
[+] fredsted|8 years ago|reply
What does the folder structure look like?
[+] rocketcity|8 years ago|reply
I have adopted this practice over the last year and have found immense value in the practice. I believe this practice is one of the cheapest ways to help leverage your previous experience on the job as you keep yourself from having to repeat previous learnings over and over again as you try to remember what those compiler flags you used 6 months ago actually were :).

My system is quite simple. I have created a github repo called journal and an alias which opens vim to a markdown file for today's date. If I leave the file and come back later vim will drop me in at the end of the journal. I intend to add some vimscript in the future to automatically add a timestamp when I reopen the file but haven't gotten there yet.

This system has been quite helpful to me as it is resilient to data loss as I can push to multiple backup systems. Easily searchable (grep). And can support prettier documents if I want to open my Markdown formatted journals in tools like Macdown.

TL;DR I use vim to manage this. See alias below.

alias journal='vim + "/Users/username/journal/$(date +%Y)/$(date +%Y%m%d).md"'

[+] 0x03|8 years ago|reply
A script might finesse things a little more, but you could get a naive implementation of the timestamp behavior with something like:

  alias journal='vim + "/Users/username/journal/$(date +%Y)/$(date +%Y%m%d).md" -c "execute \"normal! Go$(date +%T)\<CR>========\<CR>\" | startinsert "'
[+] stevenleeg|8 years ago|reply
If you're looking for a more structured version of this, I'd highly recommend checking out vimwiki[1]. Its diary feature does exactly what you mention, plus it will autogenerate a nice index file for you (among many other features).

[1] https://github.com/vimwiki/vimwiki

[+] sbov|8 years ago|reply
I have a "notes" directory in dropbox. I use .md files, with a text editor that does some highlighting based upon the markdown.

95% of them go into what I call my "dev diary". Each day gets a new file. Any questions, answers, thoughts, etc. while programming or designing go in there. For searchability, I also add any reference Jira/Trello/etc ids.

Beyond that, I have cheatsheets and checklists. Cheatsheets are short, useful things (e.g. commands, undocumented gotchas, etc.) for a given piece of tech. E.g. I have a git.txt, mysql.txt, etc.

Checklists are things I should go over whenever I do X. Am I about to commit something? Well, there's a list of things I tend to forget to do. So I have a checklist for this.

Each project also gets a file (or folder if it grows large enough) in my notes directory.

I previously used Evernote. Then Onenote because it was nicer on the surface pro. But I had to abandon Onenote because they don't have a good Linux client. I decided I was going to stop throwing my notes away and just switched to regular files.

[+] c-smile|8 years ago|reply
Try my http://notes.sciter.com - I did it for exactly that purpose. The only thing it renders and edits notes as HTML/WYSIWYG.

I've found that for some reasons I simply cannot consume/read .md in source form.

After all having <h1> rendered in larger font means something, right?

[+] bluedino|8 years ago|reply
What do you with screenshots or other graphics?
[+] dageshi|8 years ago|reply
http://tiddlywiki.com/

Sits in a browser tab, insanely quick to use which is the most important thing.

Consider it perfect for taking notes.

I use seperate systems for todo and project management (the project management one I wrote myself)

[+] warent|8 years ago|reply
Cool, I've never heard of this before. Completely blew my mind when I saw it's a single html file that I can use offline! I love it and definitely will start using it.
[+] Guyag|8 years ago|reply
Seconded. What I was looking for in a note-taking system was something wiki based, and this has a really nice take on the concept.
[+] herge|8 years ago|reply
Also have a very quick search, which beats any server based wiki.
[+] richdougherty|8 years ago|reply
My first notebook entry is a file called 2011-01-20.txt. I just opened it and had a look what I was doing on that day - really interesting!

These files have useful commands, todo lists, debugging notes, stack traces, useful links.

Don't overcomplicate things - just make a text file each day and dump everything in. Don't bother too much with structure or tidiness. If you need info from the previous day, just copy it across.

Once you've got enough stuff in your notebooks to be valuable do remember to get an automated backup going.

[+] Xeoncross|8 years ago|reply
https://gist.github.com is my notebook which has 800+ collections of my own samples and notes along with 800+ stared notes from other developers.

It's one thing to keep notes. It's another to have a social network around notes.

[+] accordionclown|8 years ago|reply
that link is a generic one, to an account's own gists. it is not a link to your gists, _except_ for you.
[+] sizzzzlerz|8 years ago|reply
Back in college, going for my BSEE degree, the professor or lab assistant were constantly stressing the need to record everything done at the bench in a log book with pages dated and signed. Part of this was to keep a history of lab experiments and remind you of what you did and/or learned. Really, however, it was to develop the habit to carry over into your professional career. There are many examples of this but the most important one I remember is the notebook kept by John Bardeen of Bell Labs wherein he recorded the path taken to invent the transistor along with Shockley and Brattain. Not only did it win credit and and patent for its invention, it also won him the Nobel Prize. All because they could conclusively prove their invention along with the when of its discovery.
[+] krallja|8 years ago|reply
> dated and signed

What's the signature for?

[+] barrkel|8 years ago|reply
I knew what I wanted in a digital notebook, so I spent an afternoon writing one. This:

* no explicit saving or loading, everything automatically persisted as plain text

* infinite persistent undo / replay of text entry, for retracing steps; can double as work logging; this uses a log file alongside the plain text; this also lets text be deleted without being lost, which helps with clutter

* "most recently edited first" page order, so you can forget about those old pages

* minimal ceremony to create a new page; use the first line as the title

* quick searching and navigation in time and space, using keyboard

https://github.com/barrkel/scratch

[+] irl_zebra|8 years ago|reply
Nice! This is basically Notes on MacOS, which seems to basically do this (minus infinite undo/redo). I use Notes extensively and it works really well.
[+] seltzered_|8 years ago|reply
When I started my first 'engineering' job at a chip startup over a decade ago, I remember being given a leather blue notebook with gold lettering titled 'ENGINEERING NOTEBOOK'. It was great for focusing on a single project and especially made sense for areas beyond code like drawing electrical schematics, but over time I've become addicted to digital approaches.

These days? I rely heavily on Trello for personal projects - organize things by keeping a separate Done board (list for every month) for the project for times I want to flip back through older work done. Also keep paper notebooks around for times when I want to sketch ideas out.

Additionally there's a 'rhizomatic frankenstack' of nvAlt+simplenote / pinboard / markdown files in project folders etc. I keep wanting to set up something where it's all searchable and ideally dream of something like federated wiki with a lot more UI polish.

[1]: term from Venkat Rao's post http://mailchi.mp/ribbonfarm/frankenstacks-and-rhizomes?e=96...

[+] quantumhobbit|8 years ago|reply
I keep a notebook partially to help myself but also to protect myself James Comey style. It really comes in handy when non-coding managers ask what I do all day or when they try to gaslight me on my commitments.

And yes, I am looking for a new job.

[+] rayascott|8 years ago|reply
This might seem like over-kill at first, but I've been running the Enterprise version of http://www.xwiki.org on my laptop, and couldn't be happier. Its served up using Jetty, and starts up automatically when I login. I like using tools that I know will scale well in the future, should the need arise (no reason why you couldn't run it on a free Heroku instance, if it handles the load, and it should, as it just idles along most of the time).

It has a Code Macro plugin that supports Pygments syntax highlighting. It all comes down to how you structure the wiki (I often turn to Wikipedia for help with that). I've been an Engineer for 20 years, and the info-flood doesn't really end, unless you dial back your enthusiasm. I like to document the tools and services I discover so that I know what's valuable, and what's out there, should I ever need them.

Once installed, its very easy to create new pages and edit them. Plus there are hundreds of plugins.

[+] xor1|8 years ago|reply
Quiver is the closest thing I've found to what I consider my perfect note-taking software. I think getting any closer would involve writing my own at this point, and that's simply not going to happen.
[+] crispinb|8 years ago|reply
Same here. It's perhaps a slight shame that development has been so slow over the last year or two. But OTOH to be honest it's good enough as it is to be far more useful to me than any other note taking app I've used.
[+] shams93|8 years ago|reply
Usually they're work specific notes for my coworkers, its always nice to share a line by line account of your build process when you're using some arcane systems that require being built from source with really complex configuration issues to handle.
[+] shams93|8 years ago|reply
The only tool I use are markdown files and gitlab to do this, I've found the .md markdown format to be extremely useful compared to the bad old days of emailing word docs.
[+] BjoernKW|8 years ago|reply
While I don't keep a software development notebook as a rigorous practice I note down interesting or useful insights, findings and patterns in Evernote. I sometimes keep code snippets as GitHub Gists but usually I find source code to be too transient and dynamic to keep it anywhere else than in the actual project it's needed in.

More importantly though I use my blog for keeping more polished, more elaborate versions of my more relevant notes. Sharing your thoughts with others not only benefits others but it also makes you think over and refine your original ideas.

Distilling your thoughts for the benefit of others will help you better understand the matter as well. While we teach, we learn.

[+] bloaf|8 years ago|reply
I've been actively looking for a good tool for personal note taking, and finding lots of things that seem not-quite-right.

Plain Text:

Pros > Simplest, works everywhere/on everything, scales well (append forever!)

Cons > Least automated, no images, no math, no code, hyperlinks depend on editor, no embedded files

WYSIWYG Doc Editor (Word, Writer, etc):

Pros > As easy as plain text, can do images/math/hyperlinks/embedded files, somewhat portable

Cons > Bad at code, not many automated features, unwieldy once file gets large

Markup language tool (Org-Mode, Wikis, etc):

Pros > Powerful, portable, good at code/links, scales well, highly configurable, highly scaleable

Cons > Have to learn the markup language, mostly bad at math, high up-front costs (e.g. needing to set up a server or learn alien keyboard shortcuts), non-wysiwyg feels less like a "quick note" and more like a professional document, image/doc embedding can be hit or miss

Note-taking app (Onenote, Evernote, etc):

Pros > Automates lots of note-taking tasks (e.g. time stamping/tagging), can do images/math*/hyperlinks/embedded files, scales... fairly well

Cons > Least portable (dependent on app-specific sync features which may change), less simple (best use of notebook features are non-obvious), still bad at code

Electronic lab notebooks (ELNs) (labfolder, benchling, etc):

Pros > Good for organizing/presenting data, embedding files, scale very well, have collaboration features

Cons > Very not-simple (most require either require server config, or you don't own your data), most are somehow still bad at math/code, frequently expensive and not portable

Task Managers / To-do lists (Trello, Asana, etc):

Pros > Very intuitive to use, can sort of embed documents/images, have collab features

Cons > Not good for longer-form notes, bad at math, not meant for note-taking, you mostly don't own your data

Others:

Jupyter/Mathematica can do WYSIWYG text, code, math, data, and images all in one document. However, Mathematica is expensive, and Jupyter is non-trivial to get up and running. These tools are primarily geared towards creating a polished document, and are less good at documenting the evolution of ideas inherent in note-taking.

Personal Diary/Journal software is mostly underwhelming, most of these seem like cut-down note-taking apps sold to the non-tech-savvy

[+] akvadrako|8 years ago|reply
Jupyter is trivial to get up and running. I did it just a couple weeks ago and it took under 5 minutes. There is a docker image but I think I just did "pip install jupyter" and "jupyter notebook".
[+] gglitch|8 years ago|reply
Someone should make a database of every HN comment on the best ways to manage notes and/or tasks. I might even buy a subscription.

Edit: My current stack, to show good faith: paper for log and tasks, Org and Git for memoranda.

[+] mumonz|8 years ago|reply
I have a small website in the wild. Every page is a markdown file with notes and ideas about a subject. The entire website is git versioned. I keep a copy of the git repo on Dropbox to easily edit articles and push to upstream (that is the place where the website is hosted). Ah... there is an sqlite file with a single table with the following cols: article-name, file-path, creation-date, tags (for search by argument). Article last edit time is taken directly by the file. Every meta-info is shown on top of the html rendered markdown file (i use Parsedown)
[+] canadianhacker|8 years ago|reply
I like these types of discussions.

For me, note-taking is something which helps me digest complex ideas. It's a valuable thing and something you can improve at with time.

Then I read the reasoning behind why others don't take notes and I stop to think. Hmm, maybe there is such a thing as too much. Maybe note-taking has a fine balance. Breaking up complexity and summarizing it with notes can be very helpful with getting something finished. Writing journals of everything you do daily probably detracts time from actually performing the tasks required.

There's some science out there on how dopamine gets released when we TALK about performing an action with desirable results. (I'm going to create this project, I'm going to workout this summer and lose weight, I'm going to read 12 books this year). Basically, you're being rewarded for not doing but just saying. If you spend too much time note-taking on a plan or project, you might be getting the dopamine fix, which might make it harder to execute.

Similar comparisons are with startups and large companies. Startups work fast and can ship products quicker, at the cost of maintainability. Large companies have to take their time documenting, grabbing requirements, assigning the team, and following internal processes. The process kills productivity, in favor of predictability and maintainability.